Do that. >> It's killing me. My wife is in Mexico right now and she just started trying to video chat me. Let me close out Skype so I don't break the internet doing two video conferences at once. >> The wonders of the internet yeah all right cool. Here we go with Rob Sherwood from Big Switch Networks. Inventor of Flowvisor and now a big proponent of Bear Metal Switching. And I guess we're going to about both these things today, so thanks Rob. >> Thanks for having me. >> Cool so so just maybe we'll start talking about FlowVisor. Actually this is, it's a very opportune time to start talking about it. This particular week the students are going to be you know, implementing a poor mans version of it in pox. I guess you could say, so it's great to, to be able to ask you some questions about it. I guess like, just motivation, first of all. What caused you to kind of recognize that there was a need for something like FlowVisor. Like were people talking about needing to slice networks? Where did the idea come from? >> Not, well, I mean, it's, I mean the idea comes directly from a hypervisor which is, you, you have one set of hardware. And you want multiple people to use it. It's, I fell like it's actually better motivated than actually even a server hypervisor. Just because putting together the hardware for a network is expensive. And you know it's very hard to have, give each grad student their own network, right. You can probably if you have enough money you can give each grad student their own server you know virtualization makes that easier. But you can't give each grad student their own network. And we were trying to run this CO in the Gates Building at Stanford. And we really wanted all, like, we wanted each question to have their own, place base, to work on something. It was very clear we weren't going to get multiple networks. And even more so coming out of my, dissertation work I was doing, topology discovery. I was kind of militantly bent on the idea that you can't have a simulated topology that no one really knows what the topology of the internet is. So you're better off using the actual network, and that was how that all came together. >> That's, yeah, I guess the title of your paper, can the production network be the testbed, or. Yeah, exactly. >> Really the motivation. Actually did that, did that ultimately come, I mean, did that come to fruition in some sense? Like you were able to actually play on the production network where you, you know, thanks to FlowVisor? >> It was, what I jokingly call the lowercase P production network. So, lowercase P is, if it goes crashing down then people get annoyed and people get mad but no one gets fired. you know that's much more the case on you know like a typical campus network. And prior to the you know at Stanford you know the computer. The network that I used to check my email of the everyday network was FlowVisor. Unfortunately its, its become a verb now but but yeah. So that, that everyday network that I used so everyone in the [UNKNOWN] group who I was working with at Stanford. All of their network went over, devices that were controlled by FlowVisor. And it was actually running all sorts of graduate student experiments on parallel and different slices. You know, we spent the better part of two years, debugging it, but it eventually got fairly stable. Very cool. I guess and are the, are the people who are using it now are they the same set of people. Like it's, I assume it's still running at Stanford for example but are there, are there other sort of groups of customers. I guess you might say or is it still kind of. >> Yeah although I have admit I haven't really been in the thick of it for a couple of years now. The the Open Networking lab folks Guru Parulkar and Ali Sharafat and company have taken that over. You know to the best of my knowledge there's kind of a bunch of genie folks using this so I'm still on the mailing list for that where. They announce downtime it's actually like an official kind of production service for them. There's the Ophelia group in Europe they're using it. And there's actually kind of a long tail of people using FlowVisor in weird places. Like there was a guy at an ISP in New Zealand who was telling me he had FlowVisor deployed. And you know when was it going to update it for 1.3 and that, that type of thing. So it's gone surprisingly far. I also pretty get, pretty surprised at how people use FlowVisor. I mean so I, I never really thought about the, the power of having an API. Which is the same down facing as up. And one of the, the things that kind of falls out of that, which I never even thought people even thought about as a use case. But people just started doing it. Was actually [UNKNOWN] FlowVisors. So having like a base FlowVisor and then in the slice you have another FlowVisor. And that further subdivides just that size and I, I thought, I honestly if you would ever ask me for a codec written. But work for this case I would have said probably not. But you know the first bug report I got was about a guy running this apparently three levels deep. So there's like a base FlowVisor, and then a slice was another FlowVisor, and a slice was another FlowVisor after that. And you know what he told me was he wasn't actually doing anything terrible intricate in the, in the floor space. It was actually that he was using FlowVisor almost as a monitoring tool. For, is the controller up, how many flows has it pushed down? Tell me things about the controller that are actually independent of its limitation. >> And the additional layers is buying him what exactly? Why would you need to. >> Additional levels of visibility. >> Interesting. Yeah, I guess that, that's I guess that was another thing. That I, I mean that, I imagined that there would be these, these strange use cases or unex, at least unexpected use cases for FlowVisor. Do I, were there others? I mean, besides the test bed application? Are there, are there other ways that people have used it that, that you find interesting? >> There's the network monitoring case, there's actually this other interesting case of. So it's what my, my co-author Kk Yapt used to call horizontal slicing as opposed to vertical. So if you imagine you know, cut up the OSI model. in, what I would call a vertical slice, you are required to implement the L2 forwarding, the L3 forwarding. You know, maybe some ARP agents on top, maybe some DHCP on top, you actually have your whole network. but, you know depending on how you want to cut the flow space. Maybe you actually can separate out the L3 and below from the L4 and above. And so maybe you don't want to handle ARP. Maybe you don't want to handle DHCP. Maybe you just want to handle kind of specific layer floor courts. And how the, the layer two forwarding works for that, for them. And, and so that's another way that you can actually slice the network. We called it horizontal slicing, kind of like across the OSI layer as opposed to, like up and down, creating parallel OSI stacks. >> interesting. Yeah, it seems like in, like a theme that's coming out is like you, I mean the, the this sort of visibility, or lack thereof. That FlowVisor can give you allows you to roll your own abstractions. >> Yeah. The other one that floored me, folks at Internet 2 came back and said they were having performance issues with FlowVisor. And it had been running well for a couple years and I was a little surprised about it. They had such a complicated floor space that horizontal in the order of 60,000 floor walls. Apparently the problem was and that was never written for this. It actually kept going because internally. It kept to views of views of things that was fairly efficient kind of in FlowVisor it is fast [UNKNOWN] even that's on the control path. But every time you updated a rule it would actually write the entire config from scratch, down to the disk. Which made sense when you think of a config as being a couple hundred lines, because that's what we had to play with. And, so you know the newer versions of FlowVisor actually use kind of an acid style database. Where it performs updates on the rules of incrementally as opposed to all at once. >> I know, it's a very fun experience. We've tried to explain this to grad students of having a piece of code that you actually support and get users for. Brings up all sorts of different aspects of the, the research, rather than the one shot use case that you had in mind. And it's something that I highly recommend to people if they ever get a chance. >> Yeah, I know that's interesting actually. I mean, I think it's, it's fitting that we also talk with Larry. Because, I mean, he wrote a paper on this whole, I think, public light inner experience driven systems research, that, that kind of. Exactly describes what you just said, which is that the process of deploying something actually opens up all these new, these unexpected directions were not where you started. >> Well and practically it gets people closer to the real problems that operators are having. Rather than kind of the interesting problems that are fun to white board. And, and I, kind of a personal soap box of mine is trying to get the researcher communities. And kind of the engineering operator communities closer together. And so this is one of [UNKNOWN] of doing that. >> Yeah fantastic no I mean I think that's, that seems to be like a perpetual struggle in the networking area. And, we continue to see in the SDN space as well. I guess along those lines as well, you're talking about the evolution of FlowVisor. Do you see it, how do you see it evolving in the future? It seems like there are you know, there may be other needs coming down the road. Like this whole, do you see distractions, do you see different use cases, coming? >> There's a couple different directions that the folks at Owen Lab are, are taking it. One is, so, what FlowVisor provides, is, flow space virtualization, but not really topology virtualization. Let me go on a small tangent of virtualization versus slicing. So, in virtualization, you are given a slice of the pie in practice, is my thinking, but you aren't told that you're given a slice of the pie. You're given an abstraction of a pie. Right, right. >> And then, in slicing, you know exactly that you've been given a slice of the pie, and you know what the borders of your pie are. And if you try to eat outside of that then you'll know that you're on your own. And so, flow advisor doesn't do topology virtualization. Meaning that it won't make up a switch that doesn't exist, it won't make up a link that doesn't exist. But you can ask it for a subset of the network, so yes include that switch, no don't include that one. What the folks with Owen Lab with the next generation of FlowVisor that they're calling Open Vertex? I'll have to look up the name. It's not open source yet, but is actually the full-on topology virtualization. And that is interesting for a researcher if they want a more complex network than what the underlying hardware allows them to do. Appropo of my earlier topology discovery research. I think it's just a little bit of a red herring. Just because people tend to make up typologies that seem like they're a good idea, but don't actually reflect the, the realist typology. So that's why I was always more on the other side. >> Yeah. >> But you know, it's a good idea, and we'll, we'll see how that works. The much harder problem, which there aren't as clean solutions for, but I continue to think about the, are how to vitualize in the multi table case. So, one of the ways that fFlowVisor works right now is you can actually have a one shot stateless. Decision for is this flow rule allowed or not. Because if you're effect, if you're writing to a table, there's all the packet processing, at a single table design all the packet processing happens at that one table in one shot. And it's easier for the floor visor roughly state lifts to figure out you know should I allow this rule or not, or do I need to modify it? And the multi table case when you write to a rule in the fifth table whether you're able to do that or not. Depends on what you've done in the first through fourth table up to that. And so if those pockets arrive either out of order or if the above controller programming down. Sends it out in a different order. It's actually very difficult to understand from just a policy perspective. Should I allow this forward, this rule change or not. And so what ends up happening your production networks with say virtual routing and forwarding VRF. Which is a, a well known virtualization primitive. Is, you end up qualifying everything on some sort of global label, like an MTLS tag, or a VLAN tag or a VRF tag, something like that. You can do anything you want in this space as long as it's qualified by this label. And so, I kind of imagine that's where FlowVisor will go in the future as people move it to multi table designs. Although practically speaking, that's no longer my day job, so, if, if, if anybody's interesting in doing that work, let me know. I've got some ideas. >> You had, you had mentioned basically the use of tags, as a way of dealing with multiple, multiple tables. And and, that actually was another question I wanted to ask. Because you mentioned, sort of like, the way that it's done with other types of protocols is by using tags. And, and another question I wanted to ask about that was. Whether there were, I mean, do FlowVisor concepts apply to things that aren't open flow? I mean, certainly they do, but. Do you know, are you aware of anyone who is using flow advisor or a variant of it, or flow advisor concepts with different [INAUDIBLE]? >> So, just so you, that no one ever thinks that they're cool or that they've done something new before. I was actually talking with Vince Surf about FlowVisor back in the, the early days of SDN. He says oh yeah, yeah, yeah we did this in the early days of the ARPA net about 30 years ago. So [LAUGH] so I guess the short story is people have been playing these types of monkey in the middle type tricks forever. I mean you can think of at some level this is what NAP does. This is what my [UNKNOWN] sidecar work did in the TCP space. This is what it, long before there was the VFX instructions on the the VC instructions on the intel chips. This is what the, the VM ware folks had to do in the XDD6 instruction level. Any time that you are doing the protocol in or the API. in is the same as the API out. they'll, there, there's kind of a certain set of things that you're like, okay, well, I'm trying to do isolation in this space. I'm trying to. Really hack my way through the protocol to provide the, the same virtualized feel up, as you are down. I mean, you, you'll see this even in RAID, right. No I, I, I feel like the same paradigm is repeated, through many parts of computer science. >> Mm-hm, Wh-what about, types of isolation and things that, that FlowVisor doesn't do, I mean. It's, it's a, you know, I think you mentioned before it doesn't do topology virtualization. But are there other, are there other things that like it, it, it doesn't do that you think it should do? I mean it seems to me, that it's pretty focused or, or sort of operates m, on f, slicing on float space primarily but there are, are there other things that you would like to see it do? >> So, there are a bunch of things, and we actually even get into this in the paper, of what it should be doing versus what it, you know, it does do. So there's a control channel between the switch and the controller. And yeah performing isolation and yeah essentially bandwidth, I, protection between that, that channel, is, is actually kind of like a well known hard problem even without virtualization. So if we look like at a standard router, trying to ensure that, the BGP announcement messages that arrives to the front panel line card, correctly punched it up to the control plan supervisor module. And that they don't compete with the LDP packets that, and there might be, you know, some sort of arc storm, or, or DCP storm going up to there. They're trying to make sure that those things don't saturate that bus, it's kind of a, a well known hard problem, but, it has functional solutions, but it, it doesn't have great kind of closed-form solutions. Which is to say, you insert a whole bunch of rate limiters and you kind of know that the sum of the bandwidth you're reserving is overprovisioned, so if everything goes badly at once, then this, this, the box is just going to die, the system is going to die. Right now, at least with Open Flow 1 dot 0, there was not enough of those knobs exposed, even though they exist in the underlying hardware. To really make sure that's isolated properly. So if you have, two badly written [UNKNOWN] controllers running in parallel, their performance is going to be bad. >> Yeah, that's definitely one of them. Another one, which is a, a hard thing to do, for the underlying forwarding memory, everybody talks about TCAM as one of the, the memory of choice, and it's certainly the most flexible one. And what's nice about TCAM it is, it, it's logically like an array of wildcards, and so how do you partition up an array of wildcards between end slices? Okay we, we, that's kind of easy, you could chop them up into end waves or you could do some sort of optimistic locking or something like that, but, there are other memories that don't have that nice structure, so, like a CAM or, you know, any sort of hash based memory. Yep, that's much harder to provide hard isolation for, so, you imagine you, if you've hatched into seven of the eight available buckets, you know, shall I have stopped you? Right? [LAUGH] and, and so those are things that I, I think at least with the current forwarding A six and the forwarding memory it's going to be hard to, provide real h, like bulletproof isolation the way you would get on a hypervisor, like a server hypervisor. >> Mm-hm, yeah, and, and I guess, yeah, I, I, I, it seems like there are other cases too where you might have overlapping controllers, but like if, if one of them is. Only needing to read and not write, and you could maybe, there are cases where maybe you can get away with some overlapping of, and-. >> Yeah, and I don't know how well-publicized it is, but even though the FlowVisor code actually has a, a read-only notion of a flow space and a write notion, and so those are separate bits, almost like on a file system. And so we actually had that for, for monitoring purposes even back in the, the Stanford gates days. >> cool. >> Yeah. >> Yeah so it seems like, it seems like there may be other cases like that where we could allow sort of, certain types of concurrents >> Well there was an interesting one that we did with our, our wireless work which was asymmetric flow space, so the path from A to B belong to one slice, and from B to A belong to the other. And so, the idea was, you know, in a wireless network, you should be able to control all of the packets coming to you. And so, the idea was, everything going to you, you could decide, do you want it? How, what path does it take? And the, the, kind of the pitch was your mobility agent can communicate with a network and you know, help better figure out how to get packets to you when you want them. And, but you know, as soon as you put a packet on that we're going to someone else, it's no longer in your control. >> I see, yeah that brings up just like broader issues too, about authorization and, and things like that. Right, I mean, it seems like, I mean, did you, have you thought about, sort of expanding, the notion of, of ownership of flow spaces beyond the sort of like, oh I own this part but. Maybe like I'm authorized to do something on this product flow space. >> So yes, and actually sort of you know with my, my current work at big switch people ask me often you know, so are you using like FlowVisor technology underneath? And the answer is, not at all so, FlowVisor is very nice if you have these very hard silos of, yeah, slice A is not going to touch slice B. But we actually have exactly the opposite thing, where we almost want in trust slice modularity, where I want to reuse code there, I'll only have one slice because there's only one thing this part of the network's doing. But in there I'll have almost kind of like controller A, that takes a first pass at things and it might do nothing, or it might decide to do something, but then controller B, might logically override that, or it might augment it, or it might do it's own thing on top. >> Mm. >> And try to like, and-. >> Like one controller subsumes the, the operations of another. >> Yeah and so you know, one audits and one supplements. But it's certainly not this hard isolation between them. >> Yeah interesting that seems like wh, that seems like one interesting case. The other one I was thinking about too is like, things like extranets right. Where you have maybe two different networks or two different control pa, parts that you want to basically, you know these two shouldn't be able to talk to one other but they should both be able to talk to something in the middle. But you don't want, you know, transitive properties to, to start. >> Yeah. Well, and so, what's interesting, and this throws people for a loop, a little bit, you know, the entirety of a FlowVisor is very much for hardware owned by one, kind of la, administrative energy, like, kind of at an AS level. And so it doesn't really make sense in my mind to have a FlowVisor that spans AS's. So,if actually the flow visor is, it, it has like super user access to each of the switches it controls, and then provides kind of a, a crit, you know, a, a single user, a multi user experience to controllers on top. But its not multi user in a sense of multi administrative domain, and so what is the protocol that FlowVisors in different administrative domains talk to each other to make sure there's end to end flow space? This is some of the work we got into wi, with, with Jamie, and you're reserving kind of end to end flow space across a, a multi administrative domain network. >> Is there a solution to that yet? Or >> I mean i know the GD people are doing it, just along the networking dimension of that problem, I think it's effectively solvable. what, what's, the sleight of hand trick that FlowVisor does, is it actually describes the policy language using flow space. Which is the same language that the, the flow mod open flow uses. So doing things like a mask, like policy says you can do X, you have tried to do y. What is the intersection of this, is actually, almost the same operation of a packet look up of. What it, yeah, what, what role applies here and so it's something that, is a bit subtle and maybe is obvious in retrospect but we actually got wrong a couple times beforehand. >> Hm, interesting, what about what about security concerns. I mean it seems like sort of almost by its nature FlowVisor tries to provide some level of, you know, you might call it security you know through isolation. But did either security cons, like do you think there are sort of open security issues either in terms of things that FlowVisor itself might be vulnerable to? Or, thing you know, security applications that you might implement with FlowVisor or an extension of FlowVisor? >> So, security applications definitely. Let me go on, on a little bit of a tangent about, why there are no new security application with FlowVisor, like just, by having the control. >> Cool. >> So, if you think about, like, a modern, multi-line card chassis based system. So, if you're familiar with the model numbers of stuff, like so, Cisco's, cats 3000, you know, next is seven K, Juniper's MX, things like that. Basically what you have is you have a whole bunch of line cards. And then you have, kind of a, a supervisor module or a route engine which is, basically a PC. And that PC runs user space processes and those processes communicate information over, an internal backplane down to the line cards. And the, the protocol of that communication is very proprietary, it's very closed. I actually joked that that's closed flow. And so the, that notion actually helps people better understand what open flow is trying to do. Right it's not trying to act like a routing protocol between these models, it's really trying to extend and yep, be the same architectural thing as closed flow, that is the, the logical programming protocol from the, the supervisor model down to the line cards. And into the back, the fibred backplane, and so, you know, from that sense people are like, well, isn't essentialized control plane inherently less secure. Well, no I mean it gives, it puts a lot of eggs in one basket but you already have that one basket with the, the supervisor module. And that's why you have lots of best practices of, you don't put the IPs of your supervisor module on publicly accessible network. Use two factor authentication getting on there. You restrict via IPs even in that case and there's a bunch of things that you do because, yes that's a, a, such a, a high value tar, it's a high value target and the overflowalyzer is a high value target. But there's a whole bunch of standard things that people know to do to secure them because it's the same problem that they have today. Now, that said, to your question about, are there interesting security applications that have been enabled? Definitely, so even at Stanford this is where the, the read only flow space comes. We had A slice which spanned the other slices and could monitor the sum of what was going on in the network. In fact, if you look at some of the FlowVisor videos, you get this kind of like six panel view of six different slices and the, the center middle topology was the, the overlapping one that showed you the sum of all the flows. >> Right, right. >> And you're having that super views, super user view, but in a real, only way. Well, I actually thought that was pretty cool. >> That's, that's, that's pretty neat. So that's sort of a monitoring application, I guess you might say. >> Exactly. >> I see. >> And then, you know, folks from, openflowsec.org, did, a couple of interesting things, so they, they had their, Fort Knox and eventually they ported that work to, to Floodlight it was called SE Floodlight. They had kind of the, the next phase of that which is a, a FlowVisor type application, which would actually enforce other forwarding paradigms, you know, maybe something more complex than, a flow space style isolation. For trying to say are you allowed to make this flow entry or not. And so you, like, like having this, policy point in the middle between the, the underlined switch board in hardware and your, your control application it's a fairly powerful tool. >> Yeah I mean it seems to me like you could use it, you know, to implement all kinds of different separation. I mean as you, as you mentioned sort of, you know separating different parts of the topology but also separating control of one another. >> Yup. >> You know or not, as the case may be as you already pointed out, so yeah it seems like a pretty useful primitive for, for doing that, for, enforcing that kind of isolation and separation. cool, so I guess you have a, you have a day job as well now, as you, as you alluded to which >> Little bit. >> Does not involve FlowVisor ah-. >> Yeah. >> Full time. Like, and, and you've been giving some, some some talks and things on something called bare metal switching. And I actually think the next week of the course. After virtualization, people will be looking at a, all this programmable data plan stuff that, you know, basically didn't exist a year ago and now it's kind of all the rage, but u, What's bare metal switching, like what's, what's that about and why should we care? Why should people doing SDN care about bare metal switching? >> So, let me answer that in a little bit of a round about way. It has a short answer but I'm going to indulge myself. 2008, 2007, open flow, the entire purpose of this was effectively as a bad engineering compromise. Which is to say, everybody who wanted a program that works, wanted to program the lowest level affording chip. I with, with, with no software in, in between like you know, this is, you know, [UNKNOWN] you know, back when men were men and wrote their own device drivers. Like yeah, that's what people wanted but the reality of the business world was that you know, the large [UNKNOWN] vendors were not going to let you, just throw away all of their software that they've spent you know, tens of years working on. And so, kind of a compromise was to expose a high, a higher level AP, remote API, which eventually became open flow, to do some of that programming. And so, open flow pushed forward as you know, this is kind of a compromise, where, we're not going to let you put all of those we're not going to let you put new software in the box, but we'll give you some books to, to do interesting things. And so, fast forward, what bare metal switching is, is you can actually just buy just the bare metal, that is no software, and put all the software on it yourself. And so, it's like that next stepped past writing a controller, is you now have to write an operating system for the switch as well. And, what's interesting, is once you have that, actually the value of open flow, as an architectural piece becomes reduced, because now it's just an R.P.C. language. And, you know, whether you're communicating the, the flow information over open flow, or if you want to put it to thrift or Google protocol buffers, or just on R.P.C., it doesn't really matter. You're communicating, you can. >> Starting from base, you're starting from zero so you can define what that control protocol is. Do not. >> Exactly. >> Run by, by, by whatever software the switch ships with. >> And more to the point, folks who make those chips are actually coming more and more aggressive in terms of making a lower level hardware details available. So, like I'm under NDA with all of the chips makers: Broadcom, Intel, Marvell. These people mellonox and they give me the lowest level hardware information so that I can actual build up the complete software stack. >> What caused this shift, I mean you mentioned like initially when open flow was kicking off there was some push back against doing this. But now that's, the, the you know tide has turned so what cause that, that to happen? So they're a whole bunch of things came together at once. One is hyper scale networking, so just you know the Googles and Facebooks of the world just started buying so many boxes, that the number of nodes being sold, changed the commodity of scales of the economics of things. And so now it was actually possible to have. Third parties, building all of these different components to make a big solution right? So if you look at you know every network vendor in the world right now sells a chip, a switch based off of broad com silicon with a box built by somebody else. And has software in it that might be written by them, but probably has some stuff from other third parties, and so this whole space, because there's become so much money in it, has horizontalized and has become kind of this component ecosystem. So, that's the, kind of the hyper scale network info aspect. At the same time. Cisco did this thing, and I absolutely understand why they did it, but I, it would be interesting to see, when all is said and done if it was a good idea or not. They actually produced a product that competed with all of their, their partners, so the UCS product was a combined server network storage. Where it used to be that HP, Dell, and IBM were the number 1, 2, and 3 Cisco resellers. Because they would sell the servers, and Cisco would sell the networking. And as soon as Cisco started competing with them, what happened is, HP bought 3Com, IBM bought Blade Networks, and Dell bought Force10. And now they all turned into. competitors. And so those competitors were looking for what is the number one thing we can differentiate from our customers? And they talked to their customers and they all said that the, the number one thing that we're unhappy with is vendor lock in. We want the system to be more open. And the last thing that happened was, The companies who have been supplying the boxes, and the underlying silicon, to, to the general personal Cisco's and bouquets of the world made the tipping point to decide to sell these boxes directly. By saying you know. We're getting, you know, lower margins, as Cisco gets like 70, 80 percent margins, on selling a box. They're selling them at like 20, 30 percent margin. you, this doesn't seem right. Let's go direct. And you know, the Facebooks and the Googles and the Amazons of the world have been doing this for a while, but, something happened at the beginning of last year and I haven't quite figured out what. But enough people started talking about, isn't this actually a commoditized market and then all of the sudden this huge ground swole of things started changing. It was very fascinating to watch and so right now you can buy, a favorite website of mine is whiteboxswitch.com. You go to the website, you click this, you pick the switch you want, you see what the price is, you click a button, you insert your credit card and you go to switch. And then on it it comes with a piece of software called Oni which is basically a open source boot loader. And it will let you pull up the operating system, the switch operating system of your choice on there. And so actually, I don't even know if you've heard about this, we should chat more often. We now have an open source project called Open Network Linux, which is a base operating system you could put on that switch. So that, you know, you as a, a, a grad student or, or you know, a DYI professor type. You don't have to write your own fandrivers, you don't have to write your own SFP modules. I have learned so much about the hardware internals of switches that no one ever really needs to know. And we, we solve a lot of those problems for you, so that if you wanted to write your own protocol that was open flow plus plus, you don't have to deal with all the rest of those problems. That sounds like a huge, a huge win for getting food strapped really quickly. >> Yes. That, that's certainly the intent, and actually it's the underlying part of the operating system that we put on the switches commercially. So you know, we're trying to support it, and make sure it's useful for people. So that's open netlinux.org if folks are interested. >> Yeah, I'll definitely check that out. What are the, so what are the implications for for switch vendors. I mean it seems like you know scary times or, or, or maybe huge opportunities depending on who you are right? >> Well >> You described the whole thing just broke apart right? You've got people signing components and. I, I think, you know, the, the reality is the pricing modes for these things have been broken for a long time. If you look at, like, what the costs are, the costs are predominantly on the software side. And people have been selling, a vertically integrated solution. And so now as people start to see, okay, well the hardware is actually really cheap and it's the software I'm paying for. It will take some changing of people's mindsets, and frankly, you can't have a 50 plus billion dollar industry at 70% margins for too, too long. Eventually that's going to start going down. But I think that you know, a lot of the incumbents have good quality software that people are still willing to pay for, and it's just a matter of. What, what I'm hoping to see and I'm hearing rumours that it will happen that big name people will actually start selling their software stacks independent from their hardware. And so, you're actually, sort of, I don't know if you saw, but, Dell made a big announcement on this recently, they're the first to take that plunge, so if you buy a switch from Dell. You get only this boot loader on it, and then you can put the software from Dell, from their Force10 acquisition, or you can put the software from Cumulus Networks, which is another start up, or you can put the software from my company Big Switch Networks, and there are more people filling in this space. Or you can put the open source, open network Linux that I was talking about. And you know, you can actually dual boot your network, which is just a fantastic thought. >> That's amazing, so that's, that's how, how is it going to change the game for, for open floor. I mean you mentioned before kind of open floor being this you know compromise if you will, and, and maybe. You know, if you had to, if you, if you, if you hadn't found yourself in that environment, you know, you might have designed the, you know, people might of designed the, you know, that control protocol differently. But, now we are there, right? So like what would you like, I mean, is it going to be open flow, or is, is open flow going to be, you know. Is, is this the end of the line for open flow, are we going to see, something totally new as a control channel there, or? >> So, we'll have to see. You know, we spent a lot of time, internally, building up open flow tools, and so we have this tool, loxy, which will, given a high level description of open flow. Produce the Java bindings for it, the C binding for it, the Python bindings, the Wireshark dissector, and that type of stuff is useful. >> Yeah, I gave that a little bit of air time in the course, actually. I spoke about it briefly. >> At it's heart, open flow is basically a TLV based language that's fairly extensible. And, you know, we've actually been able to continue to use Open Flow for even our commercial products, and with some extensions on top. But, you know, through the standard extension mechanisms. They're all out of valid extensions if you will. And so from that stand point it's actually, been, you know, it's still continues to have legs. Other end. And this is, the part where I talk about fairly often professionally, so it's something [UNKNOWN] it sounds like a soap box. The open floor go to market model, which is you know, the way that you the customer might build a network, has in mind been fairly broken and so the idea that you would buy a switch, that had. Open closed software all on it already. You'd buy it from company A and you'd buy a controller from company B and an application from company C. These things would all kind of magically work together. I don't know that, that will ever happen and so what we have is, and this, you know, I'm espousing this, you know, [INAUDIBLE] my commercial hat on and not. We have a model where you buy the controller and the application from us and the switch will actually netboot the operating system from the controller. So, you're actually getting the hardware from a completely different company, it has no software on it, that's the bare middle vision. And then you get all the software from one company. >> Hm, hm. >> And then ,there are, at least, you're guaranteed the thing will work. You know, there's open A.P.I.s into the top of the solution, but into the intermediary parts, that is, you know, from the switch to the controller, and from the controller to the application. It's not, at least right now, a user-facing thing. >> Right, so there's almost, I mean there's no need to inter operate at the moment. >> Exactly. And you know, But this, what, note, and this is maybe a bit of a subtle but it's a critical point. There's no vendor lock in between the hardware and the software. So, if you decide that the software you get from company A is crap, you can put the software from company B on it. Or, if you decide software from company A is great but you want the next generation of hardware, as long as that's on the qualified list, you can do that. And so you actually have decoupling there. Other people are calling it the disaggregation model. >> Yeah, I guess that will keep both parties on their toes, as, as you've alluded to. Right? >> Yeah. Well it also, it's something that I, I take as a truism that you know, people could argue with or not, but I think in the history of tech there's great hardware companies, and there's great software companies, but there's extremely rarely a great company that does both hardware and software. >> [LAUGH] Yes >> But, I mean they're just fundamentally different mindsets and it's very hard to build up the expertise to do both. >> Absolutely. So what do you think? I mean, we, we, we talked about I mean, at least in our discussion, you brought up a few interesting kind of open problems on with respect to Flow Visor. But with regards to, to bare metal switching, and this is something we're just starting to hear about now, like when, when I do this course in a year. You know as, as students start to think about, like grad students start to think about things to work on, do you think that there are you know, interesting problems to pay attention to now? >> So I will pitch to you the hardest problem that I know of, that I have decided not to bother solving. And then, you can take it as you will, whether it should be solved or not. And, independently, if you think it has a solution. So, a compiler has a relatively straightforward job, which is to say, if I'm trying to add 2 numbers together,. The compiler doesn't have to decide, okay does this turn complete chip support addition? You know it always supports addition, and then it's just a register scheduling problem. It's a memory, you know, when do I load, when I do loads and stores and that type of thing, right? The question is, can you build a compiler for A6? And this is a completely different problem because you, then you have to solve the problem of, if I were to do something as simple as addition, maybe ASIC A supports it and ASIC B doesn't. And inherent to being an ASIC, you, I mean, if it doesn't support a given operation, ASICs are not trying to complete. You can't emulate it in hardware. You can't fake it somewhere. Either it goes in the register that's dedicated for that purpose, or it just won't do it. >> Or it chokes basically, yeah. >> Yeah, and so, open flow at a high level, you know it was trying to present this kind of unified abstraction of decouple, like abstract away the hardware as well the decouple the the hard, the data plane from the control plane. And what I'm finding is that it's basically technically infeasible to build a single open flow agent that is all things to all possible applications. >> Hm, hm. >> Because you know, exactly you can't generalize things that are so specific as an async. Stuff. >> And, and maintain wire speed [CROSSTALK]. >> Some might, yeah I mean some might claim that at the async level like the, the, the set of operations that you need is limited enough that you might be able to just sort of you know. You might not have to deal with unknowns I guess because, because You know, there's just a [UNKNOWN] unlimited instruction set that, that could ever exist there. I don't know if that's true or not. >> What's interesting is, between vendors and even within vendors and different chips in the same product family, you have vastly different ways that this, the hardware is laid out. And so things that are possible in, in one are not possible in others or if they are possible you have to do it a completely different way. I's very much like assembly programming. >> Right. >> And what we're finding is take all the chips from one vendor and one product family right. So this is chips made by the same people for roughly the same purpose. There's, actually, even then lots and lots of differences. So, these companies will actually produce their own SDKs which are the best guess at the extraction model from the people who designed the chips for this purpose, and even then we're finding that we have to bypass this extraction, not a lot, but certainly not infrequently So, even the best extraction created by the people who made the chips. you know, it's still imperfect, and so, to, to claim that you could do better than that, across product families, across chip vendors, its just a problem that I'm not, don't think has a good solution now and, certainly, from a commercial perspective, I've stopped trying to solve that problem. >> Yeah. >> Yeah, what we do right now is we have open floor agents, where we send an application message down that says reconfigure yourself for this application. And you know what it means to configure yourself for the application is something that is just completely, not a real standard, and basically can't be standard. And, I don't know of a better way to solve that problem. >> It may just be that, that you're going to [UNKNOWN] something specialized for ACIS or at least for families, or something. >> And, in general, the people think of this as a horrible thing, and I actually don't think it is. Right I mean even at the highest level of our applications you know, we need to understand what the, the, this information is. We you know, software is, has all sorts of different levels of abstraction and trying to standardize software layers is an inherently difficult and messy and frankly unproductive thing to do. And so exposing the low level hardware, and letting people figure it out for their application what is the right abstraction, I think makes a lot more sense. >> Yeah, yeah. Cool, I guess I guess we'll, we'll wait and see. Like maybe this time next year, we'll have more answers. >> Yeah. Well, and certainly this time next year, we'll have the thing that I'm most excited about, is more and more chip vendors are starting to publish. Not the low level hardware details, but actually open APIs into like a closed source SDK. >> Uh-huh. Yeah. >> And so people are getting more and more familiar with what the actual hardware limitations are. And that's forcing a, a more productive conversation. Another thing that I would point you to is, Broad Comm's OFDPA. It's a binary SDK on top of some of their chips. That you can actually, the source code for the SDK is not open but the API is, so you can at least see what's possible. >> Cool cool, interesting. >> Subset of what's possible, and I can send you these things. >> Excellent yeah no it'd be good to get the pointers. I'd definitely be interested in seeing those. Well. awesome. I guess we probably taken up enough of your time, its for one day. This is fun so. >> Word Yeah. >> We should definitely do it, do it again sometime. Especially with the speed at which these things are going. >> Yeah, definitely. >> Lots to talk about. So, cool thanks. >> Alright, well it was great chatting with you and thanks for having me. >> Thanks a lot.
