The Consortium for On-Board Optics (COBO) was formed three years ago to develop specifications to permit the use of board-mounted optical modules in the manufacturing of networking equipment, such as switches and even servers. The specifications will cover electrical interfaces, pin-outs, connectors, thermals, etc. The idea is to drive the development of interchangeable and interoperable optical modules that can be mounted onto motherboards and daughtercards. Microsoft is a founding member of COBO, as are Arista, Broadcom, Cisco, Juniper, and Mellanox. In April, COBO issued its first on-board optics specification targeting 400 Gbps and 800 Gbps data rates and leveraging two electrical interfaces: 8 and 16 lanes of 50-gigabit PAM-4 signals.
I caught up with Brad Booth, president of COBO, at the recent NetEvents 2018 in San Jose, California. In his day job, Brad is also a Network Architect with Microsoft Azure.
Jim Carroll, Optical Networks Daily – Hi Brad, good to see you again. Tell us about your work and the latest developments in the drive toward on-board optics.
Brad Booth, COBO: I am the president of COBO. That is my volunteer job. My full-time job is a Network Architect for Microsoft Azure.
A few months ago, COBO, the Consortium for Onboard Optics released our specification 1.0 specification in time for OFC and now we're going around letting people know about it because it is a game changer for how to use embedded optics. Traditionally, embedded optics have been proprietary solutions. We knew that as we progress the technology and move from 10G to 40G to 100G to 400G, and whatever speed comes next, the ability to continue to use faceplate optics is presenting greater complexities.
First, the thermal environment was getting a little harder to deal with. Signal integrity was becoming more difficult to deal with. The fact that you had people plugging these modules in, which were very sensitive to ASD and sensitive to noise, resulting in damage. And we decided that we needed to start the progression of moving the optics away from the faceplate and closer to the ASIC, if not on to the same footprint of the ASIC eventually. We knew that at 400G it was possible to do faceplate pluggable. We knew in time people would figure out how to do an optics embedded with ASICs, but what was going to be the thing that would allow us that transition in between. What would allow us to have the learnings and the understanding of how this will change our business models. That’s why we came up with COBO.
What is COBO implementation actually going to look like?
Brad Booth, COBO: This is a COBO module. It is a by 16 wide, which means it is a 16-lane interface on one side. This module was designed for the output of four 100G PSM4 signals This is the medium size version of the module. It comes in a by-8 and a by-16 version. It comes in three lengths: slightly shorter than this, this length, and slightly longer. That allows us to be able to use this module literally from multimode fiber all the way up to long-haul coherent transmission.
The module at its widest form factor is capable of accepting up to 40 watts of power to drive whatever component tree we need in it.
It's capable of having a full 1RU heatsink, or greater if we want. You can even do liquid cooling on this if you want. There's the capability of latching either on the high-speed pins or the low-speed pin connector on the bottom. We separated the low-speed power and ground and control away from the high speed to improve the signal integrity. The nice thing is these pins on the bottom are .8mm pitch, which means that we get lots of good contact and power. There is also .6mm pitch, which should be able to handle up to 100 Gbps PAM4 electrical signaling. This means that future iterations of this module will be capable of 1.6 Tbps worth of bandwidth.
What kind of support or feedback are you getting from the industry?
Brad Booth, COBO: We have over 60 member companies participating in COBO now. Many of them are very well known in the industry. Cisco, Arista, Juniper. Mellanox. We have connector companies involved too., like SEMTECH, TE Connectivity, Luxtera. We have companies that specialize in optical connectors, like SENKO, Huber+Suhner, etc. We have a strong ecosystem.
We have to figure out how to move those optics from the faceplate further inside the systems to improve the signal integrity, and we have to be able to do it without drawing massive amount of power. And that's one of the advantages of having so many people in the ecosystem that play in this area, we have a lot of good contributions.
Is COBO targeting 400G and above? Are higher rates going to be supported?
Brad Booth, COBO: This module is actually a four by 100. So it has 400 gigs worth of bandwidth in it right now. If you were to build it to handle 400G Ethernet, you could put two 400G Ethernets in it, and so that would be 800G worth of bandwidth. When this electrical interface goes to 100G PAM4 in the next version of our specification, this module will be able to do 1.6 terabytes.
Is that the endpoint? Probably not, because we've actually designed the module to be able to handle the coherent light, or the 400 ZR, which is being worked on by the OIF. That's a 15 watt part and it does 400G per lambda. If the power of that technology continues to drop, you could potentially put more of those in this and actually achieve even greater bandwidth.
What about the manufacturability of COBO? How is the industry progressing in this regard?
Brad Booth, COBO: The interesting aspect of building something like this is being able to dissipate the power and the heat. The biggest thing for heat dissipation is actually the maximum height that you can make the heat fin on the heat sink. So, if you get a full 1RU height, you get significantly better thermal performance than just a small one. We've done some general evaluations here. We’ve actually done some thermal modeling. As a next step, we're looking forward to building out true thermal systems and testing it.
One of the key aspects of this is that you've got a full 1 RU height. Second, you're not blocking the faceplate with optics, and you don't need as much airflow to pull through the system. That means you don't need fans running full out just to be able to pull enough air through to cool it. Another aspect is that COBO will allow you can to change how you implement your system design. You could place these modules in different spots across the motherboard that are potentially cooler or further away from the hotter ASIC. You could implement airflow channels to cool the optics, separate from the actual switch ASIC.
There is the possibility of using liquid cooling. This module doesn't have a heat sink on it right now, so you could drop a liquid cooling plate on top of it. There's also the expansion capability provided by extra pins on the bottom of the module, which could be used to provide more power and more control signals if necessary for some next-generation technologies.
What about the challenge of fiber alignment with COBO?
Brad Booth, COBO: Regarding fiber alignments, an interesting aspect is that we expect most people in the first generation to build these modules with pigtail optics. In other words, the optics will already be pre-attached. This module, which is just a mock-up, is shown without them.
We have work going on to actually do a connectorized interface. We have some people working on how that would actually be implemented. If we can connectorize it here and connectorized it on the faceplate, that means that you wouldn't have to worry about it at manufacturing. You could attach the module in the factory when they build the switch. They could put the optics right in. Then, after the optics and systems are all installed, you could come and hand fiber, trace it out to the pigtails or to the front faceplate and that would actually help eliminate the worry, if you have to replace this, that you have to unstrand the whole pigtail. This provides the option for people to be able to do both. We’ve discussed even short pigtail versions within our optical conductivity study group.
We invite people to contribute their thoughts and opinions. Come and work with us on this because this is an area that we think is going to become critically important as we start progressing towards actually putting optics embedded with ASICs.
http://onboardoptics.org/
I caught up with Brad Booth, president of COBO, at the recent NetEvents 2018 in San Jose, California. In his day job, Brad is also a Network Architect with Microsoft Azure.
Jim Carroll, Optical Networks Daily – Hi Brad, good to see you again. Tell us about your work and the latest developments in the drive toward on-board optics.
Brad Booth, COBO: I am the president of COBO. That is my volunteer job. My full-time job is a Network Architect for Microsoft Azure.
A few months ago, COBO, the Consortium for Onboard Optics released our specification 1.0 specification in time for OFC and now we're going around letting people know about it because it is a game changer for how to use embedded optics. Traditionally, embedded optics have been proprietary solutions. We knew that as we progress the technology and move from 10G to 40G to 100G to 400G, and whatever speed comes next, the ability to continue to use faceplate optics is presenting greater complexities.
First, the thermal environment was getting a little harder to deal with. Signal integrity was becoming more difficult to deal with. The fact that you had people plugging these modules in, which were very sensitive to ASD and sensitive to noise, resulting in damage. And we decided that we needed to start the progression of moving the optics away from the faceplate and closer to the ASIC, if not on to the same footprint of the ASIC eventually. We knew that at 400G it was possible to do faceplate pluggable. We knew in time people would figure out how to do an optics embedded with ASICs, but what was going to be the thing that would allow us that transition in between. What would allow us to have the learnings and the understanding of how this will change our business models. That’s why we came up with COBO.
What is COBO implementation actually going to look like?
Brad Booth, COBO: This is a COBO module. It is a by 16 wide, which means it is a 16-lane interface on one side. This module was designed for the output of four 100G PSM4 signals This is the medium size version of the module. It comes in a by-8 and a by-16 version. It comes in three lengths: slightly shorter than this, this length, and slightly longer. That allows us to be able to use this module literally from multimode fiber all the way up to long-haul coherent transmission.
The module at its widest form factor is capable of accepting up to 40 watts of power to drive whatever component tree we need in it.
It's capable of having a full 1RU heatsink, or greater if we want. You can even do liquid cooling on this if you want. There's the capability of latching either on the high-speed pins or the low-speed pin connector on the bottom. We separated the low-speed power and ground and control away from the high speed to improve the signal integrity. The nice thing is these pins on the bottom are .8mm pitch, which means that we get lots of good contact and power. There is also .6mm pitch, which should be able to handle up to 100 Gbps PAM4 electrical signaling. This means that future iterations of this module will be capable of 1.6 Tbps worth of bandwidth.
What kind of support or feedback are you getting from the industry?
Brad Booth, COBO: We have over 60 member companies participating in COBO now. Many of them are very well known in the industry. Cisco, Arista, Juniper. Mellanox. We have connector companies involved too., like SEMTECH, TE Connectivity, Luxtera. We have companies that specialize in optical connectors, like SENKO, Huber+Suhner, etc. We have a strong ecosystem.
We have to figure out how to move those optics from the faceplate further inside the systems to improve the signal integrity, and we have to be able to do it without drawing massive amount of power. And that's one of the advantages of having so many people in the ecosystem that play in this area, we have a lot of good contributions.
Filmed at NetEvents 2018 in San Jose, California
Is COBO targeting 400G and above? Are higher rates going to be supported?
Brad Booth, COBO: This module is actually a four by 100. So it has 400 gigs worth of bandwidth in it right now. If you were to build it to handle 400G Ethernet, you could put two 400G Ethernets in it, and so that would be 800G worth of bandwidth. When this electrical interface goes to 100G PAM4 in the next version of our specification, this module will be able to do 1.6 terabytes.
Is that the endpoint? Probably not, because we've actually designed the module to be able to handle the coherent light, or the 400 ZR, which is being worked on by the OIF. That's a 15 watt part and it does 400G per lambda. If the power of that technology continues to drop, you could potentially put more of those in this and actually achieve even greater bandwidth.
What about the manufacturability of COBO? How is the industry progressing in this regard?
Brad Booth, COBO: The interesting aspect of building something like this is being able to dissipate the power and the heat. The biggest thing for heat dissipation is actually the maximum height that you can make the heat fin on the heat sink. So, if you get a full 1RU height, you get significantly better thermal performance than just a small one. We've done some general evaluations here. We’ve actually done some thermal modeling. As a next step, we're looking forward to building out true thermal systems and testing it.
One of the key aspects of this is that you've got a full 1 RU height. Second, you're not blocking the faceplate with optics, and you don't need as much airflow to pull through the system. That means you don't need fans running full out just to be able to pull enough air through to cool it. Another aspect is that COBO will allow you can to change how you implement your system design. You could place these modules in different spots across the motherboard that are potentially cooler or further away from the hotter ASIC. You could implement airflow channels to cool the optics, separate from the actual switch ASIC.
There is the possibility of using liquid cooling. This module doesn't have a heat sink on it right now, so you could drop a liquid cooling plate on top of it. There's also the expansion capability provided by extra pins on the bottom of the module, which could be used to provide more power and more control signals if necessary for some next-generation technologies.
What about the challenge of fiber alignment with COBO?
Brad Booth, COBO: Regarding fiber alignments, an interesting aspect is that we expect most people in the first generation to build these modules with pigtail optics. In other words, the optics will already be pre-attached. This module, which is just a mock-up, is shown without them.
We have work going on to actually do a connectorized interface. We have some people working on how that would actually be implemented. If we can connectorize it here and connectorized it on the faceplate, that means that you wouldn't have to worry about it at manufacturing. You could attach the module in the factory when they build the switch. They could put the optics right in. Then, after the optics and systems are all installed, you could come and hand fiber, trace it out to the pigtails or to the front faceplate and that would actually help eliminate the worry, if you have to replace this, that you have to unstrand the whole pigtail. This provides the option for people to be able to do both. We’ve discussed even short pigtail versions within our optical conductivity study group.
We invite people to contribute their thoughts and opinions. Come and work with us on this because this is an area that we think is going to become critically important as we start progressing towards actually putting optics embedded with ASICs.
http://onboardoptics.org/