Co-packaged optics for hyperscale data centres
The continuing push for greater efficiency, smaller form factors and faster throughput is leading to the development of next-generation co-packaged optics for datacentres and related applications. EPIC explores continuing research from major manufacturers working to keep the data moving while the world’s appetite from broadband grows with each passing day.
BY IVAN NIKITSKIY, PROGRAM MANAGER, PHOTONICS TECHNOLOGIES, EUROPEAN PHOTONICS INDUSTRY CONSORTIUM (EPIC)
With data centre traffic growing at an unprecedented pace, fuelled by advances in AI and Machine Learning, networking infrastructure must scale in capacity while maintaining or even reducing its total power consumption and footprint. How is the industry going to move forward to hyperscale data center operations with the introduction of next-generation / second- generation Co-Packaged Optics (CPO)? Here we overview how the large industry players, members of EPIC – the European Photonics Industry Consortium – address user requirements for CPO from different perspectives.
HiSilicon (a Huawei company), China
Huawei is a major player in co-package optics, and their Advanced Photonic section develops the photonic optical components that go into Huawei’s larger systems. They are currently addressing the challenges of developing 100 Tbit/s co-packaged optics. Eric Bernier, Leader of Advanced Photonics at HiSilicon Technologies in Canada, the former ASIC Design Center of Huawei explains: the general consensus within the industry is that at the bandwidth required by a 100 Tbit/s switch, it becomes impossible to move the data electronically without consuming the entire power budget for the switch chip. As a consequence, at 100 Tbit/s, co-packaged optics are essential. But from a module perspective, it’s not possible to double the number of modules because of reliability issues and also because it becomes harder to package. As a result, the solution for achieving the required higher density will be to increase the capacity of the co-packaged optics to around 200 Gbit/s per fiber coming out of each module with multiple wavelengths per fiber. This will require inputting more optical power into the system, and although a lot of progress has been made, Bernier believes that ultimately, they will need new technology.
For this reason, they are presently engaged on two research projects initiated by The International Photonics Advocacy Coalition (IPAC). One aims to develop a standard form factor for the external laser source, and the other is looking at the issues, the system architectures and the evolution of the electronics that are limiting 100 Tbit/s co-packaged optics. Currently, it is generally assumed that any future device will incorporate a connector because it will be easier for the supply and assembly chains. However, if the aim is to increase density while simultaneously reducing the power requirement on the staircase, the connector may have to be eliminated.
Senko Advanced Components Inc., Japan:
Tiger Ninomiya, Senior Technologist at Senko Advanced Components in the US, identifies four main challenges for CPO connectors in a data centre switch applications: 1) an increase in fiber count and how to arrange the fibers in and out; 2) a use of external laser sources; 3) a change in face plate density that requires reserved spaces for laser sources and TRx channels, and 4) the challenge of internal fiber routing as fibers are now inside the system.
As regards to fibre count increase, the 12.8 terabit switch typically has 32 ports with eight fibers per transfer module that adds up to a total of 256 fibers in the case of using parallel optics. Similarly, the 51.2 terabits switch with CPO has 16 modules embedded on the switch ASIC substrate. And with parallel optics, each one of CPO optical engines can have up to 64 fibers, which adds up to a total of 1,024 fibres just for TRx – 4 times what they are dealing with now.
The issue with faceplate density derives from the need to have more fibers and to find space for external laser sources. The MPO connector has better density over duplex types of connectors. However, there is a correlation between fiber count per connector and optical performance. With a larger fiber count, it becomes challenging to maintain the lower loss, especially having multiple rows of multi fibers such as MPO-24 and MPO-32.
Senko is addressing this issue with their SN-MT connector carrying 16 fibers per connector, which improves fiber density at the panel while maintaining lower loss. Compared with MPO connectors, the SN-MT is roughly half the size and has a 2.7x density increase compared with MPO-16F. SN-MT even provides a better density than MPO-24 and MPO-32, while using a 1-row type ferrule, the optical performance is comparable to 1-row MPO. Senko also uses other technologies to overcome face plate density issues. These include fiber routing options for mid-board connectors; a fibre routing shuffle box, and backplane connectors.
The Consortium of On-Board Optics (COBO) and Co-Packaged Optics Working Group, which Tiger is chairing, aims to provide technical guidance and standards for CPO implementations focusing on optical connectivity and remote laser sources. In July 2022 they released a white paper on optical connectivity that details how these technologies can be utilized. [1]
Figure 1. Evolution of fiber density, insertion losses, and fiber count per connector by Senko Advanced Components Inc.
OFS Optics (a Furukawa company), USA:
OFS Optics has been involved across many different specialty fibre applications and markets for the better part of 30 years. Specifically in the telecom and datacom space, they work mainly with OEMs supplying erbium doped fibres, polarization maintaining fibres, and low bend loss fibres. Recently, OFS has been developing new novel fibre types such as hollow core and multi-core fibres. They are also developing external laser source
(ELS) modules for co-packaged optics.
For John Earnhardt, Director of Sales at OFS Speciality, the transition of going from copper dominant to fiber dominant has created a number of challenges. From a fiber optic perspective, with both PM and single mode fibres, there’s the issue of mode field diameter. In some cases, users want a conventional nine micron type mode field diameter and in other cases they want atypical mode field sizes that interface directly with the chip – options might be in a three micron mode field and a couple of options in between three and nine micron.
Low bend loss is becoming increasingly important as transceiver modules shrink in size and go to QSFPDD and OSFP and there will likely be pressure on low bend loss as well in the CPO area. With PM fibres, there are potential issues regarding traditional PM properties like beat length and PER and so on. Another area of concern is multi-path interference, particularly with the move to low bend loss fibres together with potential mis-match the mode field diameters when splicing together two fibres.
In addition to optical considerations, there are mechanical ones. In this regard, they are seeing increasing pressure to improve their tolerances, in relation, for example to core clad offset, core diameter and clad diameter. The mechanical properties directly impact the overall efficiency of laser delivery through fibres. As regards the question of clad diameter i.e., 80 micron or
125 micron, their initial ELS samples were based on a 125 cladding diameter, but they are seeing pressure for more 80 micron fibres, both PM and SMF for transceivers and other types of applications. OFS is also seeing two emerging trends in fiber coating. One is a smaller diameter, for example, moving from 165 to 135 micron for 80 micron cladding. The other is the increasing demand for alternative coatings that will survive at least for a short-term duration at higher temperatures with solder reflow, and possibly for a longer term duration in some photonic packages with local hot spots.
Figure 2. Evolution of switch and transceiver bandwidths. Adapted from [2]
Fraunhofer IZM (Institute for Reliability and Microintegration), Germany:
In the last decade, Fraunhofer IZM have been innovating in the area of computing and data center applications using photonic interconnects. They contribute in the areas of system concept and design, photonic- and RF- component design, signal integrity & board design, silicon photonics interposer, developed with through silicon wire technologies, 3D integration, flip chip assembly, co-packaging, system evaluation and benchmarking.
In the field of CPO they started with a flagship project for data centre interconnects with the goal to utilize optical interconnections and 3D integration technologies to make data centres and high-performance computers faster on all levels: rack-to-rack, board-to-board and chip-to-chip. More recently, they have been involved in the MASSTART project, which was set up to facilitate the high volume manufacturing of Tb/s inter and intra data center transceivers.
For Tolga Tekin, Group Manager at Fraunhofer IZM, a major challenge for data center network topology is that more than 70% of the traffic stays inside the data center. This means that the interfaces need to have enough capability and symmetry to connect all of the nodes. The transceiver data rate follows the Ethernet switch port speed. The serializer/deserializer (SerDes) speed is defining the port speed of the transceiver. The packaging constraints limit chip radix to 256 (512) ports/ASIC. Even though the SerDes arrays are constantly evolving to support higher bitrate, the power consumption of SerDes increases with bitrate.
The acceptance of the solutions depends on their costs. For the typical single mode data center transceiver, the target is $1 per Gbps, so the cost of the optics for a transceiver with 32 ports on the switch front panel is $10-13k, making the transceiver cost for the entire data center around $50 million. Accordingly, Fraunhofer is developing a cost cutting strategy based on decoupling the I/O from the logic system. The effect is to reduce the power consumption of chip I/O from 180 watts for their 25 terabit switch ASIC to 40 watts. At the same time, they have been able to increase the link reach thereby reducing loss from 12 dB to 1 dB. Recently completed EU-project L3MATRIX has been devoted to large-scale silicon photonics matrix for low power and low cost data centres.
This project aimed to improve the underlying network technology with photonic switching to enable scale performance whilst keeping power consumption under control. In the co-packaged optics, a fiber array is coupled directly using micro lens arrays combined with integrated III-V and silicon photonics and directly attached to the switch ASIC. They have focused on 25 terabit switches with co-packaged optics using a 2D transceiver array based on silicon photonics with integrated III-V materials. These devices address up to 256 lanes and use integrated lasers by bonding III-V layers on silicon photonics in Mach-Zehnder configuration. Altogether, the future of Co-Packaged Optics is about creating a new ecosystem, which is going to involve efforts from component manufactures and higher level system integrators to create a network, involving everyone in the industry, to develop the right standards and to design and integrate these new components and to scale production.
Reference
[1] https://www.onboardoptics.org/_files/ugd/8abe6c_708f0fd7e75c4415b5eca3e35f7d1116.pdf
[2] A. Ghiasi, “Large data centers interconnect bottlenecks,” Opt. Express, no. 23, pp. 2085-2090, 2015.
