Standards pave the way to a future beyond 400G Ethernet

Standfirst: The ongoing development of advanced photonic data communications solutions is vital to future commercial and consumer applications that access, analyze and otherwise leverage data. IEEE standards are the framework around which future photonic integrated circuits (PICs) will be created. IEEE’s Ethernet Task Force offers insights into how new standards are coalescing to ensure support for future requirements.

By John D’Ambrosia, IEEE 802.3 Task Force Chair and EPIC advisor

In April 2020, the IEEE 802.3 Ethernet Working Group released its 2020 Ethernet Bandwidth Assessment. The findings of this extensive 18-month effort indicated that by 2025, for a broad diversity of applications explored, traffic levels would grow from 2.3x to 55.4x the levels observed for 2017. Based on these findings, the IEEE 802.3 Ethernet Working Group launched its Beyond 400 Gigabit Ethernet Study Group in January of 2021. This study group has recommended the formation of the IEEE P802.3df 200 Gb/s, 400 Gb/s, 800 Gb/s, and 1.6 Tb/s Ethernet project, which will be considered at the IEEE 802 November 2021 Plenary for approval.

The proposed project reflects the observed needs of the industry throughout this decade and well into the next. Upon approval, it would be one of the largest projects in the recent history of the IEEE 802.3 Ethernet Working Group and reflects the desire to deliver new higher speed Ethernet solutions in a quicker manner to address the industry’s never-ending need for more bandwidth, as well as to provide lower power, higher density solutions for existing Ethernet rates.

The Relationship Between Signaling Rates and Ethernet Rates
Before diving into the IEEE P802.3df project, it would be useful to explore the relationship between the signaling rate of a “lane” and the Ethernet rate. As illustrated in Figure 1, for a given Ethernet rate there are a number of solutions that is simply the product of the number of lanes of the given solution and the signaling rate per lane. For example, for 100 GbE, many of the initial solutions were four lanes of 25 Gb/s for an aggregate 100 Gb/s throughput. As technology progressed, 100 GbE solutions based on 2 lanes of 50 Gb/s were introduced, and then finally 1 lane of 100 Gb/s.

Figure 1 – The Relationship Between Ethernet Rates & Signaling Rates

The general rule of thumb was that Ethernet networking solutions would begin to ship in high volume when a four-lane variant was introduced, with the highest volume ultimately being achieved when moving to a one-lane serial variant. A perfect example is the repeated use of the QSFP module, which has 4 electrical differential pair inputs and 4 electrical differential pair outputs to support four-lane variants of 40 GbE, 100 GbE, 200 GbE, and 400 GbE. The race to higher densities ultimately led to the introduction of form factors that were based on eight-lane solutions, which allowed the faceplate density for four-lane Ethernet variants to approximately double. OSFP and QSFP-DD are examples of these form factors. Furthermore, the initial 400 GbE architecture and many of the physical layer specifications was based on 8 lanes of 50 Gb/s. Market data forecasts highlighted growing market acceptance of eight-lane architectures, physical layer specifications, and module form factors.

Additionally, many ports today allow the user the flexibility to configure the port capacity. For example, a given eight-lane port could be configured to support one eight-lane implementation, two four-lane implementations, four two-lane implementations or even eight one-lane implementations. This provides both the system provider and end user the ability to use a single box in a variety of network topologies, which then enables leveraging economies of scale to drive costs down throughout the entire ecosystem.

The IEEE P802.3df Physical Layer Objectives
As noted, the IEEE P802.3df project, when approved, will be one of the largest projects in recent IEEE 802.3 Ethernet history. As highlighted in Table 1, the project will address specifications of at least 26 different interfaces and physical layer specifications.

Table 1 - Summary of IEEE P802.3df Objectives

There are several observations that can be made from this list of objectives:
• The project will address developing the higher speeds 800 GbE and 1.6 TbE. It will leverage the specifications developed for these speeds to define lower-power, higher-density solutions for 200 GbE and 400 GbE.
• The overall size of the project has already been highlighted. If one were to go back and visit the IEEE 802.3 Ethernet archives, these various objectives were addressed in several projects rather than a single effort. The urgency to do this in a single project points to two things. First, there is a need by the industry for all of the specifications that will be developed to support these various objectives. Second, there is a need to consider all these objectives simultaneously and employ a holistic approach to the development of an architecture that will enable development of equipment that can address all these various objectives and reduce the risk of future stranded ports on equipment that would be unable to support future interface specifications.
• As denoted by the coloring of cells within Table 1, there are several groupings of objectives that will leverage existing technology or specifications or will leverage new specifications defined in the P802.3df project. For example, the fields highlighted in yellow will leverage existing standards or standards in development, specifically IEEE Std 802.3cuTM-2020, IEEE P802.3ck, and IEEE P802.3db. The fields highlighted in purple are anticipated to leverage development of 200 Gb/s per lane for electrical signaling for specifications defining AUI interfaces and copper twin axial cabling to address all speeds.
• The fields highlighted in green are anticipated to leverage development of 200 Gb/s per lane optical signaling for specifications targeting single-mode fiber (SMF) with reaches of 500m and 2km. The two objectives in grey, targeting SMF reaches of 10 km might leverage 200 Gb/s per wavelength or coherent optical solutions, and will be one of the key debates of the future P802.3df Task Force.

Development of the IEEE 802.3df architecture
During the definition of the IEEE P802.3df project, it was recognized that a single project addressing the objectives noted in Table 1 would be advantageous to the Ethernet community. By considering all these objectives simultaneously, the future IEEE P802.3df Task Force will be able to address the architecture holistically when developing the standard. This will enable the development of equipment to support all of the noted physical layer specifications. If the development of these specifications was accomplished via multiple standards, the risk that a port might not be able to support all of the noted specifications would increase. This could result in a port being “stranded” from supporting specifications defined by later standards. This happened during the course of 100 GbE, as the IEEE Std 802.3ba-2010 that initially defined 100 GbE did not define forward error correction (FEC) for 25 Gb/s per lane optics. This became problematic for equipment defined to meet the initial 100 GbE standard when the later IEEE Std 802.3bmTM-2015 that defined RS (528, 514) KR4 FEC, which was introduced by IEEE Std 802.3bjTM-2014, necessary for 100GBASE-SR4 optics was ratified. These ports became stranded since they could not support these optical ports defined by 802.3bm, impacting the return on investment of this equipment for the component vendors, system vendors, and end-users.

Given the various physical layer specifications that the IEEE P802.3df project will define, there are several different forward error correction schemes that might be employed. Figure 2 illustrates a number of potential example FEC schemes that the IEEE P802.3df architecture may need to accommodate: 1) full end-to-end FEC; 2) full end-to-end FEC with a concatenated inner FEC; 3) partially segmented FEC; and 4) fully segmented FEC. The coding gain required by each FEC depends upon the characteristics of each link segment and the overall FEC scheme but must also be balanced with latency and power considerations.

Figure 2- The Impact of Forward Error Correction on the IEEE P802.3df Architecture

It is quite plausible that the IEEE P802.3df architecture will enable at least two of these FEC schemes. For example, the end-to-end FEC scheme illustrated in Option #1 of Figure 2 might be the best approach for a backplane or copper cable solution. Option #4 could provide the most flexibility and performance by allowing different FEC codes to be employed in the optical module to address different optical physical layer specifications. Option #2 could provide a balance between performance, latency, and complexity. It is anticipated that the discussions, analysis, and technical decisions that will accompany this topic will be a key part of early task force discussions.

The Promise of Ethernet Interoperability
As noted, the IEEE P802.3df project will address multiple Ethernet rates, signaling rates, and physical layer specifications. which will result in several interoperability challenges for the Task Force to consider.

At the time of writing this article the development of 100 Gb/s signaling to address electrical interfaces, backplanes, copper twin axial cabling, and multimode fiber physical layers is still underway within the IEEE 802.3 Ethernet Working Group. It is anticipated that these development efforts will be leveraged, via 8 x 100 Gb/s solutions, to address electrical interfaces and physical layer specifications that support 800 GbE. However, project objectives were adopted which will lead to the development of electrical interfaces and physical layer specifications based on 4 lanes of 200 Gb/s.

The future IEEE P802.3df Task Force will need to consider the future scenario where system boxes based on 8 x 100 Gb/s electrical interfaces will need to coexist with future system boxes based on 4 x 200 Gb/s. This will need to be considered during the development of the IEEE P802.3df architecture, including situations where the fully segmented FEC approach shown in Option #4 of Figure 2 will need to be supported.

As noted in Table 1, it is expected that this project will also leverage the anticipated development of 200 Gb/s signaling to create lower-power, higher-density solutions for 200 GbE and 400 GbE. The IEEE P802.3df Task Force will need to review existing 200 GbE and 400 GbE solutions and consider any potential interoperability issues that may arise.

Conclusions
Given the findings of the IEEE 802.3 2020 Bandwidth Assessment Report and the forecasted bandwidth growth by 2025, the effort to begin definition of 800 Gb/s and 1.6 Tb/s Ethernet to address the growing diverse bandwidth requirements throughout the industry needs to begin. Leveraging existing Ethernet standards and efforts to develop 100 Gb/s electrical and optical signaling will enable initial 800 GbE solutions. The anticipated development of 200 Gb/s electrical and optical signaling, however, will enable 800 GbE and 1.6TbE solutions, assuming a x8 architecture. Additionally, while a building block for these higher speeds, the development of 200 Gb/s signaling will also enable the next generation of 200 GbE and 400 GbE solutions.

While the development of these multiple interfaces and physical layer specifications will be challenging, it is the development of the IEEE P802.3df architecture and the one or more FEC schemes it may need to support that will be critical. It is anticipated that this is where the future IEEE P802.3df Task Force will spend much of its initial time.

At this time further information about the work of the Beyond 400 Gigabit Ethernet Study Group can be found at https://www.ieee802.org/3/B400G/index.html.