What is 800G Ethernet?

800G Ethernet is a high-speed Ethernet technology used for data transmission and communication networks that provides a data transmission rate of 800 Gigabits per second (800Gbps).

Twice as fast as the previous generation of 400G Ethernet, 800G Ethernet offers greater bandwidth and is primarily used in large data centres, cloud environments and high-bandwidth applications. It can provide higher rates, greater throughput and better network performance for these scenarios, supporting faster and more efficient data communications.

800G Ethernet uses 8 channels with a transfer rate of 100Gbps per channel. This doubles the speed of PAM4 from 50Gbps in the previous generation to 100Gbps.

The 800GbE specification introduces a new Media Access Control (MAC) and Physical Coding Sublayer (PCS). It essentially reuses two sets of existing 400GbE logic from the IEEE 802.3bs standard with some modifications to distribute data across eight physical 106Gb/s channels. Because the PCS is reused, the standard RS(544, 514) forward error correction is retained for easy compatibility with existing physical layer specifications.

Implementing 800G Ethernet typically requires advanced network hardware and is typically designed with low power consumption for improved energy efficiency.

The Institute of Electrical and Electronics Engineers (IEEE) is responsible for 800G Ethernet standardisation, which helps ensure interoperability between equipment from different vendors.

Why do We Need 800G Ethernet?

The most direct answer is to cope with the explosion of data. When it comes to training large-scale AI models, public data shows that from GPT-1 to GPT-4, the number of model parameters has grown from 110 million to 500 billion, and may even exceed the trillions. According to research firm TrendForce, the GPT-3.5 large model requires 20,000 GPUs to process the training data using the processing power of NVIDIA’s A100 graphics card.

Therefore, in the supercomputing centre of the arithmetic cluster, advanced chips and advanced arithmetic are not equal, arithmetic chips only provide arithmetic, advanced arithmetic is actually following the “barrel effect”, arithmetic, storage and network transmission of the three core links, a short board, the performance of the entire system will appear a huge decline. This is why cloud service providers are actively deploying 800G Ethernet.

800 Gb/s Ethernet Specification

Architectural Overview

The 800 Gb/s Ethernet technology is designed as an interface using eight 106 Gb/s lanes with a 2xClause 119 PCS (400G) to connect a single MAC operating at 800 Gb/s. The figure below illustrates the high-level architecture. It is possible to form an 800G interface using two 400G PMDs, such as 2x400GBASE-DR4 modules, although skew management would be necessary to stay within the specification. This architecture could also support slower interfaces, such as 8×106.25G or even slower options, but the main focus is on the use of 8×106.25G.

Leverage Existing Standards

The capability of 800 Gb/s can be achieved by utilizing two 400 Gb/s Physical Coding Sublayers (PCSs) with integrated Forward Error Correction (FEC) and supporting eight lanes of 106.25G each. The IEEE 802.3 standard for 400 Gb/s employs multi-lane distribution (MLD) to allocate data from a single Media Access Control (MAC) channel to 16 PCS lanes. In this 800G standard, an MAC scaled up to 800 Gb/s, along with two modified 400 Gb/s PCSs, will be used to drive 8x100G lanes. This will result in a total of 32 PCS lanes (2×16 from the 400G standard), all equipped with RS(544,514) FEC, as established in the 400G standard.

A crucial aspect of the MLD striping technique is the implementation of a unique alignment marker (AM) for each virtual lane. For 400 Gb/s, AMs are inserted into the striped data stream every 163,840 x 257b blocks. This practice will continue at 800 Gb/s with the same spacing per 400G stream; however, there will be twice as many AMs inserted, and the AMs will require modifications to ensure both a coherent 800 Gb/s stream and to prevent a misconfigured 400 Gb/s port from synchronizing with the 800 Gb/s stream. The 802.3ck standard is employed for the Chip-to-Module (C2M) and Chip-to-Chip (C2C) interfaces, operating at 106.25G per lane.

800G Ethernet Network Timeline

The development of 800G Ethernet builds on the foundation of the previous generation, 400G Ethernet. In recent years, standards organisations such as IEEE (Institute of Electrical and Electronics Engineers) and OIF (Optical Internetworking Forum) have developed standards for 400G networks, laying the foundation for the development of 800G.

2022: First 51.2T switch chip released

In 2022, the networking industry reaches a major milestone with the release of the first 51.2T switch chip. These switch chips will support 64 ports at 800Gb/s, marking the development of 800G Ethernet into actual hardware. At the same time, verification of the first batch of 800G optical modules begins during this period.

2023: Standards release and development verification

In 2023, the standards organisations made significant progress. First, IEEE released the first version of the IEEE 802.3df standard, which defines the physical layer specifications for 800G Ethernet. At the same time, OIF also released the 224 Gb/s standard, which provides guidelines for building 112 Gb/s and 224 Gb/s channels for 800G and 1.6T systems.

Physical layer standards to be finalised in 2024-2026

Over the next two years, standards organisations are expected to continue their efforts to finalise the physical layer standards for 800G Ethernet. This will involve further refinement and testing of the specifications to ensure interoperability and performance of network equipment.

Multiple Application Scenarios of 800G Ethernet

Data Center

• AI Data Center: Artificial intelligence training demands substantial data transmission and computing power. High-speed Ethernet technologies can connect GPU clusters and data storage, supporting the training of deep learning models and AI inference.

• Ultra-High-Density Data Storage: Data centers need large storage capacities and rapid data transmission to address growing data demands. 800G Ethernet can connect storage servers, enabling ultra-high-density data storage. For instance, a large social media company can leverage these technologies to manage the vast number of photos and videos uploaded by users.

• Virtualization and Containerization:Technologies like virtualization and containerization require swift data transmission to share resources between different virtual machines or containers. 800G Ethernet can provide high-bandwidth for virtual machine migration and container communication. For example, a cloud service provider can use these technologies to support customers’ virtualized workloads.

Cloud Computing

• Elastic Computing Resources: Cloud computing offers the ability to elastically compute resources, necessitating high-speed network connections. 800G Ethernet can facilitate rapid data transmission between cloud computing users. For example, a scientific research institution can utilize these connections to run complex simulations and data analysis tasks in the cloud.

• Cloud Storage and Backup: Cloud storage and backup services require significant capacity and high-speed transmission to ensure data security and availability. High-speed Ethernet technologies can connect cloud storage devices and data backup servers. For example, enterprises can use them to back up critical business data.

High Performance Computing

High-performance computing applications, like scientific computing and computational model training, require high-speed data transmission and processing capabilities. 800G networks enhance data transmission speeds and network performance to execute high-performance computing tasks. This is crucial for applications in scientific research, big data analysis, and intelligent computing training, which involve processing complex calculations. The introduction of 800G Ethernet will further drive innovation and development in high-performance computing.

Big Data

• Data Transfer and Analysis:Big data analysis demands extensive data transmission and processing power. 800G Ethernet can transfer large-scale data sets from sources to analysis platforms, accelerating data processing. For instance, a healthcare organization can use these networks to analyze vast amounts of patient medical records, improving diagnosis and treatment.

• Real-Time Data Streaming: Real-time data streaming necessitates extremely low latency in data transmission. High-speed Ethernet technologies can support real-time data stream applications, such as financial transaction monitoring and smart city surveillance. For example, a financial institution can use them to monitor and analyze extensive transaction data to detect potential fraud.

Internet of Things

800G Ethernet can connect more IoT devices and sensors, enabling large-scale device interconnection. IoT solutions will have higher scalability and capabilities, fostering innovative applications in smart cities, smart transportation, smart manufacturing, and other fields.

Autopilot

• High-Definition Maps and Sensor Data: Autonomous vehicles need high-resolution maps and sensor data for accurate positioning and environmental perception. 800G Ethernet technologies will transmit these large-scale data, enhancing the safety and reliability of autonomous driving.

• Vehicle Communications: Communication between vehicles and with infrastructure is key for autonomous driving. High-speed Ethernet will support real-time communication between vehicles, helping to prevent collisions and improve traffic efficiency.

In summary, 800G networks play a pivotal role in promoting innovative applications, accelerating digital transformation, and advancing technological progress.

Introducing the Asterfusion 800G Ultra Ethernet Switch

Introducing the Asterfusion 800G Ultra Ethernet Switch, the pinnacle of speed and efficiency! Powered by the cutting-edge Marvell Teralynx 10 51.2 switching chip, this switch delivers lightning-fast performance with port-to-port latency under 560ns for 800GE ports. Enjoy the best lead time and price available, all while harnessing the power of the market-leading SONiC enterprise Distribution, AsterNOS. Here are its standout hardware and software features:

Highlights

• 51.2T switch with 64x800G OSFP or 128x400G/512x100G in 2RU
• World’s fastest switch with port-to-port latency under 560ns fo800GE ports
• Max TDP of 2200W with 64x800G SR8 ports under full traffic load
• Large on chip buffer of 200+MB for better ROCE (RDMA over Coonverged Ethernet) performance
• 10ns PTP and SyncE performance supports strictly synchronized AI parallel computing
• Advanced INT (in-band network telemetry) for packet delays, dropsand path traversal, enabling more advanced congestion control algorithms
• Open AsterNOS based on SONIC with best SAI support, more robust aand reliable.
• Compatible with heterogeneous GPUs and SmartNICs from all leading vendors
• Line-rate programmability to support evolving UEC (Ultra Ethernet Consortium) standards

Asterfusion 800G Ethernet Switch Hardware Highlight:

• OSFP800 switch ports, each supporting 1 x 800 GbE (100G PAM4), or via breakout cables 2 x 400G GbE,4x 200 GbE, or 8 x 100 GbE.
• 512 x 112G Long Reach (LR) best-in-class SerDes, supporting 64x800G, 128x400G, or 512x100G interfaces via optical modules/splitters
• Up to 14.4Bpps of L2/L3 full wire-speed forwarding
• Load-aware per flowlet/packet balancing to prevent congestion and ensure efficient utilization of available bandwidth
• Extra two 10G SFP+ ports for network telemetry management on front panel
• Up to 24 W power budget per OSFP800 port.
• Incorporates Marvell Teralynx10 switch series silicon.
  • Offering the industry’s lowest port-to-port latency under 560ns for 800GE ports
  • Proven, robust 112G Serdes, the industry’s lowest bit error rate (BER).
  • Comprehensive data center feature set: including IP forwarding, tunneling, rich QoS and robust RDMA.
  • Programmable forwarding: offers permutable flex-forwarding to enable operators to program new packet forwarding protocols as network requirements evolve, with no compromise to throughput, latency or power.
  • Extensive real-time network telemetry, including P4 in-band network telemetry (INT).
  • Advanced shared buffering: 200+ MB of on-chip buffer, dynamically shared by every port, ensures superior network quality with fewer packet drops
• Intel Xeon 8-core CPU for line-rate programmability, enabling future networking protocols through software upgrades
• BMC module with serial-over-LAN support
• Supports hot/cold aisles with front-to-back airflow
• 2 RU form factor
• Hot-swappable, load-sharing, redundant 3200W AC PSUs.
• 3+1 hot-swappable fans
• Hardware switch pre-loaded with Open Network Install Environment (ONIE)
• Hardware switch pre-load with Asterfusion Enterprise SONiC Distribution (AsterNOS)

Asterfusion Enterprise Distribution of SONiC- AsterNOS

Imagine a world where network infrastructure isn’t a roadblock, but a superhighway. That’s the world Asterfusion is building, one network switch at a time. While other white box vendors leave you to cobble together solutions, Asterfusion pre-installs its enterprise Distribution of SONiC, giving you a one-stop, turnkey solution that just works.

We didn’t get here overnight. Since 2017, our dedicated team of over 100 SONiC R&D experts has been laser-focused on a single mission: building the world’s best SONiC enterprise network operating system (NOS). The result is AsterNOS, a powerhouse of an OS, custom-built for our own range of campus and data center switches, from 1G all the way up to 800G.

But compatibility is key. That’s why AsterNOS plays nice with all major industry chips. And we’re not just talking bare-minimum compatibility – our commercial NOS outshines the community version in both feature development and rock-solid stability. Plus, our top-tier support team is always ready to jump in and keep your network humming.

The proof is in the pudding. Over the past 7 years, Asterfusion’s SONiC enterprise solution has been battle-tested in the trenches of public clouds, telecom operators, massive internet companies, private clouds, and enterprise networks. We’ve learned a thing or two about what it takes to keep modern networks running at lightspeed.

Software Highlights

• Pre-installed AsterNOS is an enterprise version of SONiC with SAI as the kernel
• Integrated rich L2/L3 network features, complete support for network virtualization, QoS policies and other services
• Advanced functional containerized & event-driven system architecture to accelerate network service development/customization
• Provides an open REST API interface that can be centrally managed and invoked by third-party applications such as cloud management platform
• Provides KLISH command line for traditional network engineers

Speaking of modern AIDC, AsterNOS is built from the ground up to handle the demanding workloads of tomorrow, like generative AI. Our streamlined design simplifies the management and monitoring of even the most powerful Ethernet infrastructures, so you can focus on what matters most – delivering incredible experiences to your users. And with adaptive routing and RoCE congestion control, Asterfusion SONiC optimizes traffic flow like a finely-tuned orchestra, ensuring your workloads perform at their absolute best.

Ultra Low Latency Network

• Unmatched Speed: Featuring Marvell Teralynx10 silicon, the world’s fastest switch, with port-to-port latency under 560ns for 800GE ports. Perfect for latency-sensitive applications like AI/ML, HPC, and NVME.
• Enhanced Performance: RDMA enables direct memory access, improving latency performance to microsecond-level.

Lossless Network

• Zero packet loss: ROCEv2 ensures microsecond-level low latency, high throughput, and near-zero packet loss, ushering in an era of AI-driven network performance and reliability.
• Advanced Congestion Control: Benefit from rich QoS features including ECN, PFC, DCBX, QCN, DCQCN, and DCTCP for large-scale RDMA deployments.
• Intelligent Network Telemetry (INT): Monitor packet delays, drops, and path traversal for advanced congestion control algorithms.

High Reliability Network

• Robust Load Balancing and Redundancy: Up to 8192-way equal-cost multi-path (ECMP) routing.
• Seamless Connectivity: BGP multi-homing for multiple server connections with automatic load balancing and failover.
• Active/Active Multipathing: Multi-chassis link aggregation group (MC-LAG) for superior L2 multipathing.
• Rapid Failover: BFD for BGP and OSPF in just 50ms.

Time-Sensitive Network

• Precise Synchronization: Achieve 10ns PTP and SyncE performance, essential for synchronized AI parallel computing.

Automated O&M Network

• Effortless Operation: Integrated with Python and Ansible to support automated operations and maintenance.
• Zero-Touch Provisioning (ZTP): Automatically obtain and load deployment files, simplifying device setup.

Open Network

• Open Enterprise SONiC Distribution: AsterNOS provides the best SAI support, ensuring robust and reliable performance.
• Future-Proof: Line-rate programmability to support evolving UEC (Ultra Ethernet Consortium) standards.

Asterfusion 800G AI Network Solution

RoCEv2 AI Solution with NVIDIA DGX SuperPOD

Experience the future of networking with the Asterfusion 800G Ultra Ethernet Switch – where speed meets reliability and innovation.

Reference:

https://ethernettechnologyconsortium.org/wp-content/uploads/2021/10/Ethernet-Technology-Consortium_800G-Specification_r1.1.pdf