written by Asterfuison
Inband Network Telemetry (INT) is a network monitoring framework for collecting and reporting network status, which collected by a data forwarding plane without the intervention of control plane.
AI 、big-data based Internet applications drive rapid upgrading of Internet data center products and technologies. On one hand, the access bandwidth was upgraded from the traditional 10Gbps to 25Gbps/100Gbps,the basic network is required to provide high forwarding capability to ensure the high availability of services. On the other hand, with the widespread application of Remote Direct Memory Access (RDMA) lossless Ethernet technology, realize microsecond-level delay from computing node to storage node,which greatly optimizes forwarding performance of end-to-end service. And that also means higher challenges to network operation and maintenance–So, how to achieve more sophisticated traffic visualization and controllability in a large, complex HPC (high performance computing) network; how can we achieve end-to-end seconds- level failure positioning for service ?
This article introduces the In-band network telemetry technology , which provides new ideas for realizing the whole network traffic visualization.
Inband Network Telemetry (INT) is a network monitoring framework for collecting and reporting network status, which collected by a data forwarding plane without the intervention of control plane .In an INT-defined framework, the messages can carry INT instructions which can be interpreted by the intermediate forwarding device and filled in with the expected data, and then continue to be forwarded until the destination. INT could be able to observe flow pattern changes caused by microbursts, packet transmission delays, delays per node, and new ports in the flow path.
While the network interface rate develops from 1Gbps to 25Gbps100 Gbps, the increase of the cache capacity of the switch chip has not kept up, and the available cache time decreases instead.
Instantaneous many-to-one causes packet congestion on the egress. When the egress cache is used up, packets are discarded based on the tail discard mechanism . The application monitors packet loss and initiates TCP retransmission, causing the data end-to-end delay to worsen, seriously affecting the service experience.
In traditional networks, the SNMP protocol, which is initiated by external applications to request network status information. It can only passively wait for the timeout or the peer feedback to be received ,which feels like the delivery cannot be tracked.
However, for service failures caused by network packet loss, the network monitoring system is required to quickly locate which switch and port has lost packets due to insufficient cache. Meanwhile, when the critical application end-to-end delay exceeds expectations, it’s also necessary to locate the forwarding delay of each node on the traffic forwarding path.
In order to solve the problem of “invisibility” that cannot be solved by traditional network , the industry has widely introduced INT(Inband Network Telemetry). Compared to SNMP, It realizes the ability of network equipment to actively push status information, has stronger timeliness, and solves the problem of invisible forwarding path and forwarding delay.
INT technology has been developed for near 8 years, which involves many addition and deletion of technical details. It is currently dominated by the P4.org applications working group. Its main development process is as follows:
For more about P4 knowledge https://cloudswit.ch/blogs/what-you-should-know-about-p4-programming-language-p4-switch/
Generally speaking, an INT domain contains three main function nodes, INT Source, INT Sink and INT Transit Hop.
For the telemetry operater, the traffic that needs to be telemetry will add an INT header to the source node, which contains an instruction set (INT Instruction) indicating the collected information, thus becoming an INT message. When the INT Transit Hop node is installed, the collected information (INT Metadata) will be inserted into the INT message according to the instruction set, and finally all the INT information will be popped up on the INT Sink node and sent to the monitoring device.
For users, the INT processing of traffic is completely transparent and users cannot and do not need to perceive the process.
Let us review the process in detail:
The information defined by the INT protocol is mainly divided into the following categories:
The Asterfusion CX-N low latency switches based on Marvell Teralynx and X-T based P4 programmable switches based on Intel Tofino support INT(In-band Network Telemetry) which provides accurate and comprehensive real- time network telemetry information including port interfaces, packet latency, packet queue lengths, etc. Cloud data center network administrators can leverage INT information to optimize their business applications and network operations, helping to build efficient, intelligent and highly resilient data center networks.
For the next generation of data center networks for HPC services,INT-based technology enables end-to-end network traffic visualization ,which breaks the “network black box” and provide overall solutions and necessary technical support for refined network operation and maintenance.
Asterfusion Networks is the leading provider of open networking infrastructure solutions. We provide an open, disaggregated, and highly programmable network fabric for next generation data centers and campus with white-box switching.
For more information on Asterfusion’s products and solutions, please visit https://cloudswit.ch/. For sales inquiries, please send an email to [email protected]
Floor 4, Building A2, Shahutiandi, No.192 Tinglan Road, SIP
23F - E08, Dinghe Tower, Jintian Road, Futian, Shenzhen