Understanding Packet Deduplication in NPB Platforms
Running FusionNOS

As enterprise networks continue to grow in scale and complexity, organizations increasingly deploy network visibility solutions to monitor, analyze and secure network traffic. Network Packet Brokers (NPBs) play a critical role in these architectures by collecting traffic from multiple sources, filtering and distributing relevant traffic to various monitoring and security tools, such as IDS, NDR, NPM and traffic analysis platforms.

In large-scale network environments, the same packet may be captured at multiple points through TAPs, SPAN ports or monitoring interfaces. When these duplicated packet copies are forwarded by the NPB to backend analysis tools, they can introduce unnecessary processing overhead, consume additional bandwidth and storage resources, and reduce the efficiency of security inspection and traffic analytics systems.

However, not all visibility applications require duplicate traffic removal. For example, full-link tracing systems rely on packet copies collected from different network locations to reconstruct forwarding paths, while payload analysis systems typically require only one packet copy for application-layer inspection.

Therefore, packet deduplication in NPB must be implemented based on specific application requirements. A flexible deduplication mechanism should accurately identify duplicate packets while preserving the traffic information required by different visibility applications.

This paper introduces the principles of packet deduplication, packet identification methods, accuracy optimization techniques, and implementation approaches based on FusionNOS.

In network visibility architectures, duplicate traffic is primarily generated because the same traffic is collected from multiple monitoring points. To provide complete network visibility, organizations deploy TAPs, SPAN ports or monitoring interfaces at different locations, such as access switches, aggregation switches, core switches and security boundaries.

As a result, the same original packet may be captured multiple times along its forwarding path. For example, a packet entering a data center can be collected at the access layer, aggregation layer and core layer, creating multiple copies of the same traffic instance.

These duplicate copies are intentional and provide valuable visibility information for different monitoring and analysis requirements. However, when forwarded to backend security and analysis tools, redundant copies may increase bandwidth consumption, processing overhead and storage requirements, making packet deduplication an important function of NPB platforms.

These tools reconstruct the complete transmission path of a single packet across all network devices and links, to troubleshoot packet loss, latency, forwarding anomalies, tunnel encapsulation failures and layer-2 loops.

Typical examples:

1. Full traffic trace and recording system (NTA / Full traffic forensics)

Synchronously capture identical flow packets from multiple computer rooms and TAP points. Compare packet timestamps, TTL values and header changes across different nodes to locate packet loss points, forwarding latency and mirroring time offset.

2. Network Performance Monitoring platform (NPM)

Calculate hop-by-hop latency, jitter and retransmission statistics. Original packet copies from every capture point are required for time sequence alignment.

3. Fault diagnosis tools (Distributed Wireshark capture, network diagnostic platform)

All packets of the same flow captured at ingress switches, intermediate NPBs and egress devices must be retained completely. Header differences are compared to identify forwarding exceptions.

Key Characteristics：

Packet deduplication must be disabled: Multiple copies of the same source packet represent capture snapshots taken at distinct network nodes. Loss of any copy will break the full-link tracing chain.

These tools only parse the application-layer payload content of packets, regardless of the network devices the packets traverse. A single original packet copy per flow is enough to complete all analysis tasks.

Typical Examples：

1. Intrusion Detection/Defense Systems (IDS/NDR)

They conduct deep DPI parsing on application payloads, match attack signatures and detect malicious files. Duplicate packets provide no additional detection value.

2. Business Audit & Log Parsing Platforms

Extract payload fields from HTTP, GTP, database and other services to implement behavior auditing, transaction reconstruction and compliance forensics.

3. Application-Layer Decoding Tools (Protocol parsing, traffic reports, business metric statistics)

Count interface invocations, business packet lengths and transaction success rates. Metric calculation only requires one single packet copy.

4. Threat Sandboxes & File Reconstruction Systems

Extract attachments and malicious samples inside payloads; duplicate packets carry no extra analytical value.

Key Characteristics：

• Packet deduplication in NPB is enabled: Multiple copies of identical source packets can be discarded, with only one copy forwarded to backend tools.
• The core focus lies on Layer 4 and above payload content, with no reliance on link forwarding logic or outer packet headers.

In a network visibility deployment, duplicate traffic does not necessarily mean that multiple captured packets are bit-by-bit identical. The same original packet may be captured at different observation points along its forwarding path, and each captured copy may contain different header information due to network processing.

For example, as shown in above, a PC accesses Google through multiple network devices. The same original packet A may be captured at three different monitoring points, resulting in three packet copies: A1, A2 and A3.

Although A1, A2 and A3 represent the same original traffic instance, their packet contents may not be identical because intermediate network devices can modify packet fields during forwarding.

Therefore, a simple full-field packet comparison cannot accurately identify duplicate traffic.

When packet A passes through different network devices, the following fields may be modified:

Packet deduplication identifies multiple copies of the same original packet generated from different network observation points and forwards only one copy to downstream analysis tools.

Since packet fields may change when traffic traverses different network devices, deduplication cannot rely on a full packet byte-by-byte comparison. Instead, users can define which fields should be considered during packet matching and which fields should be ignored.

As shown above, taking MAC address–ignored deduplication as an example, packets captured from different monitoring points may contain different MAC addresses while carrying identical IP information and payload content.

For example:

• Packet A:
  • Source MAC: MAC(A)
  • IP Header: Same
  • Payload: Same
• Packet B:
  • Source MAC: MAC(B)
  • IP Header: Same
  • Payload: Same

If MAC Address is configured as an ignored field, the MAC field is replaced with a fixed zero value during packet normalization.

After normalization, the both packets are 0,IP, Payload, then both packets generate the same Hash Key and are identified as duplicate traffic.

After packet normalization, the system calculates a Hash Key based on the selected packet content.

The Hash Key is used as an index to efficiently identify previously received packets instead of performing expensive full-packet comparison.

The deduplication process works as follows:

1. Hash Key lookup

When a packet arrives, the system calculates its Hash Key and checks whether the same key exists in the deduplication cache.

2. First packet detection

If the Hash Key does not exist:

• The packet is considered the first copy of this traffic instance.
• The packet is forwarded to the destination tool.
• The Hash Key is stored in the cache with a timestamp.

3. Duplicate packet detection

If the same Hash Key already exists within the configured deduplication window:

• The packet is identified as a duplicate copy.
• The packet is discarded instead of being forwarded.

The deduplication window defines the time period during which identical packets are considered duplicates.

A longer deduplication window:

• Increases the probability of detecting delayed duplicate packets.
• Requires more Hash Key entries to be maintained.
• Increases memory consumption.

In practical network visibility deployments, duplicate packets generated by multi-point traffic collection usually arrive within a short interval, typically less than one second. Therefore, the deduplication window can be optimized according to the network topology, traffic characteristics and memory resources.

Hash-based deduplication provides efficient duplicate detection, but hash comparison alone may occasionally introduce false positives due to hash collisions or insufficient packet differentiation. To further improve duplicate identification accuracy, the Asterfusion Network Packet Broker platform provides additional verification mechanisms based on packet fields and payload content.

The platform can perform secondary verification using selected packet fields after the initial Hash Key matching.

As shown in Figure 5, the system first calculates the Hash Key based on the configured deduplication criteria. When a potential duplicate packet is detected, additional fields are compared to confirm whether the packets are truly identical.

The verification fields can include:

• IP quintuple:
  • Source IP address
  • Destination IP address
  • Source port
  • Destination port
  • Protocol
• IPID

The payload portion is verified through CRC calculation rather than full payload comparison to maintain processing efficiency.

This two-stage verification mechanism improves duplicate detection accuracy and reduces false positives caused by hash collisions.

For scenarios requiring higher duplicate identification accuracy, the platform can additionally verify packet payload content.

As shown in Figure 6, after Hash Key matching, the system calculates the payload hash and compares the last 4 bytes of the payload between packets.

This additional verification provides a higher confidence level for duplicate identification, especially when packet headers are similar but payload content may differ.

However, because payload inspection introduces additional processing overhead, this feature is generally disabled by default and should only be enabled in scenarios where higher deduplication accuracy is required.

Packet deduplication in NPB requires both high-speed packet processing capability and sufficient storage resources to maintain deduplication states.

Depending on the hardware architecture, deduplication can be implemented either directly on the switch ASIC data plane or on a programmable DPU platform.

Each approach has different advantages and limitations in terms of throughput, memory capacity and flexibility.

In this implementation mode, deduplication functions are integrated into the switch ASIC pipeline. The switch performs packet normalization, Hash Key calculation and duplicate lookup directly in the hardware forwarding pipeline.

 Advantages:

High throughput with near wire-speed processing

Because packet processing is performed directly in the switch ASIC, this approach provides extremely low latency and can achieve near wire-speed deduplication performance.

 Limitations:

1. Limited deduplication table capacity

The ASIC relies on dedicated on-chip resources, such as SRAM and TCAM, to maintain deduplication entries.

Due to limited hardware resources, the size of the Hash Table is restricted. In large-scale environments with millions of concurrent flows, the available table capacity may not be sufficient to maintain all deduplication states.

 2. Limited programmability

Switch ASIC pipelines are highly optimized for fixed packet processing operations. Custom deduplication algorithms, complex matching logic or advanced packet analysis functions are difficult to implement.

 3. Limited packet parsing capability

ASIC-based processing is restricted by the maximum packet parsing length supported by the chip.

When full-packet content verification is required, packets beyond the supported parsing range cannot be fully analyzed, which may result in missed duplicate identification.

In this implementation mode, deduplication is performed on a programmable DPU platform running FusionNOS.

Traffic is redirected from the switch to the DPU, where packet parsing, Hash Key calculation and deduplication logic are executed using programmable processing resources.

Advantages:

1. Large memory capacity for deduplication state storage

Compared with switch ASIC resources, DPU platforms provide significantly larger memory capacity.

The platform can maintain approximately 20 million deduplication table entries, enabling large-scale flow tracking and wire-speed deduplication in typical deployment scenarios.

 2. Flexible programmability

The DPU architecture allows customized deduplication algorithms, configurable packet parsing and advanced matching logic.

It can support complex deduplication scenarios, including full-packet analysis and customized field matching.

 Limitation:

Processing capability and interconnect bandwidth constraints

Unlike ASIC-based processing, DPU-based deduplication depends on general-purpose processing resources and data transfer bandwidth.

Therefore, the maximum deduplication throughput is limited by:

• DPU computing capability;
• Memory access performance;
• Switch-to-DPU interconnection bandwidth.