De-duplication

De-duplication lets you detect and choose the duplicate packets to count or drop in a network analysis environment.

Duplicate packets are common in network analysis environments where both the ingress and egress data paths are sent to a single output. They can also appear when packets are gathered from multiple collection points along a path. The de-duplication application lets you eliminate these packets, only forwarding a packet once and thus reducing the processing load on your tools.

Feature Overview

There are two actions that can be specified for handling the duplicate packets detected:

■   drop, which drops the duplicate packets
■   count, which counts the duplicate packets, but does not drop them

A time interval can be configured within which an identical packet will be considered a duplicate. The greater the interval over which traffic can be checked for duplicates, the higher the accuracy of the de-duplication detection and subsequent elimination.

For example, if two of the same packets are seen in the specified time interval, the packets will be detected as duplicates. If one packet is seen in the time interval and another packet is seen in a later time interval, the packets will not be detected as duplicates.

For IPv4 and IPv6 packets, to determine if a packet is considered to be a duplicate, parts of the IP headers (Layer 3 and Layer 4), as well as part of the payload are compared.

For non-IP packets, a packet is considered to be a duplicate if it is identical.

Configure De-duplication Application

To add a de-duplication application:

  1. Drag and drop Dedup from APPLICATIONS to the graphical workspace.
  2. Click the Dedup application and select Details. The Application quick view appears.
  3. In the Application quick view, enter the information as follows:

    Parameter

    Description

    Action

    Specifies whether duplicate packets are to be counted or dropped as follows:

    o Count– The de-duplication application counts the duplicate packets, but does not drop them.
    o Drop– The de-duplication application drops the duplicate packets.

    The default is drop.

    IP Tclass

    IP TOS

    TCP Sequence

    VLAN

    These options are useful when applying de-duplication operations to packets in a NAT environment. Different NAT implementations can change certain packet header fields (for example, the TCP sequence number). If you want to be able to detect duplicates without requiring that these fields match (ToS field, TCP sequence number, VLAN ID), you can disable the corresponding option.

    o IP Tclass – Ignore or include IPv6 traffic class. Use for IPv6. The default is include.
    o IP TOS – Ignore or include the IP ToS bits when detecting duplicates. Use for IPv4. The default is include.
    o TCP Sequence – Ignore or include the TCP Sequence number when detecting duplicates. The default is include.
    o VLAN – Ignore or include the VLAN ID when detecting duplicates. The default is ignore.

    Include means the field will be included when the application compares packets.

    Ignore means the field will be ignored when the application compares packets.

    Timer <Value: 10-500000 μs>

    Configures the time interval within which an identical packet will be considered a duplicate. The greater the interval over which traffic can be checked for duplicates, the higher the accuracy of the de-duplication detection and subsequent elimination. The default is 50,000µs.

    For example, if two same packets are seen in the specified time interval, the packets will be detected as duplicates. If one packet is seen in the time interval and another packet is seen in a later time interval, the packets will not be detected as duplicates.

    NOTE: Retransmissions are not counted as duplicates.

  4. Click Save.