De-duplication

De-duplication application targets, identifies, and eliminates duplicate packets, blocking unnecessary duplication and sending optimized flows to your security and network monitoring tools. De-duplication lets you detect and choose the duplicate packets to count or drop in a network analysis environment.

Duplicate packets are common in network analysis environments where both the ingress and egress data paths are sent to a single output. They can also appear when packets are gathered from multiple collection points along a path. The de-duplication application lets you eliminate these packets, only forwarding a packet once and thus reducing the processing load on your tools.

Feature Overview

There are two actions that can be specified for handling the duplicate packets detected:

■   drop, which drops the duplicate packets
■   count, which counts the duplicate packets, but does not drop them

A time interval can be configured within which an identical packet will be considered a duplicate. The greater the interval over which traffic can be checked for duplicates, the higher the accuracy of the de-duplication detection and subsequent elimination.

For example, if two of the same packets are seen in the specified time interval, the packets will be detected as duplicates. If one packet is seen in the time interval and another packet is seen in a later time interval, the packets will not be detected as duplicates.

For IPv4 and IPv6 packets, to determine if a packet is considered to be a duplicate, parts of the IP headers (Layer 3 and Layer 4), as well as part of the payload are compared.

For non-IP packets, a packet is considered to be a duplicate if it is identical.

Configure De-duplication Application

To add a de-duplication application:

  1. Drag and drop Dedup from APPLICATIONS to the graphical workspace.
  2. Click the Dedup application and select Details. The Application quick view appears.
  3. In the Application quick view, enter the information as follows:

    Parameter

    Description

    Alias

    Enter a name for the application.

    Action

    Specifies whether duplicate packets are to be counted or dropped as follows:

    o Count– The de-duplication application counts the duplicate packets, but does not drop them.
    o Drop– The de-duplication application drops the duplicate packets.

    The default is drop.

    IP Tclass

    IP TOS

    TCP Sequence

    VLAN

    These options are useful when applying de-duplication operations to packets in a NAT environment. Different NAT implementations can change certain packet header fields (for example, the TCP sequence number). If you want to be able to detect duplicates without requiring that these fields match (ToS field, TCP sequence number, VLAN ID), you can disable the corresponding option.

    o IP Tclass – Ignore or include IPv6 traffic class. Use for IPv6. The default is include.
    o IP TOS – Ignore or include the IP ToS bits when detecting duplicates. Use for IPv4. The default is include.
    o TCP Sequence – Ignore or include the TCP Sequence number when detecting duplicates. The default is include.
    o VLAN – Ignore or include the VLAN ID when detecting duplicates. The default is ignore.

    Include means the field will be included when the application compares packets.

    Ignore means the field will be ignored when the application compares packets.

    Timer <Value: 10-500000 μs>

    Configures the time interval within which an identical packet will be considered a duplicate. The greater the interval over which traffic can be checked for duplicates, the higher the accuracy of the de-duplication detection and subsequent elimination. The default is 50,000µs.

    For example, if two same packets are seen in the specified time interval, the packets will be detected as duplicates. If one packet is seen in the time interval and another packet is seen in a later time interval, the packets will not be detected as duplicates.

    NOTE: Retransmissions are not counted as duplicates.

  4. Click Save.

The de-duplication application is successfully configured.

What's Next

You can configure the traffic health monitoring for this application in the THRESHOLDS tab. You can select an existing template from the Threshold Templates drop-down menu or provide the threshold values. For more details on Traffic health monitoring and how to create threshold template, refer to Traffic Health Monitoring.

You can view the configuration health status and the traffic health status of the application in the HEALTH STATUS tab. For more details on configuration health and traffic health, refer to Monitor Cloud Health.

You can view the statistics of the application in the STATISTICS tab.