GigaSMART De-Duplication
Required License: De-duplication
Supported Devices : GigaVUE- HC1 Gen 2, GigaVUE- HC1 Gen 3,GigaVUE‑HC2,GigaVUE- HC3 Gen 2, GigaVUE-HC3 Gen 3 ,GigaVUE-HC1 Plus.
Refer to Supported GigaSMART Operations for more details on the devices that support GigaSMART operations.
GigaSMART de-duplication detects duplicates of the following types:
IPv4 packets |
IPv6 packets |
non-IP packets (including non-IPv4 and non-IPv6 packets) |
Duplicates are packets in which the fields (including the headers and payload) are the same, with the exception of some fields such as Time-to-Live (TTL). For example, if two packets are identical except for TTL, they will be counted as duplicates.
Duplicate packets are common in network analysis environments where both the ingress and egress data paths are sent to a single output (for example, as a result of a SPAN operation on a switch). They can also appear when packets are gathered from multiple collection points along a path. GigaSMART de-duplication lets you eliminate these packets, only forwarding a packet once and thus reducing the processing load on your tools.
There are two actions that can be specified for handling the duplicate packets detected:
drop, which drops the duplicate packets |
count, which counts the duplicate packets, but does not drop them |
A time interval can be configured within which an identical packet will be considered a duplicate. The greater the interval over which traffic can be checked for duplicates, the higher the accuracy of the de-duplication detection and subsequent elimination.
For example, if two of the same packets are seen in the specified time interval, the packets will be detected as duplicates. If one packet is seen in the time interval and another packet is seen in a later time interval, the packets will not be detected as duplicates.
For IPv4 and IPv6 packets, to determine if a packet is considered to be a duplicate, parts of the IP headers (Layer 3 and Layer 4), as well as part of the payload are compared.
For non-IP packets, a packet is considered to be a duplicate if it is identical.
Keep in mind the following when configuring GigaSMART de-duplication:
Feature |
Description |
Layer 2 Retransmissions Not Removed |
Valid Layer 2 retransmissions are part of normal network behavior and are not removed by the de-duplication feature. Layer 2 retransmissions will show differences in the IP Window ID field. |
Encapsulated Duplicates Not Removed |
If the same packet is seen once with encapsulation (for example, GRE) and once without encapsulation, the GigaSMART will not detect and remove the duplicate. |
No NAT or PAT |
Packets tapped on opposite sides of a NAT or PAT boundary will differ in the Network layer and will not be detected as duplicates. |
MPLS and VLAN Tags |
De-duplication properly parses VLAN and MPLS tags to get to the IP headers. |
VN-Tag Packets |
VN-Tag packets are treated as non-IP packets. User Header Stripping to strip VN-Tag to get to the IP headers for de-duplication. Refer to GigaSMART Header Stripping. |
GigaSMART Engine Ports |
De-duplication operations can be assigned to GigaSMART groups consisting of multiple engine ports. Refer to Groups of GigaSMART Engine Ports for details. |
To configure de-duplication, use the following steps:
Configure GigaSMART parameters on a specified GigaSMART group. |
Configure a GigaSMART operation. |
Configure a map that will use the de-duplication GigaSMART operation. This ties de-duplication to rules defined in a flow map, which applies the GigaSMART operation to specific traffic flows. |
These steps are detailed in Example – GigaSMART De-duplication.
Use the Dedup sectionunder GigaSMART Parameters on the GigaSMART Groups configuration page to configure options for GigaSMART de-duplication operations. The following table describes the configuring parameters for de-duplication on a specified GigaSMART group:
Parameter |
Description |
||||||||||||
Action |
Specifies whether duplicate packets are to be counted or dropped as follows:
The default is drop. |
||||||||||||
IP Tclass IP TOS TCP Sequence VLAN |
These options are useful when applying de-duplication operations to packets in a NAT environment. Different NAT implementations can change certain packet header fields (for example, the TCP sequence number). If you want to be able to detect duplicates without requiring that these fields match (ToS field, TCP sequence number, VLAN ID), you can disable the corresponding option.
Include means the field will be included when GigaSMART compares packets. Ignore means the field will be ignored when GigaSMART compares packets. |
||||||||||||
Timer <Value: 10-500000 μs> |
Configures the time interval within which an identical packet will be considered a duplicate. The greater the interval over which traffic can be checked for duplicates, the higher the accuracy of the de-duplication detection and subsequent elimination. The default is 50,000µs. For example, if two same packets are seen in the specified time interval, the packets will be detected as duplicates. If one packet is seen in the time interval and another packet is seen in a later time interval, the packets will not be detected as duplicates. NOTE: Retransmissions are not counted as duplicates. |
To access GigaSMART within GigaVUE‑FM, access a device that has been added to GigaVUE‑FM from the GigaVUE‑FM interface. GigaSMART appears in the navigation pane of the device view on supported devices. Refer to Access GigaSMART from GigaVUE‑FM for details.
This example shows the configuration steps for a de-duplication operation in which the GigaSMART application drops duplicate packets.
Task |
Description |
UI Steps |
|||||||||||||||||||||||||||
|
Configure a GigaSMART group and associate it with a GigaSMART engine port. |
|
|||||||||||||||||||||||||||
|
Configure parameters on the GigaSMART group. |
Action: drop IP Tclass: Include IP TOS: Ignore TCP Sequence: Ignore Vlan: Ignore Timer (us) 500000
|
|||||||||||||||||||||||||||
|
Configure the GigaSMART operation for de-duplication and assign it to the GigaSMART group. |
|
|||||||||||||||||||||||||||
|
Create a map. |
Select Pass, then select Bi Directional, and then select Port Source from the drop-down list and set the Min to 0 and Max to 443.
|
To display the statistics for de-duplication in a cluster environment, select GigaSMART > GigaSMART Operations (GSOP) > Statistics. The de-duplications statistics will be in the row labeled Dedup in the GS Operations column.
Refer to De-duplication Statistics Definitions for descriptions of these statistics as well as to GigaSMART Operations Statistics Definitions.