GigaSMART De-Duplication

Required License: De-Duplication

GigaSMART de-duplication detects duplicates of the following types:

•

IPv4 packets

•

IPv6 packets

•

non-IP packets (including non-IPv4 and non-IPv6 packets)

Duplicates are packets in which the fields (including the headers and payload) are the same, with the exception of some field such as Time-to-Live (TTL). For example, if two packets are identical except for TTL, they will be counted as duplicates.

Duplicate packets are common in network analysis environments where both the ingress and egress data paths are sent to a single output (for example, as a result of a SPAN operation on a switch). They can also appear when packets are gathered from multiple collection points along a path. GigaSMART de-duplication lets you eliminate these packets, only forwarding a packet once and thus reducing the processing load on your tools.

There are two actions that can be specified for handling the duplicate packets detected:

•

drop, which drops the duplicate packets

•

count, which counts the duplicate packets, but does not drop them

A time interval can be configured within which an identical packet will be considered a duplicate. The greater the interval over which traffic can be checked for duplicates, the higher the accuracy of the de-duplication detection and subsequent elimination.

For example, if two of the same packets are seen in the specified time interval, the packets will be detected as duplicates. If one packet is seen in the time interval and another packet is seen in a later time interval, the packets will not be detected as duplicates.

For IPv4 and IPv6 packets, to determine if a packet is considered to be a duplicate, parts of the IP headers (Layer 3 and Layer 4), as well as the payload are compared.

For non-IP packets, a packet is considered to be a duplicate if it is identical.

Keep in mind the following when configuring GigaSMART de-duplication:

Feature	Description
Layer 2 Retransmissions Not Removed	Valid Layer 2 retransmissions are part of normal network behavior and are not removed by the de-duplication feature. Layer 2 retransmissions will show differences in the IP Window ID field.
Encapsulated Duplicates Not Removed	If the same packet is seen once with encapsulation (for example, GRE) and once without encapsulation, the GigaSMART will not detect and remove the duplicate.
No NAT or PAT	Packets tapped on opposite sides of a NAT or PAT boundary will differ in the Network layer and will not be detected as duplicates.
MPLS and VLAN Tags	De-duplication properly parses VLAN and MPLS tags to get to the IP headers.
VN-Tag Packets	VN-Tag packets are treated as non-IP packets. User Header Stripping to strip VN-Tag to get to the IP headers for de-duplication. Refer to GigaSMART Header Stripping.
GigaSMART Engine Ports	De-duplication operations can be assigned to GigaSMART groups consisting of multiple engine ports. Refer to Groups of GigaSMART Engine Ports for details.

De-Duplication Configuration Steps

To configure de-duplication, use the following steps:

•

Configure GigaSMART parameters on a specified GigaSMART group.

•

Configure a GigaSMART operation.

•

Configure a map that will use the de-duplication GigaSMART operation. This ties de-duplication to rules defined in a flow map, which applies the GigaSMART operation to specific traffic flows.

These steps are detailed in Example – GigaSMART De-Duplication.

Configuring GigaSMART Parameters for Packet De-Duplication

Use the Dedup section under GigaSMART Parameters on the GigaSMART Groups configuration page to configure options for GigaSMART de-duplication operations. The following table describes the configuring parameters for de-duplication on a specified GigaSMART group:

Parameter

Description

Action

Specifies whether duplicate packets are to be counted or dropped as follows:

Count – GigaSMART counts the duplicate packets, but does not drop them.

Drop – GigaSMART drops the duplicate packets.

The default is drop.

IP Tclass

IP TOS

TCP Sequence

VLAN

These options are useful when applying de-duplication operations to packets in a NAT environment. Different NAT implementations can change certain packet header fields (for example, the TCP sequence number). If you want to be able to detect duplicates without requiring that these fields match (ToS field, TCP sequence number, VLAN ID), you can disable the corresponding option.

IP Tclass – Ignore or include IPv6 traffic class. Use for IPv6. The default is include.

IP TOS – Ignore or include the IP ToS bits when detecting duplicates. Use for IPv4. The default is include.

TCP Sequence – Ignore or include the TCP Sequence number when detecting duplicates. The default is include.

VLAN – Ignore or include the VLAN ID when detecting duplicates. The default is ignore.

Include means the field will be included when GigaSMART compares packets.

Ignore means the field will be ignored when GigaSMART compares packets.

Timer <Value: 10-500000 μs>

Configures the time interval within which an identical packet will be considered a duplicate. The greater the interval over which traffic can be checked for duplicates, the higher the accuracy of the de-duplication detection and subsequent elimination.
The default is 50,000µs.

For example, if two same packets are seen in the specified time interval, the packets will be detected as duplicates. If one packet is seen in the time interval and another packet is seen in a later time interval, the packets will not be detected as duplicates.

NOTE: Retransmissions are not counted as duplicates.

Example – GigaSMART De-Duplication

To access GigaSMART within GigaVUE-FM, access a device that has been added to GigaVUE-FM from the GigaVUE-FM interface. GigaSMART appears in the navigation pane of the device view on supported devices. Refer to Access GigaSMART from GigaVUE-FM for details.

This example shows the configuration steps for a de-duplication operation in which the GigaSMART application drops duplicate packets.

Task

Description

UI Steps

Configure a GigaSMART group and associate it with a GigaSMART engine port.

From the device view, select GigaSMART > GigaSMART Groups.

Click New.

Type an alias in the Alias field. For example, gs2port1.

Click in Port List field and select an engine port. For example,
2/1/e1.

Go to Task 2.

Configure parameters on the GigaSMART group.

Under the Dedup section on the GigaSMART Group configuration page, set the parameters as the follows:

Action: drop

IP Tclass: Include

IP TOS: Ignore

TCP Sequence: Ignore

Vlan: Ignore

Timer (us) 500000

Click Save.

Configure the GigaSMART operation for de-duplication and assign it to the GigaSMART group.

From the device view, select GigaSMART > GigaSMART Operations (GSOP) > GigaSMART Operation.

Click New.

Type a name for the operation in the alias field. For example, testdedup.

Select the GigaSMART Group create in task 1.

Select Deduplication from the GigaSMART Operations (GSOP) list and select Enable.

Click Save.

Create a map.

Select Maps > Maps > Maps.

Click New.

Type an alias in the Map Alias field that will help you identify this map. For example, testingdedup

Select Regular and By Rule for the map type and subtype.

Specify the network and tool ports in the Source and Destination fields. For example, 2/2/x4 and 2/2/x6 for Source and 2/2/x9 for Destination.

From the GigaSMART Operation (GSOP) drop-down list, select the GigaSMART operation configured in Task 3. For example, testdedup in this example.

Click Add a Rule under Map Rules and create the following rule:

Select Pass, then select Bi Directional, and then select Port Source from the drop-down list and set the Min to 0 and Max to 443.

Click Save.

Displaying De-Duplication Statistics

To display the statistics for de-duplication in a cluster environment, select GigaSMART > GigaSMART Operations (GSOP) > Statistics. The de-duplications statistics will be in the row labeled Dedup in the GS Operations column.

Refer to De-duplication Statistics Definitions for descriptions of these statistics as well as to GigaSMART Operations Statistics Definitions.