Troubleshoot GigaSMART 5G CUPS Issues

Use the following command to determine if any of the defined rules are not being affected with the traffic for both first level and second level map:

show map stats all

Problem: If the rules defined in the second level map are correct and the vport receives traffic but it does not match with the rule or the rules, check in the var/log/messages in engine to determine if the configurations ( maps ) are properly pushed to the engine or the engines. Refer to the logs at the time of configuration.

Fix:

• Reboot engine / engines.
• Remove configuration and re-configure.

Corrective Action:

Collect the logs during configuration and provide to the Customer Support for further analysis.

Use the following command to check the flow ops table/session table:

show gsgroup flow-ops-report alias <upn gsgroup> type flow-filtering any

• Check if there are any XAUI drops.
• Determine whether the following GTP-U interfaces receives traffic:

  • N3 for 5G CUPS

  • S1u or S5u for 4G CUPS

• Check if GTP-U packets are sent to collector or not received.

Fix

1. If XAUI drops are found, determine the packet rate by looking at the packet rate from the engine by using the following command:

show port stats port-list <engine>

Either add one more engine to the engine group or reduce the ingress traffic. Also log into each engine and verify XAUI lane distribution of traffic.

2. Based on the GTP resource statistics, try to isolate the interface that is not receiving traffic.

3. If GTP-U packets are sent to the collector, check the session table if the IP and TEID are populated correctly for the given session or if it is incomplete. If there are plenty of sessions with incomplete sessions, will need to verify CPN control plane correlation ( discussed in CPN section).

   Copy

From the Flow Ops table,

Gsgroup: gsgrp, Flow Filtering Report Summary
=================================================================
        Control Tunnels             : 2
        Control & User Tunnels      : 4
 
        GTP Session Stats        Sessions     Tunnels
        ===================== =========== ===========
        Gn/Gp                           0           0
        S1u/S11                         2           4
        S11u/S11                        0           0
        S5/S8                           0           0
        S3/S4                           0           0
        S2b                             0           0
        Other                           0           0
 
        Pending Session                 0
        Control Only Session            0

Actions Required:

hc3 (config) # show fabric advanced-hash box-id 22 brief
===========================================================================================
                                    Advanced Hash Configuration
===========================================================================================
Params:        ip4  ip4  prot  prt  prt  ip6  ip6  ip6  ip6  ip6  mac  mac  eth   mpls  gtp   ing
               src  dst  src   dst  src   dst  nxt  prt  prt  src  dst  type        teid  prt
                                                   hdr  src  dst
-------------------------------------------------------------------------------------------------
                       BOX ID = 22
------------------------------------------------------------------------------------
                     On   On   On    Off  Off  On   On   Off  Off  Off  Off  Off  Off   Off   Off   Off
==========================================================================================

To check whether the connection from the CPN is established to all engines in UPN or just partial, use the following command:

show apps listener stats all

Problem: If the connection between the CPN and UPN are not established correctly, GTP-u traffic will be sent to the collector as UPN do not receive session information from CPN. The session information tells the UPN on how to treat traffic.

Fix: Verify the following in listener configuration:

Log into each engine of the UPN and issue the following command:

Problem: If the counter under Received is not incremented or fail counter increments, it means CPN had stopped sending session table information to UPN. This will happen due to the following reasons:

1. The traffic feed changes in CPN.

2. The change in SFFP profile.

3. Control packet rate is very high.

Issue the command ./gs_5g_test sffp-stats and check for counters.

********************** TA STATS **********************

Node Role : UPNMESSAGE STATS

Received:

Create: 0 ------------------------------> Number of sffp packets that creates sessions received by UPN

Update: 0 ----------------------------> Number of sffp packets that updates sessions received by UPN

Delete: 0 ------------------------------> Number of sffp packets that deletes sessions received by UPN.

Keep Alive: 0

Sent:

Create: 0

Update: 0

Delete: 0

Keep Alive: 0

TA MSG MEM:

TA Msg Alloc: 0

TA Msg Dealloc: 0

TCP Msgs Rcvd: 0

TX Failed:

SFFP Lookup Fails: 0

Num TCP Write Fails: 0

Num TCP TX NON EST State: 0

Fix:

Capture the output for this command multiple times and provide it to the Customer Support for further debugging along with /var/log/messages

Check the following CPN CLI counters:

• Map Statistics

• Flow ops table/session table

• Exporter Statistics

• Corrective Action:

Collect the logs during configuration and provide to the Customer Support for further analysis.

Use the following command to determine if any of the defined rules are not being affected with the traffic for both first level and second level map:

show map stats all

Problem: If the rules defined in the second level map are correct and the vport receives traffic but it does not match with the rule or the rules, check in the var/log/messages in engine to determine if the configurations ( maps ) are properly pushed to the engine or the engines. Refer to the logs at the time of configuration.

Fix:

• Reboot engine / engines.

• Remove configuration and re-configure.

Corrective Action:

Collect the logs during configuration and provide to the Customer Support for further analysis.

Use the following command to determine if there are any XUAI drops, and if there are N11/N4 interfaces get generated using a specific SUPI:

show gsgroup flow-ops-report alias <cpn gsgroup> type flow-5g any

Fix

show port stats port-list <engine>

If the connections between CPN and UPN are not established correctly, CPN cannot send the session table information to all engines of the UPN, which leads to incorrect GTPU correlation. Use the following command to determine if connection from CPN is establised to all engines in UPN or only to few engines:

show apps exporter stats all

Fix:

Verify the following in the exporter configuration: