GigaVUE‑FM High Availability

The GigaVUE‑FM High Availability (HA) feature supports a highly available fabric management environment with minimal interruption. The GigaVUE‑FM HA architecture consists of a minimum of three GigaVUE‑FM instances that run together as a highly available group. The highly available group provides protection from failure of any one of the members in the group.

The following figure shows the high-level architecture of the GigaVUE‑FM HA feature.

Starting in software version 5.14.00, you can add additional GigaVUE-FM instances to the high availability group. Refer to the Dynamic Addition of GigaVUE-FM Instances to HA Group section for details.

To configure the GigaVUE‑FM HA feature, you must have access to at least three authenticated GigaVUE‑FM instances that reside on a trusted network. All GigaVUE‑FM instances must run the same software version. The interfaces in the GigaVUE‑FM instances must be up and must be assigned IPv4/IPv6 addresses. You can also choose to use DNS host names.

Note:  You can configure the GigaVUE‑FM HA feature only if you have administrative privileges. You must have Prime license installed for forming the GigaVUE-FM HA group.

The GigaVUE‑FM instances are not required to be in the same subnet, but still must be able to communicate with each other.

In addition, ensure that the instances have unique host names. You must be able to ping a GigaVUE‑FM instance from the other instances using the hostname or the IP address.

To add a GigaVUE‑FM instance to a GigaVUE‑FM HA group:

■

The GigaVUE‑FM instances in the HA group must be reachable to each other.

■

If host names are used to configure the HA group, the host names of the GigaVUE‑FM instances must be resolvable through a DNS server.

You must install a Prime license on the active GigaVUE‑FM instance to configure a High Availability group.

If the Prime license expires or if you accidentally delete the license, the existing configurations will still be present in the GigaVUE-FMs that are part of the HA group, but you will not be able to perform any new configurations. Moreover, if you disassemble the HA group, you cannot reconfigure the HA group without installing a valid Prime license. You can re-use the same Prime license to form the HA group again.

The GigaVUE‑FM HA feature is supported on the following platforms:

■

VMware vSphere

■

GigaVUE‑FM Hardware Appliance

■

OpenStack

■

VMware NSX-T Manager

■

Gigamon Containerized Broker

Keep in mind the following rules and notes when you configure the GigaVUE‑FM HA feature:

■

The GigaVUE‑FM instances in the HA group must be identical in terms of system configuration such as hard disk, memory, and network interfaces, which include domain server, NTP server, and name server.

■

When creating a GigaVUE-FM HA group, it is highly recommended to create the High Availability group using DNS name. GigaVUE-FM HA cannot manage the individual GigaVUE-FM instances if the IP address changes (if GigaVUE-FM IP(s) are dynamic).

■

It is recommended to configure reverse DNS configuration while configuring FMHA with FQDN.

■

The GigaVUE-FMs in the high availability group can be accessed using the IPv4/IPv6 address (DNS name) that is used to form the High Availability group.

■

You can deploy the GigaVUE‑FM HA virtual machines on a WAN link with a maximum latency of 200 ms.

■

You cannot add a GigaVUE‑FM Hardware Appliance and a GigaVUE‑FM virtual machine in the same HA group.

■

VIP support is deprecated from software version 5.13.00. If VIP is configured in earlier GigaVUE-FM versions, it will be continued after upgrading to software version 5.13.00. However, you will be only able to delete the existing VIP, adding or updating the VIP is restricted.

■

Use the orchestrated upgrade procedure to upgrade the GigaVUE-FM instances if the software version of GigaVUE-FM is 5.10.00 and above.

■

The Reload All or Reload any FM commands will work only if there is an active GigaVUE-FM instance in the HA group.

■

Third Party Orchestration is supported with GigaVUE-FM High Availability only in OpenStack environment.

■

The GigaVUE-FM HA Cluster functions like standalone GigaVUE-FM’s when a fail-over occurs during long-running operations, such as Fabric Launch, Monitoring Session Deployment, or Fabric Upgrade. In such cases, manual intervention is required to re initiate these operations from the standalone or GigaVUE-FM HA nodes.

Until software version 5.13.00, GigaVUE-FM HA architecture is based on 2N+1 redundancy model (N = 1) with one active and two standby instances providing resiliency and fault tolerance. With horizontal scaling, the services and application requests are distributed to standby GigaVUE-FMs. As the HA group was limited to only three GigaVUE-FM instances, there was a need for higher computing and memory intensive resources in the HA group because of the following use cases:

• Longer data retention period (up to 13 months).
• Better search performance with more horizontal distribution of Elastic Search for features such as Fabric Health Analytics (FHA) and Topology Visualization.
• Better indexing performance with the number of instances collecting various statistics on the configured components.

Therefore, the number of GigaVUE-FM instances in the HA group had to be increased for scalable performance of the various features in GigaVUE-FM.1 GigaVUE-FM software version 5.13.00 is scaled to manage 3000 nodes. However, with use cases such as Topology Visualization, Fabric Health Analytics that rely on Elastic Search database, Elastic search has become a bottleneck for efficient performance. Therefore, it is required to increase the number of ES instances for enhanced performance and scalability of the HA group. Starting in software version 5.14.00, GigaVUE-FM allows you to add more than three GigaVUE-FM instances to the HA group. The hardware requirements and resource allocation process for the additional instances are the same as that of the active and standby instances. The additional instances can be configured either as supporting instances or as active-eligible instances:

Supporting instances are GigaVUE-FMs that run minimal services, thereby providing more resources to elastic search services. Supporting instances:

• do not distribute application requests (all application services are turned off by default) nor run any distributed services.
• offer more scalability.
• cannot become candidates for active instance election

Active-Eligible instances are GigaVUE-FMs that distribute application requests and run distributed services. Active-eligible instances:

• Add more value to fault-tolerant and high available system.
• Can be part of active instance election.

Refer to the following diagram for the roles in the HA group.

Note:  When you add a GigaVUE-FM instance as either a supporting instance or a standby instance, the current configuration of that GigaVUE-FM instance will be removed once it is added to the existing GigaVUE-FM HA cluster.

The following table compares GigaVUE-FM HA in software version 5.13.00 and less with GigaVUE-FM HA software version 5.14.00:

	GigaVUE-FM less than or equal to 5.13.00	GigaVUE-FM 5.14.00
Number of GigaVUE-FM Instances that can be configured in the HA group	Only three GigaVUE-FM instances can be added.	Minimum three. Additional GigaVUE-FM instances can be configured as supporting instances or active-eligible instances.
Status and Roles Supported	Active (One) Standby (Two)	Active-Eligible (Seven) active standby Supporting (Any number of instances) Note:  GigaVUE-FM cannot support more than seven active eligible instances in a HA group.
Number of GigaVUE-FM instances that can be configured as Active Eligible.	All three GigaVUE-FM instances are active eligible.	While creating a high availability group, there must be three active-eligible instances in the group. Both active-eligible and supporting instances are mutually exclusive. That is, GigaVUE-FM instance cannot act as both active-eligible and supporting role at any point in time.
Remove GigaVUE-FM instances	You cannot remove the GigaVUE-FM instances if the HA state is "At Risk".	You can remove the supporting GigaVUE-FM instances "At Risk" state.

To add instances and to configure the HA group:

1.

On the left navigation pane, click and select High Availability.

2.

Click Create. The High Availability wizard appears.

3.

In the Group Name field, enter a unique name for the HA group, and then click Continue.

4.

In the Add GigaVUE-FM Instance page, the details of the GigaVUE-FM instance from which the HA group is created will be fetched automatically.

5.

Enter the External IP Address and Third-Party Authentication details, if required. Click the save icon to save the configuration.

6.

Click Add New Node to add the second and third GigaVUE-FM instances. Enter the following details.

Field	Description
Status	Status of the GigaVUE-FM instance. Can be Active or Standby.
IP Address/DNS Name	IP address/DNS name of the GigaVUE-FM instance (s for communication between the GigaVUE-FMs and to configure the HA group) It is always recommended to use the DNS name while creating the GigaVUE-FM High Availability group.
External IP Address	External IP address of the GigaVUE-FM instance (for accessing the GigaVUE-FM HA from outside the internal network.)
Role	Role of the GigaVUE-FM instance. Can be configured as either: Active-eligible: provides resiliency and load distribution Supporting instances: provides scalability and performance
Username	GigaVUE-FM User Name
Password	GigaVUE-FM GUI password
Third Party Authentication	Third-party authentication URL

7.

Click the Save Node icon to validate the details entered for the instances.

Once the first three instances are added to the HA group, the Continue button is enabled.

Note:  When creating a HA group, all the existing data on the first configured GigaVUE-FM instance will be preserved, allowing the system to continue using the data after it is fully set up. However, the data on the subsequent nodes will be lost during the setup.

8.

Use the Add New Node button to continue to add the GigaVUE-FM HA instances. When adding the instances, define the role of the instances either as:

• Active-eligible
• Supporting instances

9.

Review the details of the GigaVUE-FM instances. Use the Back button to go back and change the details.

10.

Click the Edit Node icon to edit the details of the GigaVUE-FM instances, if required.

After adding the instances to the HA group, click Submit. The HA group is configured and will be displayed as follows:

Click the ellipses on a GigaVUE-FM instance to perform the following tasks:

• Edit: Edit the instance details.
• Reboot: Reboot a GigaVUE-FM instance.
• Remove from group: Remove an instance from the HA group.

The HA page displays the following details:

Table 1:

Field	Description
Status and Reachability	Status of the GigaVUE-FM instances. Can be: Active Standby Supporting Reachability: - indicates that the instances are reachable. - indicates that the instances are unreachable.
IP Address/DNS Name	IP address/DNS name of the GigaVUE-FM instances
Role	Role of the GigaVUE-FM instance. Can be configured as follows: Active-eligible: provides resiliency and load distribution Supporting instances: provides scalability and performance
Entity ID	Entity ID of GigaVUE-FM. It is a unique alpha-numeric ID that must be configured for SSO authentication.
Host Name	Host name of the GigaVUE-FM instance.
Application Status	Status of application services that are responsible for handling the user requests: Up: Indicates service is ready to handle the user requests Down: Indicates service is not ready to handle the user requests with one or more underlying services being down. In Progress: A state that appears between up and down indicating that the services are initializing.
Software Version	Software version of the GigaVUE-FM instance.
System Uptime	Time since the instance is up and active.
Third Party Authentication URL	Third party authentication URL required for SSO configuration.
Upgrade Status	Upgrade status of GigaVUE-FM. This is displayed only when the upgrade is triggered. Otherwise, this field is empty.

You can add instances to the HA group by clicking the Add New Node button, as required. You can also remove the instances from the HA group and reduce the size of the HA group as required. However, while removing the instances ensure to retain the quorum2 Quorum number of instances is the minimum number of instances that must be active-eligible in a HA group. In a HA group with five instances, the quorum number is 3 (based on the formula n/2+1)). number of instances.

Note:  To scale components such as Fabric Health Analytics, topology visualization and to improve the performance of GigaVUE-FM by fully utilizing the resource available, it is recommended to configure three active eligible instances and additional supporting instances.

The following are the behavioral changes observed in the HA group with active, standby and supporting instances:

Task	Scenario	Behavior
Backup and Restore Operation	All active-eligible instances are up and few or all supporting instances are up.	Backup will happen in active instance. During restore, all active-eligible GigaVUE-FMs will be rebooted and will perform the restore operation. Supporting instances will not participate in the restoration.
	Few active-eligible instances are up (Quorum number of instances are up)	Backup will happen in active instances. Restore will not be initiated in this case.
Upgrade Operation	Few instances not being up (either active-eligible or supporting instances not up)	Upgrade will not be initiated.
	Few supporting instances are removed from the HA group after formation of the group	Upgrade will be initiated, and the supporting instances present in HA group will be upgraded first followed by the active-eligible instances.
	All instances are up and running	Upgrade will be initiated in an active instance. Upgrade will be done in a rolling fashion starting with supporting instances and ending with the active-eligible instances.
System Configurations	All instances are active-eligible	All GigaVUE-FM's GUI will be accessible, and user can configure the system through the same
	HA with active-eligible and few Supporting instances	For supporting instances, GUI will not be available (as application service is shut down). An option is provided in the GUI to turn on the application service and its dependent service(s) and the status can be seen in HA page. You can then configure the system related configuration from the logged in GigaVUE-FM instance once the application status of the supporting instance is up.
Background processes
-FM Service Distribution	All GigaVUE-FMs are active-eligible instances.	Services will be equally distributed/redistributed among the instances based on the availability of the instances.
	HA group with active-eligible instances and supporting instances.	Services will be distributed/re-distributed only to active eligible instances
-Load Balancer	All GigaVUE-FMs are active-eligible instances.	All GigaVUE-FM API requests will be distributed to all instances in the HA group.
	With supporting instances.	All GigaVUE-FM API requests will be distributed only to active-eligible instances.

When you login to any of the GigaVUE-FM instances of the HA group, the dashboard page appears. Use the drop-down option in the GigaVUE‑FM GUI footer available on specific pages to view the details pertaining to that GigaVUE‑FM. For example, the following pages in the GigaVUE‑FM GUI have drop-down option in the footer:

• FM Health
• IP Resolver
• NTP Server for Clock Synchronization

GigaVUE‑FM instances participating in the HA group act as Load Balancers allowing better distribution of load within the GigaVUE-FM instances. GigaVUE-FM load balancer functionality provides the following capabilities:

• Seamless access to the GigaVUE-FM Dashboard page: Accessing any GigaVUE-FM GUI always takes you to the GigaVUE-FM dashboard page after successful login. This provides a cluster view of the GigaVUE-FM GUI rather than individual views for the active and standby instances.
• Ability to access the available GigaVUE-FM GUI even during a failover: However, there will be an impact to the write operation, until a new active instance takes over. For example, when you create, update or delete any of the resources in GigaVUE-FM such as maps, GigaSMART groups, tags, etc. during failover, then the operation will fail with the following error message: Unable to Connect to Server.
• Enhanced distribution of load across the members of the HA group: This allows faster response to the HTTP GET requests.
• Ability to perform backup/restore/upgrade operation from any of the GigaVUE-FM instances.

 

To remove or replace a standby GigaVUE‑FM instance from the GigaVUE‑FM HA group follow the below listed steps:

1.

Login to any GigaVUE‑FM instance.

2.

On the left navigation pane, click and select High Availability.

3.

Select the standby GigaVUE‑FM instance that you want to remove from the HA group.

4.

Click the ellipsis on the GigaVUE‑FM instance widget in the High Availability page as shown in 2.

2

Disabling GigaVUE‑FM Instance

5.

Select the Remove from group option. The selected standby GigaVUE‑FM instance is removed from the GigaVUE‑FM HA group.

Note:  The status of the GigaVUE‑FM HA group changes from Healthy to At Risk. You will not be allowed to remove the other standby GigaVUE‑FM instance after the HA status changes to At Risk.

To completely disassemble the GigaVUE‑FM HA group:

1.

Login to the active GigaVUE‑FM instance.

Note:  You cannot remove an active GigaVUE‑FM instance or disassemble the GigaVUE‑FM HA group by logging in from a standby GigaVUE‑FM instance.

2.

On the left navigation pane, click and select High Availability.

3.

Click the ellipsis on the GigaVUE‑FM instance widget in the High Availability page.

4.

Select the Delete HA group option. The GigaVUE‑FM HA group is disassembled and each of the GigaVUE‑FM instances become standalone GigaVUE‑FM instances.

Note:  Executing the above steps disassembles the GigaVUE‑FM HA group completely. The database of the individual GigaVUE-FM HA group members (active, standby, and supporting instances) will be erased and reinitiated with the default database. If you had performed orchestrated configuration flows (such as the Flexible Inline Arrangement or Intent Based Orchestration) you must manually remove and recreate these if they do not backup the GigaVUE‑FM database before choosing Delete HA Group.

If you cannot access the active GigaVUE-FM instance's GUI, use the following CLI command in all the GigaVUE-FM instances to disassemble the HA group:
/opt/fmcs/bin/fmcs leave force
Before disassembling the instances, take a backup of the configuration. After the GigaVUE‑FM instances are disassembled from the HA group, the configurations present in the instances will be completely removed.

The GigaVUE‑FM HA state depends on the status of the three GigaVUE‑FM instances. The following table lists the various states of the GigaVUE‑FM HA group. You can view the HA group state from Administration > High Availability in the GigaVUE‑FM GUI.

 

Table 3: High Availability States

State	Number of GigaVUE‑FM Instances	Description
Healthy	Three GigaVUE‑FM instances are up and running	One GigaVUE‑FM instance is in active state. Other two instances are in standby state.
At Risk	Two GigaVUE‑FM instances are up and running	One GigaVUE‑FM instance is in active state. Another GigaVUE‑FM instance is in standby state. The third GigaVUE‑FM instance has either not joined the HA group or has left the HA group. Note:  You must recover the standby GigaVUE-FM instance with in 3 days (72 hours). Failure to do so will cause the instance to move out of the High Availability group.
Incomplete	One GigaVUE‑FM instance is up and running	Only one GigaVUE‑FM instance is in active state. The other two GigaVUE‑FM instances have either not joined the HA group or have left the HA group.
Standalone	One GigaVUE‑FM instance is up and running	HA is not configured on the GigaVUE‑FM instance.
Suspended	Two or more GigaVUE‑FM instances are up and running	HA is configured, but an active GigaVUE‑FM instance is yet to be elected. Note:  The GigaVUE-FM high availability group can move to a Suspended state from the At Risk state when the HA group has only one active-eligible and reachable GigaVUE-FM instance. Events will not be captured for the status of the GigaVUE-FM instance that went down (which causes the HA group to move into a suspended state) as there is no active/leader instance in the High Availability group.

In case of FMHA going to suspended mode from At-Risk mode, the reachability status of node which went down (due to which HA went into suspended mode) won't be audited as writes are not accepted if there is no active/leader instance in the DB cluster

4

High Availability States

The active GigaVUE‑FM instance in the high availability group may fail at times resulting in one of the standby instances to take over and become the active instance. This process is called failover.

The following table provides the reasons for failover:

Reason for Failover	Description
Reloading the active GigaVUE‑FM instance	An active GigaVUE‑FM instance is reloaded (using the Reboot option) to bring back the HA group to healthy state again.
Planned downtime of the active GigaVUE‑FM instance	An active GigaVUE‑FM instance is brought down due to various reasons, for example to upgrade to a newer software version.

The High Availability page ( On the left navigation pane, click and select High Availability) displays the current state of the GigaVUE‑FM HA group. When a failover occurs, the HA group state changes in the GUI.

The following table lists the GUI changes for the various scenarios:

Scenario	Changes in GUI
What happens to the High Availability page immediately after a failover?	The High Availability page may not update immediately or may not show all the GigaVUE‑FM instances. Note:  Refresh the page after a minute to view the new active GigaVUE‑FM instance. However, the page will get updated automatically after 5 minutes.
What happens to an active GigaVUE‑FM instance when it fails (either by itself or if failover is triggered manually)?	● One of the standby GigaVUE‑FM instances changes to the active state. The GigaVUE‑FM instance that was initially in the active state changes to the standby state. It takes a few seconds for this transition. ● The GigaVUE‑FM GUI of the new active instance will have all the menus and dashboards.
What happens to the GigaVUE‑FM instances that were previously in standby state?	● One of the standby GigaVUE‑FM instances changes to the active state ● The other standby GigaVUE‑FM instance remains in the standby state.
What happens to the embedded devices when a new Active instance takes over?	There are no changes to the devices except that they are being managed by the new active GigaVUE‑FM instance.
How do you trigger a failover?	Click on the 'Reboot' option on the current active GigaVUE‑FM instance to trigger a failover.

Use the following table to troubleshoot issues that you might encounter while working with the HA feature.

Problem	Solution
Unable to add a license to GigaVUE‑FM HA group after a failover   Reason: Using the MAC Address in the About page to generate the license	Always use the Challenge MAC Address in the Licenses page of the active GigaVUE‑FM instance to generate licenses. Add the licenses to the HA group.
Unable to join the GigaVUE‑FM HA group after changing the password   Reason: Not logging out of GigaVUE‑FM after changing the password and before joining the HA group	Always logout of GigaVUE‑FM after changing the password and login again with the new password before joining the HA group.

Orchestrated upgrade of GigaVUE-FM instances in a High Availability group is similar to upgrading a standalone GigaVUE-FM instance. You can upgrade using an image that is located on an external image server, or you can use GigaVUE-FM as the image server. Refer to the Upgrade GigaVUE-FM section in the GigaVUE-FM Installation and Upgrade Guide for more details.

Note:  Orchestrated Upgrade of GigaVUE-FM instances in a HA group is supported from software version 5.10.01. For GigaVUE-FM software version 5.10.00 and above, it is recommended only to use the orchestrated upgrade procedure to upgrade the GigaVUE-FM instances in a HA group.

Prerequisites

Before upgrading the GigaVUE-FM instances in a High Availability group, ensure the following:

• The High Availability group must be in a healthy state.
• The latency between the GigaVUE-FM instances must be less than 100ms.
• The config disk space allocated to GigaVUE-FM must have a maximum sustained transfer rate of above 100MB/s. A low disk rate impacts both file sync and installation.

Steps

To upgrade the GigaVUE-FM instances in a High Availability group from the GUI, click the Upgrade option from the User icon. Always trigger the upgrade from the active GigaVUE-FM instance.

Note:  Do not run any scripts (example: scripts for fetching polling statistics of maps) or queries (example: database or stats queries) prior to or during the upgrade operation as this will drain GigaVUE-FM's memory and CPU resources and impact the upgrade operation.

The following is the sequence of events that occur in the background:

1. Active GigaVUE-FM Instance: Software image download process is triggered and the image is downloaded.
2. Active GigaVUE-FM instance: Syncs and copies the downloaded image with one of the standby GigaVUE-FM instances - the first standby GigaVUE-FM instance.
3. First Standby Instance: Image is synced, and the standby instance will get upgraded first and rebooted.
4. Second Standby Instance: Image is then synced by the second standby instance and the second standby instance will get upgraded and rebooted.
5. Active GigaVUE-FM Instance: Once the standby instances are upgraded, the active GigaVUE-FM will start to upgrade and will be rebooted.
6. A new active GigaVUE-FM instance will be elected while the active GigaVUE-FM reboots.

Note:   As orchestrated upgrade is a background process, device management tasks will be carried out seamlessly. The overall time consumed for the upgrade process is around 60 minutes and the total management loss time during orchestrated upgrade is around 1 minute. This is the time required to elect the new active GigaVUE-FM instance when the current active GigaVUE-FM instance reboots post upgrade.

In a GigaVUE-FM high availability environment, you can perform the GigaVUE-FM and device configurations only from the active GigaVUE-FM instance. Standby instances provide minimal configuration options. In case of failover, it is important to be aware of the IP addresses of the three GigaVUE-FMs and also the DNS host names of the GigaVUE‑FM instances so that you can access the active GigaVUE‑FM instance.

To overcome this restriction and still access the GigaVUE‑FM instances, you can use one of the following options:

• Use Load Balancer
• Assign a DNS Name for the GigaVUE‑FM instances