GigaVUE‑FM High Availability

The GigaVUE‑FM High Availability (HA) feature supports a highly available fabric management environment with minimal interruption. The GigaVUE‑FM HA architecture consists of three GigaVUE‑FM instances that run together as a highly available group. The highly available group provides protection from failure of any one of the members in the group.

The following figure shows the high-level architecture of the GigaVUE‑FM HA feature.

To configure the GigaVUE‑FM HA feature, you must have access to three authenticated GigaVUE‑FM instances that reside on a trusted network. All three GigaVUE‑FM instances must run the same software version. The interfaces in the GigaVUE‑FM instances must be up and must be assigned IPv4/IPv6 addresses. You can also choose to use DNS host names.

Note:  You can configure the GigaVUE‑FM HA feature only if you have administrative privileges.

The three GigaVUE‑FM instances are not required to be in the same subnet, but still must be able to communicate with each other.

In addition, ensure that the three instances have uniques host names. You must be able to ping a GigaVUE‑FM instance from the other two instances using the hostname or the IP address.

To add a GigaVUE‑FM instance to a GigaVUE‑FM HA group:

■

If host names are used to configure the HA group, the host names of the GigaVUE‑FM instances must be resolvable through a DNS server.

■

The GigaVUE‑FM instances in the HA group must be reachable to each other.

You must install a Prime license on the active GigaVUE‑FM instance to configure a High Availability group.

If the Prime license expires or if you accidentally delete the license, the existing configurations will still be present in the GigaVUE-FMs that are part of the HA group, but you will not be able to perform any new configurations. Moreover, if you disassemble the HA group, you cannot reconfigure the HA group without installing a valid Prime license.

The GigaVUE‑FM HA feature is supported on the following platforms:

■

VMWare vSphere

■

GigaVUE‑FM Hardware Appliance

■

OpenStack

Keep in mind the following rules and notes when you configure the GigaVUE‑FM HA feature:

■

You can access the GigaVUE-FMs in the high availability group using the IPv4/IPv6 address (DNS name) that is used to form the High Availability group.

■

Do not access all three GigaVUE‑FM instances at the same time, as this will impact the performance of the HA group.

■

You can deploy the three GigaVUE‑FM HA virtual machines on a WAN link with a maximum latency of 200 ms.

■

You cannot add a GigaVUE‑FM Hardware Appliance and a GigaVUE‑FM virtual machine in the same HA group.

■

The three GigaVUE‑FM instances must be identical in terms of system configuration such as hard disk, memory, and network interfaces, which include domain server, ntp server, and name server.

■

VIP support is deprecated from software version 5.13.00. If VIP is configured in earlier GigaVUE-FM versions, it will be continued after upgrading to software version 5.13.00. However, you will be only able to delete the existing VIP, adding or updating the VIP is restricted.

■

Use the orchestrated upgrade procedure to upgrade the GigaVUE-FM instances if the software version of GigaVUE-FM is 5.10.00 and above.

■

The Reload All or Reload any FM commands will work only if there is an active GigaVUE-FM instance in the HA group.

To enable GigaVUE‑FM HA, create a HA group with three GigaVUE‑FM instances.

Before you proceed to create the HA group, ensure that the following prerequisites are met:

■

You have the IPv4/IPv6 addresses or DNS names of the three GigaVUE‑FM instances.

■

All the three GigaVUE‑FM instances are running the same software versions.

To create a HA group:

1.

Log in to one of the GigaVUE‑FM instance. This instance will be the Active instance after you have created the HA group.

2.

On the left navigation pane, click and select High Availability.

3.

Click Create. The High Availability wizard appears.

Note:  It is recommended to take a backup of the Fabric Manager before creating the HA group.

4.

In the Group Name field, enter a unique name for the HA group, and then click Continue.

5.

The Add GigaVUE-FM section appears with the first GigaVUE‑FM instance (the instance that you are logged-in) added to the HA group. The GigaVUE-FMUserName, and Password are added to the fields. Enter the following details:

• IP Address/DNS Name

Note:  You can use either IPv4/IPv6 address or DNS name.
- Combining IPv4 and IPv6 address to create a GigaVUE‑FM HA group is not allowed.
- In case of dual stack environment, if the DNS name resolves to both IPv4 and IPv6 addresses, then IPv6 is preferred.

• External IP Address/DNS Name
• Third Party Authentication URL (Optional) details, if required.

6.

Click Add to add the first instance to the HA group.

7.

To add the second and third GigaVUE-FM instances, enter the following details:

Field	Description
IP Address/DNS Name	IP address/DNS name of the GigaVUE-FM instance (s for communication between the GigaVUE-FMs and to configure the HA group)
External IP Address	External IP address of the GigaVUE-FM instance (for accessing the GigaVUE-FM HA from outside the internal network.)
Username	GigaVUE-FM User Name
Password	GigaVUE-FM GUI password
Third Party Authentication	Third-party authentication URL

8.

Click Continue. You can view information such as the Group Name and details of the three GigaVUE‑FM instances that are added.

9.

Click Submit to create theGigaVUE‑FM HA Group. GigaVUE‑FM takes some time to create the group.

Important Recommendations

■

Do not close the GigaVUE‑FM tab until the GigaVUE‑FM HA group is created. Open a new tab to simultaneously perform other tasks.

■

Do not make any changes to the two standby instances while creating the HA group. This is because, the database of the two standby GigaVUE‑FM instances are overwritten with the contents of the active GigaVUE‑FM instance’s database after the formation of the HA group.

The first instance becomes the active instance of the HA group. The second and third GigaVUE‑FM instances are in standby mode. When the active instance of the GigaVUE‑FM fails, one of the other two instances becomes the active instance. You can view the health state of the HA group after the creation of the GigaVUE‑FM HA group.

A standby instance allows you to:

• reboot the other standby instances or active instance in case of failures.

Note:   A standby instance does not allow you to edit or remove any other instance from the HA group.

When you login to any of the GigaVUE-FM instances of the HA group, the dashboard page appears. Use the drop-down option in the GigaVUE‑FM GUI footer available on specific pages to view the details pertaining to that GigaVUE‑FM. For example, the following pages in the GigaVUE‑FM GUI have drop-down option in the footer:

• FM Health
• IP Resolver

GigaVUE‑FM instances participating in the HA group act as Load Balancers allowing better distribution of load within the GigaVUE-FM instances. GigaVUE-FM load balancer functionality provides the following capabilities:

• Seamless access to the GigaVUE-FM Dashboard page. Accessing any GigaVUE-FM GUI always takes you to the Fabric Manager dashboard page after successful login. This provides a cluster view of the GigaVUE-FM GUI rather than individual views for the active and standby instances.
• Ability to access the available GigaVUE-FM GUI even during a failover. However, there will be an impact to the write operation, until a new active instance takes over. For example, when you create, update or delete any of the resources in GigaVUE-FM such as maps, GigaSMART groups, tags, etc. during failover, then the operation will fail with the following error message: Unable to Connect to Server.
• Enhanced distribution of load across the members of the HA group. This allows faster response to the HTTP GET requests.
• Ability to perform backup/restore/upgrade operation from any of the GigaVUE-FM instances.

To remove or replace a standby GigaVUE‑FM instance from the GigaVUE‑FM HA group follow these steps:

1.

Login to any GigaVUE‑FM instance.

2.

On the left navigation pane, click and select High Availability.

3.

Select the standby GigaVUE‑FM instance that you want to remove from the HA group.

4.

Click the ellipsis on the GigaVUE‑FM instance widget in the High Availability page as shown in Figure 1.

Figure 39

Disabling GigaVUE‑FM Instance

5.

Select the Remove from group option. The selected standby GigaVUE‑FM instance is removed from the GigaVUE‑FM HA group.

Note:  The status of the GigaVUE‑FM HA group changes from Healthy to At Risk. You will not be allowed to remove the other standby GigaVUE‑FM instance after the HA status changes to At Risk.

To completely disassemble the GigaVUE‑FM HA group:

1.

Login to the active GigaVUE‑FM instance.

Note:  You cannot remove an active GigaVUE‑FM instance or disassemble the GigaVUE‑FM HA group by logging in from a standby GigaVUE‑FM instance.

2.

On the left navigation pane, click and select High Availability.

3.

Click the ellipsis on the GigaVUE‑FM instance widget in the High Availability page.

4.

Select the Delete HA group option. The GigaVUE‑FM HA group is disassembled and each of the GigaVUE‑FM instances become standalone GigaVUE‑FM instances.

Note:  Executing the above steps not only disassembles the GigaVUE‑FM HA group, but will also revert the standalone GigaVUE‑FM instances to their default database. The managed devices will be removed and GigaVUE‑FM settings will be reset to the default values.

If you cannot access the active GigaVUE-FM instance's GUI, use the following CLI command in all the three GigaVUE-FM instances to disassemble the HA group:
/opt/fmcs/bin/fmcs leave force
Before disassembling the instances, take a backup of the configuration . After the GigaVUE‑FM instances are disassembled from the HA group, the configurations present in the instances will be completely removed.

The GigaVUE‑FM HA state depends on the status of the three GigaVUE‑FM instances. The following table lists the various states of the GigaVUE‑FM HA group. You can view the HA group state from Administration > High Availability in the GigaVUE‑FM GUI.

Table 1: High Availability States

State	Number of GigaVUE‑FM Instances	Description
Healthy	Three GigaVUE‑FM instances are up and running	One GigaVUE‑FM instance is in active state. Other two instances are in standby state.
At Risk	Two GigaVUE‑FM instances are up and running	One GigaVUE‑FM instance is in active state. Another GigaVUE‑FM instance is in standby state. The third GigaVUE‑FM instance has either not joined the HA group or has left the HA group. Note:  You must recover the standby GigaVUE-FM instance with in 3 days (72 hours). Failure to do so will cause the instance to move out of the High Availability group.
Incomplete	One GigaVUE‑FM instance is up and running	Only one GigaVUE‑FM instance is in active state. The other two GigaVUE‑FM instances have either not joined the HA group or have left the HA group.
Standalone	One GigaVUE‑FM instance is up and running	HA is not configured on the GigaVUE‑FM instance.
Suspended	Two or more GigaVUE‑FM instances are up and running	HA is configured, but an active GigaVUE‑FM instance is yet to be elected.

Figure 40

High Availability States

The active GigaVUE‑FM instance in the high availability group may fail at times resulting in one of the standby instances to take over and become the active instance. This process is called failover.

The following table provides the reasons for failover:

Reason for Failover	Description
Reloading the active GigaVUE‑FM instance	An active GigaVUE‑FM instance is reloaded (using the Reboot option) to bring back the HA group to healthy state again.
Planned downtime of the active GigaVUE‑FM instance	An active GigaVUE‑FM instance is brought down due to various reasons, for example to upgrade to a newer software version.

The High Availability page ( On the left navigation pane, click and select High Availability.) displays the current state of the GigaVUE‑FM HA group. When a failover occurs, the HA group state changes in the GUI.

The following table lists the GUI changes for the various scenarios:

Scenario	Changes in GUI
What happens to the High Availability page immediately after a failover?	The High Availability page may not update immediately or may not show all the GigaVUE‑FM instances. Note:  Refresh the page after a minute to view the new active GigaVUE‑FM instance. However, the page will get updated automatically after 5 minutes.
What happens to an active GigaVUE‑FM instance when it fails (either by itself or if failover is triggered manually)?	● One of the standby GigaVUE‑FM instances changes to the active state. The GigaVUE‑FM instance that was initially in the active state changes to the standby state. It takes a few seconds for this transition. ● The GigaVUE‑FM GUI of the new active instance will have all the menus and dashboards.
What happens to the GigaVUE‑FM instances that were previously in standby state?	● One of the standby GigaVUE‑FM instances changes to the active state ● The other standby GigaVUE‑FM instance remains in the standby state.
What happens to the embedded devices when a new Active instance takes over?	There are no changes to the devices except that they are being managed by the new active GigaVUE‑FM instance.
How do you trigger a failover?	Click on the 'Reboot' option on the current active GigaVUE‑FM instance to trigger a failover.

Use the following table to troubleshoot issues that you might encounter while working with the HA feature.

Problem	Solution
Unable to add a license to GigaVUE‑FM HA group after a failover   Reason: Using the MAC Address in the About page to generate the license	Always use the Challenge MAC Address in the Licenses page of the active GigaVUE‑FM instance to generate licenses. Add the licenses to the HA group.
Unable to join the GigaVUE‑FM HA group after changing the password   Reason: Not logging out of GigaVUE‑FM after changing the password and before joining the HA group	Always logout of GigaVUE‑FM after changing the password and login again with the new password before joining the HA group.

When you try to upgrade the GigaVUE‑FM instances that are part of the HA group, it will trigger the orchestrated upgrade, however, orchestrated upgrade is not supported from version 5.8.xx to 5.9. You must disassemble the HA group and then upgrade each of the GigaVUE‑FM instances separately. To do this, perform the following tasks:

1.

Take a backup of the GigaVUE‑FM instance that is in the active state.

Note:  This backup will not have the HA configuration and statistics.

2.

Disassemble the HA group.

Note:  When you disassemble the HA group, the three GigaVUE‑FM instances will be restored to the default configuration database and their existing database will be deleted.

3.

Upgrade the two GigaVUE‑FM instances that you want to set as the standby instances. For instructions, refer to the GigaVUE‑FM Installation and Migration Guide.

4.

Restore the backed-up configurations on the GigaVUE‑FM instance that you want to set as the active instance, and then upgrade the instance.

Note:  Ensure to restore the configurations before you upgrade the instance. Also, you must reinstall the Prime license to reconfigure the HA group.

5.

Reconfigure the HA group.

Orchestrated upgrade is not supported from version 5.9.xx to 5.10.00. Therefore, use the fmctl command to upgrade each of the GigaVUE-FM instances in the HA group. Refer to the following steps:

1. Upgrade the first standby GigaVUE‑FM instance (GigaVUE-FM 2) using the following commands:

   fmctl image fetch <image-url-path>

   fmctl image install <image-name> location <next-boot-location-id>

   fmctl image boot <next-boot-location-id>

   The first standby GigaVUE-FM instance is upgraded to 5.10.00 version. However, GigaVUE-FM GUI will not be up for this standby instance.

2. Upgrade the second standby GigaVUE‑FM instance (GigaVUE-FM 3) using the same commands. The second standby instance is upgraded to 5.10.00 version, but GigaVUE-FM GUI will not be up for this standby instance, as well.

3. Upgrade the active GigaVUE‑FM instance (GigaVUE-FM 1) using the same commands. The active instance is upgraded to 5.10.00 version and the GigaVUE-FM GUI will be up for this instance.

4. Reboot the first and the second standby (GigaVUE-FM 2 and GigaVUE-FM 3) instances for the GigaVUE‑FM GUI to come up:

   Go to fm cli

   sudo su

   reboot

Note:  The GigaVUE-FM GUI will not appear until all the instances are upgraded.

Orchestrated upgrade of GigaVUE-FM instances in a High Availability group is similar to upgrading a standalone GigaVUE-FM instance. You can upgrade using an image that is located on an external image server, or you can use GigaVUE-FM as the image server. Refer to the Upgrade GigaVUE-FM section in the GigaVUE-FM Installation and Upgrade Guide for more details.

Note:  Orchestrated Upgrade of GigaVUE-FM instances in a HA group is supported from software version 5.10.01. For GigaVUE-FM software version 5.10.00 and above, it is recommended only to use the orchestrated upgrade procedure to upgrade the GigaVUE-FM instances in a HA group.

Prerequisites

Before upgrading the GigaVUE-FM instances in a High Availability group, ensure the following:

• The High Availability group must be in a healthy state.
• The latency between the GigaVUE-FM instances must be less than 100ms.
• The config disk space allocated to GigaVUE-FM must have a maximum sustained transfer rate of above 100MB/s. A low disk rate impacts both file sync and installation.

Steps

To upgrade the GigaVUE-FM instances in a High Availability group from the GUI, click the Upgrade option from the User icon. Always trigger the upgrade from the active GigaVUE-FM instance.

The following is the sequence of events that occur in the background:

1. Active GigaVUE-FM Instance: Software image download process is triggered and the image is downloaded.
2. Active GigaVUE-FM instance: Syncs and copies the downloaded image with one of the standby GigaVUE-FM instances - the first standby GigaVUE-FM instance.
3. First Standby Instance: Image is synced and the standby instance will get upgraded first and rebooted.
4. Second Standby Instance: Image is then synced by the second standby instance and the second standby instance will get upgraded and rebooted.
5. Active GigaVUE-FM Instance: Once the standby instances are upgraded, the active GigaVUE-FM will start to upgrade and will be rebooted.
6. A new active GigaVUE-FM instance will be elected while the active GigaVUE-FM reboots.

Note:   As orchestrated upgrade is a background process, device management tasks will be carried out seamlessly. The overall time consumed for the upgrade process is around 60 minutes and the total management loss time during orchestrated upgrade is around 1 minute. This is the time required to elect the new active GigaVUE-FM instance when the current active GigaVUE-FM instance reboots post upgrade.