Dynamic Addition of GigaVUE-FM Instances to HA Group (from Software Version 5.14.00)

Existing GigaVUE-FM HA architecture is based on 2N+1 redundancy model with one active and two standby instances providing resiliency and fault tolerance. With horizontal scaling, the services and application requests are distributed to standby GigaVUE-FMs to achieve 3000 nodes. As the HA group is limited to only three GigaVUE-FM instances, there is a need for higher computing and memory intensive resources in HA group because of the following use cases:

• Longer data retention period (up to 13 months).
• Better search performance with more horizontal distribution of Elastic Search for features such as Fabric Health Analytics (FHA), Topology Visualization.
• Better indexing performance with the number of instances collecting various statistics on the configured components.

Therefore, it is imperative to increase the number of Elastic Search instances in HA group for scalable performance of the various features in GigaVUE-FM.1 GigaVUE-FM software version 5.13.00 is scaled to manage 3000 nodes. However, with use cases such as Topology Visualization, Fabric Health Analytics that rely on Elastic Search database, Elastic search has become the bottleneck for efficient performance. Therefore, it is required to increase the number of ES instances for enhanced performance and scalability of the HA group. Starting in software version 5.14.00, GigaVUE-FM allows you to add more than three GigaVUE-FM instances to the HA group. You can add any number of instances to the HA group as required. The hardware requirements and resource allocation process for the additional instances are the same as that of the active and standby instances. The additional instances can be configured either as supporting instances or as active-eligible instances:

Supporting instances are GigaVUE-FMs that run minimal services, thereby providing more resources to elastic search services. Supporting instances:

• do not distribute application requests (all application services are turned off by default) nor run any distributed services.
• offer more scalability.
• cannot become candidates for active instance election

Active-Eligible instances are GigaVUE-FMs that distribute application requests and run distributed services. Active-eligible instances:

• add more value to fault-tolerant and high available system.
• can be part of active-instance election.

Create and Add GigaVUE-FM Instances to HA Group

To add instances and to configure the HA group:

1.

Log in to a standalone GigaVUE‑FM instance that will act as the active instance after HA is formed.

2.

On the left navigation pane, click and select High Availability.

3.

Click Create. The High Availability wizard appears.

4.

In the Group Name field, enter a unique name for the HA group, and then click Continue.

5.

In the Add GigaVUE-FM Instance page, the details of the GigaVUE-FM instance from which the HA group is created is fetched automatically.

6.

Enter the External IP Address and Third-Party Authentication details, if required. Click the save icon to save the configuration.

7.

Click Add New Node to add the second and third GigaVUE-FM instances. Enter the following details.

Field	Description
IP Address/DNS Name	IP address/DNS name of the GigaVUE-FM instance (s for communication between the GigaVUE-FMs and to configure the HA group)
External IP Address	External IP address of the GigaVUE-FM instance (for accessing the GigaVUE-FM HA from outside the internal network.)
Role	Role of the GigaVUE-FM instance. Can be configured as either: Active-eligible: provides resiliency and load distribution Supporting instances: provides scalability and performance
Username	GigaVUE-FM User Name
Password	GigaVUE-FM GUI password
Third Party Authentication	Third-party authentication URL

8.

Click the Save icon to validate the details entered for the instances.

Once the first three instances are added to the HA group, Continue button is enabled.

9.

Use the Add New Node button to continue to add the GigaVUE-FM HA instances. When adding the instances, define the role of the instances either as:

• Active-eligible
• Supporting instances

10.

Review the details of the GigaVUE-FM instances. Use the Back button to go back and change the details.

11.

Click the <icon> to edit the details of the GigaVUE-FM instances, if required.

After adding the instances to the HA group, click Submit. The HA group is configured and will be displayed as follows:

Click the ellipses on a GigaVUE-FM instance to perform the following tasks:

• Edit: Edit the instance details.
• Reboot: Reboot aGigaVUE-FM instance.
• Remove from group: Remove an instance from the HA group.

The HA page displays the following details:

Table 1:

Field	Description
Status and Reachability	Status of the GigaVUE-FM instances. Can be: Active Standby Supporting Reachability: - indicates that the instances are reachable. - indicates that the instances are unreachable.
IP Address/DNS Name	IP address/DNS name of the GigaVUE-FM instances
Role	Role of the GigaVUE-FM instance. Can be configured as follows: Active-eligible: provides resiliency and load distribution Supporting instances: provides scalability and performance
Entity ID	Entity ID of GigaVUE-FM. It is a unique alpha-numeric ID that must be configured for SSO authentication.
Host Name	Host name of the GigaVUE-FM instance.
Application Status	Status of application services that are responsible for handling the user requests: Up: Indicates service is ready to handle the user requests Down: Indicates service is not ready to handle the user requests with one or more underlying services being down. In Progress : A state that appears between up and down indicating that the services are initializing.
Software Version	Software version of the GigaVUE-FM instance.
System Uptime	Time since the instance is up and active.
Third Party Authentication URL	Third party authentication URL required for SSO configuration.
Upgrade Status	Upgrade status of GigaVUE-FM. This is displayed only when upgrade is triggered. Otherwise, this field is empty.

You can add instances to the HA group by clicking the Add New Node button, as required. You can also remove the instances from the HA group and reduce the size of the HA group as required. However, while removing the instances ensure to retain the quorum2 Quorum number of instances is the minimum number of instances that must be active-eligible in a HA group. In a HA group with five instances, the quorum number is 3 (based on the formula n/2+1)). number of instances.

Note:  To scale components such as Fabric Health Analytics, topology visualization and to improve the performance of GigaVUE-FM by fully utilizing the resource available, it is recommended to configure three active eligible instances and additional supporting instances.

The following are the behavioral changes observed in the HA group with active, standby and supporting instances:

Task	Scenario	Behavior
Backup and Restore Operation	All active-eligible instances are up and few or all supporting instances are up.	Backup will happen in active instance. During restore, all active-eligible GigaVUE-FMs will be rebooted and will perform the restore operation. Supporting instances will not participate in restore.
	Few active-eligible instances are up (Quorum number of instances are up)	Backup will happen in active instance. Restore will not be initiated in this case.
Upgrade Operation	Few instances not being up (either active-eligible or supporting instances not up)	Upgrade will not be initiated.
	Few supporting instances are removed from the HA group after formation of the group	Upgrade will be initiated and the supporting instances present in HA group would be upgraded first followed by the active-eligible instances.
	All instances are up and running	Upgrade will be initiated in active instance. Upgrade will be done in a rolling fashion starting with supporting instances and ending with the active-eligible instances.
System Configurations	All instances are active-eligible	All GigaVUE-FM's GUI will be accessible and user can configure the system through the same
	HA with active-eligible and few Supporting instances	For supporting instances, GUI will not be available (as application service is shut down). An option is provided in the GUI to turn on the application service and its dependent service(s) and the status can be seen in HA page. You can then configure the system related configuration from the logged in GigaVUE-FM instance once the application status of the supporting instance is up.
Background processes
-FM Service Distribution	All GigaVUE-FMs are active-eligible instances.	Services will be equally distributed/redistributed among the instances based on availability of the instances
	HA group with active-eligible instances and supporting instances.	Services will be distributed/re-distributed only to active eligible instances
-Load Balancer	All GigaVUE-FMs are active-eligible instances.	All GigaVUE-FM API requests will be distributed to all instances in the HA group.
	With supporting instances.	All GigaVUE-FM API requests will be distributed only to active-eligible instances.

The following table compares GigaVUE-FM HA in software version 5.13.00 and less with GigaVUE-FM HA software version 5.14.00:

	GigaVUE-FM less than or equal to 5.13.00	GigaVUE-FM greater than 5.13.00
Number of GigaVUE-FM Instances that can be configured in the HA group	Only three GigaVUE-FM instances can be added.	Minimum three. There is no limit on the maximum. Additional GigaVUE-FM instances can be configured as supporting instances or active-eligible instances
Roles and states Supported	Active (One) Standby (Two)	Active-Eligible active standby Supporting (Any number of instances) GigaVUE-FM cannot support more than seven active eligible instances in the HA group.
Number of GigaVUE-FM instances that can configured as Active Eligible.	All three GigaVUE-FM instances are active eligible.	Any number of GigaVUE-FM instances can be configured as active-eligible. By default, the first three GigaVUE-FM instances will be active-eligible in HA. Both active-eligible and supporting instances are mutually exclusive. That is, GigaVUE-FM instance cannot act as both active eligible and supporting role at any point of time.
Remove GigaVUE-FM instances	You cannot remove the GigaVUE-FM instances if the HA state is "At Risk".	At risk, you can remove the unreachable GigaVUE-FM instances but not the reachable instances.

Important Recommendations and Notes

Keep in mind the following recommendations and notes when working with the HA group.

GigaVUE-FM cannot support more than 7 active-eligible nodes (this is due to the underlying constraint in MongoDB).

Shrinking the size of the HA group:

• When reducing the size of a HA group that has both active-eligible and supporting instances, it is always recommended to remove the supporting instances first so that the active-eligible instances remain intact in the HA group. If active-eligible instances are removed first from the HA group, then the HA group will become unstable. This however depends on the number of active-eligible and supporting instances in the HA group. Consider a GigaVUE-FM HA group with 7 instances. The HA group may be created as follows:

HA group 1	With 3 active instances and 4 supporting instances	The recommendation is specifically applicable for HA group 1 in which there are only 3 active eligible instances in the group. Removing the active-eligible instances may cause the HA group to become unstable.
HA group 2	With 5 active instances and 2 supporting instances.	The recommendation is not applicable for HA group 2 as removing the active-eligible instances will not impact the stability of the HA group.

 For a HA group to be stable, it must have a minimum of two active-eligible instances in the group.

• When reducing the size of HA group from 7 instance to 5 instance or 3 instance, the quorum is re-computed and adjusted accordingly. This is to ensure that the HA group remains intact with the available set of instances.
• When reducing the size of the HA cluster, during removal of an instance make sure to give time for the Elastic Search to rebalance and return to a complete (100%) healthy state before removing the next instance. If you remove the instances subsequently one after the other, the health state of the HA group will turn red as there will be no time for shard rebalancing amongst the available instances.
  Login to GigaVUE-FM CLI as a root user and execute the following command to check the ES cluster health:
  

curl -XGET "http://localhost:9200/_cluster/health?pretty"

Supporting GigaVUE-FM Instances:

• When a supporting instance is added to the HA group, its system information will not be available immediately and will be available only after a minute.
• When the HA group is upgraded, application services of the supporting instances will be turned on automatically for upgrade and once FM HA upgrade is completed it will be turned off automatically. Application services will be turned off automatically irrespective of the upgrade completion status, that is, either success or failure.