Cloud High Availability

Amazon Web Services cloud and Microsoft Azure cloud resources are hosted in multiple locations worldwide. these locations are composed of Regions and Availability Zones. Each Region is a separate geographic area and each Region has multiple, isolated locations known as Availability Zones.

In the cloud, resources can be deployed across different regions or multiple locations within a region for fault tolerance reasons. aws Availability Zones and Azure Availability Sets (or Fault Domains; Azure currently supports different resource groupings within a physical datacenter) are examples of cloud high availability offerings. When deploying cloudeos routers to enhance your cloud's network capability, deploy the cloudeos routers as a high availability pair using the cloudeos Cloud High Availability feature that fits your cloud's high availability design.

the Cloud High Availability (Cloud HA) feature adds support to make the cloudeos router deployment more resilient to various failure scenarios in the cloud, such as:
  • cloudeos router instance goes down due to underlying cloud infrastructure issues.
  • cloudeos router instance is unable to forward traffic due to connectivity issues in the cloud infrastructure.
  • cloudeos router experiences an internal issue leading to unavailability.

cloudeos router HA pair with Cloud HA is an active-active deployment model for different cloud high availability design in a region. Each cloudeos router in an HA pair provides enhanced routing capabilities as the gateway (or next-hop router for certain destinations) for the subnets to which the cloudeos routers connect. the two cloudeos router peers monitor the liveliness of each other by using Bidirectional Forwarding Detection (BFD) between the router interfaces. In case of the cloud infrastructure issues or cloudeos router failure, the active cloudeos router takes over as the gateway or next-hop for the subnets that were connected to the peer router through cloud-specific API calls that modify the corresponding cloud route table(s) according to pre-configured information.

Cloud HA Topology

This diagram shows an example of a router Cloud HA implementation.

Figure 1. Cloud high availability network topology with router instances

 

In the diagram above, a virtual network is a collection of resources that are in the same cloud region. Within this virtual network, the resources, including routers, deploy into two cloud high availability zones (Availability Zones for aws and Fault Domain for Azure) for fault tolerance reasons.

Note:For ease of discussion, we will use availability zone 1 and 2 to reference the high availability design in different clouds going forward.

Within each availability zone, the hosts/VMs interfaces are connected to their corresponding subnets when the network is operating normally. Each subnet associates to a route table within the cloud infrastructure. Static routes are configured in the cloud route tables so the traffic from the hosts/VMs are routed to routers in the corresponding availability zone as gateway or next-hop to reach certain destinations. For example, configure a default route (0.0.0.0/0) in the cloud route table with the next-hop as router's cloud interface ID or IP (varies depending on the cloud). the routing policy or protocol, such as BGP, on the routers, are user configurable based on user's network design.

the two routers in the diagram above are configured with the Cloud HA feature as HA peers. the Cloud HA on the routers would establish a BFD peering session between the two devices through ethernet or tunnel interfaces.

When BFD connectivity loss is detected by the active router, the existing routes in the backup route table in the cloud would be updated through cloud-specific API to use the active router as the next-hop. For example, if veos 2 detected BFD connectivity loss with its peer, veos 2 would update the routes in Route Table 1 so traffic from hosts in Subnet 1 and Subnet 2 for veos 1 would be forwarded to next-hop ID or IP owned by veos 2. Traffic from the hosts in availability zone 1 would first be forwarded to the corresponding subnet gateways in the cloud. After that, the subnet gateways in the cloud would forward the traffic toward the new next-hop interface ID or IP that exist on veos 2. When veos 2 received the traffic, it would forward the traffic on according to its routing table.

What about traffic going toward the hosts in availability zone 1 while connectivity to veos 1 is down? When connectivity to veos 1 is down, hosts behind Subnet 1 and Subnet 2 become unreachable to the other part of the network (routes being withdrawn by routing protocols like BGP). Since Subnet 1 and Subnet 2 are not directly connected to veos 2, a routing strategy for the two subnets as "backup" on veos 2 is to be considered as part of your network design. A typical design would be to use static routes for the subnets connected to the peer router and point them toward the cloud subnet gateways of the active router (for example, static route for peer subnet 10.1.1.0/24 would be configured on the active router as ip route10.1.1.0/24 10.2.1.1 255 where 10.2.1.1 is the gateway/next-hop for one of the ethernet interfaces) with a high administrative distance value (least preferred). the static routes would be redistributed or advertised when the original routes with better administrative distance are withdrawn or removed by dynamic routing protocol (such as BGP).

When BFD peering session is restored to UP state upon recovery, each active router would restore its locally controlled route table entries (per user configuration) to point to itself as primary gateway again.

Configuring the Cloud Proxy

 

Optional proxies can be configured if used in a deployment. the configuration is applicable for any cloud type. All web traffic for the underlying restful APIs for the Cloud provider SDK will use the configured proxies. Multiple proxies can be configured but only one can be used at any given time from the Cloud High-Availability configuration.

 

switch(config)#
switch(config)#cloud proxy test 
switch(config-cloud-proxy-test)#

 

the following example configures the cloud proxy IP, port, username, and password for HTTP.

switch(config)#
switch(config)#cloud proxy test
switch(config-cloud-proxy-test)#http 1.2.3.4 1234 username test password 7 075E731F1A
switch(config-cloud-proxy-test)#

Configuring the Cloud Provider

 

the following describes configurations required for Cloud HA on different types of clouds.

Cloud Configuration

 

To have access to the cloud services, the cloudeos and router must be provided with credentials. Additionally, a proxy may be configured for the connection to the cloud services to go through.

aws Specific Cloud

Complete the following tasks to configure aws Specific Cloud services.

  • Configure Credentials
  • Access to aws Specific Cloud API Server
  • If cloudeos is associated with a public IP address, no special configuration is required.
  • If cloudeos is not associated with an public IP address, either use aws Private Link or Proxy configuration

Configure Credentials

In the aws Specific Cloud configuration, a region must be specified. It is recommended to authorize the cloudeos router by assigning it an IAM role, but an explicit credential can also be specified.

  • IAM Role Configuration - No credentials. See Cloud Provider Helpful Tips for additional information.
  • Explicit Credential Configuration

aws Specific Cloud IAM Role Configuration

the IAM role should be configured on the aws Specific as shown below. This is the recommended configuration.

  • "Trust Relationships" has "ec2.amazonaws.com" as trusted entities.
  • "Policy" with "Permissions" for the network related EC2 actions.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:AssociateRouteTable",
"ec2:CreateRoute",
"ec2:CreateRouteTable",
"ec2:DeleteRoute",
"ec2:DeleteRouteTable",
"ec2:DescribeRouteTables",
"ec2:DescribeVpcs",
"ec2:ReplaceRoute",
"ec2:DisassociateRouteTable",
"ec2:ReplaceRouteTableAssociation",
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeInstances",
"ec2:DescribeSubnets"
],
"Resource": "*"
}
]
}

This is applicable only when running in aws cloud environment and configures various aspects of Cloud HA feature to interact with aws web services.

Note: the access-key-id and secret access-key commands are either both configured or both are omitted. If omitted, the Cloud HA Agent will try to use aws IAM role for security tokens to access and control aws route tables. Verify the IAM role for the cloudeos router Virtual Machine( VM ) is configured properly on the aws cloud. Refer to aws documentation to configure IAM role.

switch(config)#
switch(config)#cloud provider aws
switch(config-cloud-aws)#access-key 0 ATPAILIL5E982IPT7P3R
switch(config-cloud-aws)#secret access-key 0 M0RRUtAA8I8wYxJB8
switch(config-cloud-aws)#region us-west-1
switch(config-cloud-aws)#proxy test

Configure the backup-gateway, primary-gateway, Route Table ID(rtb) and local interface for aws.

the Route Table ID specifies for aws the backup-gateway and primary gateway, then the destination selects the individual route within the route table to control. the local-cloud-interface then points to the interface ID eni-867caa86 (from aws perspective) of the router that the traffic should be directed.

switch(config)#cloud high-availability
switch(config-cloud-ha)#peer veos2
switch(config-cloud-ha-peer-veos2)#aws
switch(config-cloud-ha-peer-veos2-aws)#backup-gateway rtb-40b72d24
0.0.0.0/0 local-cloud-interface eni-867caa86
switch(config-cloud-ha-peer-veos2-aws)#primary-gateway rtb-2843124c 
0.0.0.0/0 local-cloud-interface eni-867caa86

Explicit Credential Configuration

the explicit credential should be configured as shown below.

switch(config)#cloud provider aws
switch(config-cloud-aws)#region us-west-1 
switch(config-cloud-aws)#access-key 0 MYEXAMPLESECRETKEY
switch(config-cloud-aws)#secret access-key 0 MYEXAMPLESECRETKEY
switch(config-cloud-aws)#exit
switch(config-cloud)#exit

 

Azure

there are two authorization models that can be used in Azure: SDK Auth Credentials and Active Directory Credentials. SDK Auth Credentials are the recommended authorization model.
  • SDK Auth Credentials

    To generate SDK Auth Credentials, use the sdk authentication credential-file flash:startup-config command in the config-cloud-azure configuration mode.

    switch(config)#cloud provider azure
    switch(config-cloud-azure)#sdk authentication credential-file
    flash:startup-config
  • Active Directory Credentials

    the following example places the router into the config-cloud-azure configuration mode and sets the active directory credentials.

    switch(config)#cloud provider azure
    switch(config-cloud-azure)#active-directory credential 
    email subscription-id ef16892c-aa46-4aba-ae9a-d4fhsb1c612c  

Configuring BFD

 

To configure the BFD link between the HA pair of cloudeos and routers that is used to detect peer failure, the peer IP address and local BFD source interface must be provided. the following example configures Tunnel 2 as a single hop for the source interface for BFD.
switch(config)#cloud high-availability 
switch(config-cloud-ha)#peer veos2
switch(config-cloud-ha-peer-veos2)#bfd source-interface tunnel 2 single-hop

Configuring the Recovery Time

 

the recovery wait-time command in the cloud-haconfiguration sub-mode configures the amount of time to take back control of local route tables after failure recovery. the following example shows the wait time is configured to 90 seconds.

switch(config-cloud-ha-peer-veos2)#recovery wait-time 90

Cloud Provider Helpful Tips

 

the following are needed for Cloud High Availability but are not part of the cloudeos configuration on the cloudeos router. these may change or can be another way to achieve the same effect without changing the cloudeos router.

 

aws VPN Specific Cloud PrivateLink

aws VPN Specific Cloud PrivateLink allows a private (no public IP address) cloudeos instance to access services offered by aws (without using proxy).

the interface VPC endpoints enables a private cloudeos instance to connect to aws VPN Specific Cloud PrivateLink.

To configure Interface VPC Endpoints:

  1. Open the Amazon VPC console and choose Endpoints in the navigation panel.
  2. Select Create Endpoint.
  3. Choose the aws Services and select service name com.amazonaws.<your-region>.ec2.
  4. Choose the VPC and the subnets in each availability zone for the Interface VPC endpoints.
  5. Enable private DNS name and set security group accordingly.
  6. Select Create Endpoint.

Once the Endpoint(s) is created, the EC2 API IP associated with the domain-name will be updated to the endpoint IP.

Additional interface VPC endpoints information can be found at: https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpce-interface.html

Configuring Cloud High Availability

 

To enable the Cloud HA and its parameters, use the following configurations.

Enable Cloud High Availability

the cloud high-availability command places the cloudeos in the cloud-ha configuration mode. This example enables cloud high-availability and configures the peer veos2.

switch(config)#cloud high-availability 
switch(config-cloud-ha)#no shutdown
switch(config-cloud-ha)#peer veos2
switch(config-cloud-ha-peer-veos2)#

aws Specific for High Availability

 

This is applicable only when running in aws cloud environment and configures various aspects of Cloud HA feature to interact with aws web services.

Note: the access-key-id and secret access-key commands are either both configured or both are omitted. If omitted, the Cloud HA Agent will try to use aws IAM role for security tokens to access and control aws route tables. Verify the IAM role for the cloudeos router Virtual Machine( VM ) is configured properly on the aws cloud. Refer to aws documentation to configure IAM role.

switch(config)#
switch(config)#cloud provider aws
switch(config-cloud-aws)#access-key 0 ATPAILIL5E982IPT7P3R
switch(config-cloud-aws)#secret access-key 0 M0RRUtAA8I8wYxJB8
switch(config-cloud-aws)#region us-west-1
switch(config-cloud-aws)#proxy test

Configure the backup-gateway, primary-gateway, Route Table ID(rtb) and local interface for aws.

the Route Table ID specifies for aws the backup-gateway and primary gateway, then the destination selects the individual route within the route table to control. the local-cloud-interface then points to the interface ID eni-867caa86 (from aws perspective) of the router that the traffic should be directed.

aws

switch(config)#cloud high-availability
switch(config-cloud-ha)#peer veos2
switch(config-cloud-ha-peer-veos2)#aws
switch(config-cloud-ha-peer-veos2-aws)#backup-gateway rtb-40b72d24
0.0.0.0/0 local-cloud-interface eni-867caa86
switch(config-cloud-ha-peer-veos2-aws)#primary-gateway rtb-2843124c 
0.0.0.0/0 local-cloud-interface eni-867caa86

Azure Specific for High Availability

 

Azure

there are two authorization models that can be used in Azure: SDK Auth Credentials and Active Directory Credentials. SDK Auth Credentials are the recommended authorization model.
  • SDK Auth Credentials

    To generate SDK Auth Credentials, use the sdk authentication credential-file flash:startup-config command in the config-cloud-azure configuration mode.

    switch(config)#cloud provider azure
    switch(config-cloud-azure)# sdk authentication credential-file
    flash:startup-config
  • Active Directory Credentials

    the following example places the router into the config-cloud-azure configuration mode and sets the active directory credentials.

    switch(config)#cloud provider azure
    switch(config-cloud-azure)#active-directory credential 
    email subscription-id ef16892c-aa46-4aba-ae9a-d4fhsb1c612c  

Configure the backup-gateway, primary-gateway, Route Table ID (rtb), resource-group and next-hop for Azure

the resource group specified is the one which contains the route table referenced beneath it. the nextHopIp is the IP of the router interface that traffic should be directed.

Azure

switch(config)#cloud high-availability
switch(config-cloud-ha)#peer veos2
switch(config-cloud-ha-peer-veos2)#azure
switch(config-cloud-ha-peer-veos2-azure)#backup-gateway Subnet-2-veos-RouteTable 0.0.0.0/0 10.1.1.4 resource-group my_resource_group_64f86970ffe24ab 

GCP Specific for High Availability

In GCP specific cloud configuration, you must specify a Project. the GCP cloud uses either the Default Credentials, or a Service Account mode for authorization.

Sample service account file:
{
"type": "service_account",
"project_id": "project-id",
"private_key_id": "key-id",
"private_key": "-----BEGIN PRIVATE KEY-----\nprivate-key\n-----END PRIVATE KEY-----\n",
"client_email": "service-account-email",
"client_id": "client-id",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://accounts.google.com/o/oauth2/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/service-account-email"
}

For Default Credentials or Service Account authorization model, the role associated with the cloudeos instance/service account must have the following permissions:

  • compute.networks.get
  • compute.networks.updatePolicy
  • compute.projects.get
  • compute.routes.create
  • compute.routes.delete
  • compute.routes.get
  • compute.routes.list

Example

  • Cloud HA GCP configuration example.
    cloudeos(config)#
    cloudeos(config)#cloud provider gcp
    cloudeos(config-cloud-gcp)#project gcp-project-name
    cloudeos(config-cloud-gcp)#service-account file flash:.gcp_service_account.json
    cloudeos(config-cloud-gcp)#proxy test

Specify a destination prefix for Cloud HA routes on GCP to select the individual routes that we wish to control. Since route table is not present in GCP, you can specify an optional tag for each route to simulate the route table.

Example
cloudeos(config)#cloud high-availability
cloudeos(config-cloud-ha)#peer veos2
cloudeos(config-cloud-ha-peer-veos2)#gcp
cloudeos(config-cloud-ha-peer-veos2-gcp)#backup-gateway 0.0.0.0/0 tag tag2
cloudeos(config-cloud-ha-peer-veos2-gcp)#primary-gateway 0.0.0.0/0 tag tag1

Limitations

  • For Cloud HA on aws/Azure, pre-create the HA routes before configuring these routes on a cloudeos instance. For GCP, the HA routes are created automatically by Cloud HA, and thus no need to pre-create any HA routes to avoid route conflicts.
  • For GCP, though the HA routes are automatically created by Cloud HA, they do not get deleted automatically from GCP when removed from the Cloud HA configuration from cloudeos instance. Hence, you need to delete these routes from GCP manually.
  • Cloud HA on GCP allows adding routes only in the VPC network corresponding to the first interface (nic0) of the cloudeos instance.
  • Since routes in GCP are added at a per VPC level, specify the tags used to simulate route tables when adding a Cloud HA route using the primary-gateway / backup-gateway command. Use the same tag to apply manually on the Cloud HA routes to a subset of the instances in the VPC network.

cloudeos HA GCP provider commands

 

Global
  • cloud provider gcp
Interface (GCP)
  • project
  • service-account
Cloud HA GCP Configuration
  • primary-gateway
  • backup-gateway
Show Commands
  • show cloud high-availability routes
  • show cloud provider gcp
cloud provider gcp

the cloud provider gcp command places the switch in cloud-provider-gcp configuration mode, and allows you to configure cloud provider gcp command parameters. the exit command returns the cloudeos to global configuration mode.

Command Mode

Global Configuration

Command Syntax

cloud provider gcp

Example
  • these commands places the cloud provider for GCP into the configuration mode.
    cloudeos#config
    cloudeos(config)#cloud provider gcp
    cloudeos(config-cloud-gcp)#
  • the exit command returns to the global configuration mode.
    cloudeos(config-cloud-gcp)#exit
    cloudeos(config)#
project

the project command specifies the GCP project name. the no project command removes the configuration from the cloudeos running-config.

Command Mode

Global Cloud Provider GCP Configuration

Command Syntax

project name

no project name

Parameter
  • name Specifies the selected GCP project name.
Example
  • these commands configures the GCP project.
    cloudeos(config)#cloud provider gcp
    cloudeos(config-cloud-gcp)#project test-project
  • the no project command removes the GCP project.
    cloudeos(config)#cloud provider gcp
    cloudeos(config-cloud-gcp)#no project test-project
service-account

the service-account specifies the service account file when using the Service Account authorization model. the no service-account command removes the configuration from the cloudeos running-config.

Command Mode

Global Cloud Provider GCP Configuration

Command Syntax

service-account file sa-file

no service-account file sa-file

Parameter
  • sa-file Specifies the path to the service account file.
Examples
  • these commands configures the Service Account file used.
    cloudeos(config)#cloud provider gcp
    cloudeos(config-cloud-gcp)#service-account file flash:.gcp_service_account.json
    cloudeos(config-cloud-gcp)#
  • the no service-account command removes the Service Account file configuration.
    cloudeos(config)#cloud provider gcp
    cloudeos(config-cloud-gcp)#no service-account file flash:.gcp_service_account.json
    cloudeos(config-cloud-gcp)#
primary-gateway

the primary-gateway command in the cloud-ha submode adds a primary high availability route for GCP. the no primary-gateway command removes the route configuration from the cloudeos running-config.

Command Mode

Cloud HA GCP Configuration Submode

Command Syntax

primary-gateway dest-prefix [tag rt-tag]

no primary-gateway dest-prefix [tag rt-tag]

Parameter
  • dest-prefix Specifies the destination IP prefix.
  • rt-tag Specifies the route tag.
Examples
  • these commands configures a primary high availability route.
    cloudeos(config)#cloud high-availability 
    cloudeos(config-cloud-ha)#peer veos2
    cloudeos(config-cloud-ha-peer-veos2)#gcp
    cloudeos(config-cloud-ha-peer-veos2-azure)#primary-gateway 10.1.0.0/16 tag tag1
  • the no primary-gateway command removes the primary high availability route configuration.
    cloudeos(config)#cloud high-availability 
    cloudeos(config-cloud-ha)#peer veos2
    cloudeos(config-cloud-ha-peer-veos2)#gcp
    cloudeos(config-cloud-ha-peer-veos2-azure)#no primary-gateway 10.1.0.0/16 tag tag1
backup-gateway

the backup-gateway command in the cloud-ha submode adds a backup high availability route for GCP. the no backup-gateway command removes the route configuration from the cloudeos running-config.

Command Mode

Cloud HA GCP Configuration Submode

Command Syntax

backup-gateway dest-prefix [tag rt-tag]

no backup-gateway dest-prefix [tag rt-tag]

Parameter
  • dest-prefix Specifies the destination IP prefix.
  • rt-tag Specifies the route tag.
Examples
  • these commands configures a backup high availability route.
    cloudeos(config)#cloud high-availability 
    cloudeos(config-cloud-ha)#peer veos2
    cloudeos(config-cloud-ha-peer-veos2)#gcp
    cloudeos(config-cloud-ha-peer-veos2-azure)#backup-gateway 10.1.0.0/16 tag tag2
  • the no primary-gateway command removes the backup high availability route configuration.
    cloudeos(config)#cloud high-availability 
    cloudeos(config-cloud-ha)#peer veos2
    cloudeos(config-cloud-ha-peer-veos2)#gcp
    cloudeos(config-cloud-ha-peer-veos2-azure)#no backup-gateway 10.1.0.0/16 tag tag2
show cloud high-availability routes

the show cloud high-availability routes commanddisplays the configured local or peer route table, destination IP address and local Next Hop Interface.

Command Mode

EXEC

Command Syntax

show cloud high-availability routes

Example
  • the show cloud high-availability routes command displays high availability routes information.
    cloudeos(config)#show cloud high-availability routes
     Peer Route TypeDestination Tag
    ------ ------------ -------------- ------
     peer primary21.2.3.14/24
     peer primary1.2.3.4/21 tag1 
     peer backup 1.2.3.4/21 tag2 
     peer backup11.2.3.4/21 az1 
    
show cloud provider gcp

the show cloud provider gcp command displays cloud provider information for the GCP platform.

Command Mode

EXEC

Command Syntax

show cloud provider gcp

Examples
  • the show cloud provider gcp command displays the GCP cloud configuration for Default Credentials authorization model. .
    cloudeos(config)#show cloud provider gcp
    Project: project-123
    Authentication mode: default credentials
    Service account file:
    Proxy: myProxyName
  • the following example displays the GCP cloud configuration for Service Account authorization model.
    cloudeos(config)#show cloud provider gcp
    Project: project-123
    Authentication mode: service account
    Service account file: flash:.gcp_service_account.json
    Proxy: myProxyName

Full Configurations

 

aws VPN Specific Cloud Full Configuration

the following aws configuration is valid for use with the IAM role.

cloud provider aws
 region us-west-1
!
cloud high-availability
 no shutdown
 !
 peer veos2
aws
 backup-gatewayrtb-40b72d24 0.0.0.0/0 local-cloud-interface eni-26cb1d27
 backup-gatewayrtb-17b32973 0.0.0.0/0 local-cloud-interface eni-1589e714
 backup-gatewayrtb-54503330 0.0.0.0/0 local-cloud-interface eni-56cf1957
 primary-gateway rtb-a4be24c0 0.0.0.0/0 local-cloud-interface eni-26cb1d27
 primary-gateway rtb-40b72d24 0.0.0.0/0 local-cloud-interface eni-56cf1957
 primary-gateway rtb-63b02a07 0.0.0.0/0 local-cloud-interface eni-1589e714
peer address 10.2.201.149
recovery wait-time 5
bfd source-interface Ethernet1
!

 

Azure Full Configuration

the following Azure configuration is valid for the MSI.

cloud high-availability
 no shutdown
 !
 peer veos2
 azure
backup-gateway Subnet-2-veos-RouteTable 0.0.0.0/0 10.1.2.4 resource-group CloudHaAzure
backup-gateway Subnet-2-veos-RouteTable 10.1.0.0/16 10.1.2.4 resource-group CloudHaAzure
backup-gateway Subnet-3-veos-RouteTable 10.1.0.0/16 10.1.3.4 resource-group CloudHaAzure
backup-gateway Subnet-3-veos-RouteTable 0.0.0.0/0 10.1.3.4 resource-group CloudHaAzure
primary-gateway Subnet-1-veos-RouteTable 10.1.0.0/16 10.1.1.4 resource-group CloudHaAzure
primary-gateway Subnet-1-veos-RouteTable 0.0.0.0/0 10.1.1.4 resource-group CloudHaAzure
 
 peer address 10.1.0.5
 recovery wait-time 10
 bfd source-interface Ethernet1 

Cloud HA GCP Full Configuration

  • the following GCP configuration uses default credentials.

    cloudeos - 1

    cloud provider gcp
     project gcp-project-name
    !
    cloud high-availability
     no shutdown
     !
     peer cloudeos2
    gcp
     backup-gateway 0.0.0.0/0 tag tag1
     backup-gateway 0.0.0.0/0 tag tag2
     primary-gateway 0.0.0.0/0 tag tag3
     primary-gateway 0.0.0.0/0 tag tag4
    peer address 10.3.0.59
    recovery wait-time 5
    bfd source-interface Ethernet1
    !
  • cloudeos - 2
    cloud provider gcp
     project gcp-project-name
    !
    cloud high-availability
     no shutdown
     !
     peer cloudeos1
    gcp
     backup-gateway 0.0.0.0/0 tag tag3
     backup-gateway 0.0.0.0/0 tag tag4
     primary-gateway 0.0.0.0/0 tag tag1
     primary-gateway 0.0.0.0/0 tag tag2
    peer address 10.3.0.59
    recovery wait-time 5
    bfd source-interface Ethernet1
    !

General Troubleshooting Tips

If the Cloud HA feature is not working as expected, follow these tips for debugging.

  • Make sure that the network connectivity is there and DNS server is setup correctly for this feature to work.
  • If using Proxy and IAM role under aws, make sure that the HTTP traffic (TCP port 80) is not proxied to allow for temporarily security credentials to be retrieved by cloudeos instance.
  • Make sure to use a corresponding BFD source interface on the peer cloudeos instance. This makes sure that the BFD traffic ingress and egress are on the same interface on each instance.
  • For an aws Specific Cloud, if the IAM role does not work, Arista recommends temporarily using access-key id and secret access key with enough permissions to make sure the rest of the Cloud HA configuration is fine until you debug IAM role policy.

Caveats and Limitations

 

  • This feature was introduced in EOS release 4.20.5F which uses /mnt/flash/cloud_ha_config.json file for Cloud HA configuration without any CLI support. Starting from release 4.20.5.A1 onwards, Cloud HA feature supports CLI based configuration only. Deployments using JSON based configurations are not supported and will not work when the image is upgraded or downgraded. To upgrade image, the administrator must configure Cloud HA feature manually by converting the JSON configuration to equivalent CLI configuration. Downgrading will work as long as the older JSON file is still present in /mnt/flash directory.
  • Only a single resource-group is supported across all routing entries for Azure under Cloud specific config HA configuration.
  • the Cloud HA feature currently supports only a single peer.
  • the aws IAM role or Azure MSI needs to be configured properly using cloud provider's management tools and should give sufficient permissions to cloudeos instance to access and update route table entries.
  • the cloudeos instance should have connectivity to the cloud provider's web services. the access can also be via proxy or using feature like aws private-link.
  • the recovery wait-time should not be configured less than 10 seconds to avoid unnecessary route flapping when experiencing periodic instabilities.
  • the Cloud HA feature completely validates all the provided cloud configuration to make sure it is consistent and has all required permissions. However, the administrator should not change the provider's network configuration afterwards to avoid any issues during fail-over.
  • When there are BFD connectivity issues between the two cloudeos peers, each instance will take over the other's traffic. This cross traffic forwarding on provider's network should not have any adverse affect and still work as active-active even though both of the instance will report as fail-over. After the network connectivity is resolved, the traffic pattern reverts to the normal active-active mode.
  • the user can adjust the BFD specific parameters for the session used by Cloud HA feature using normal BFD commands such as multiplier, tx/rx intervals, etc. the Cloud HA fail-over and traffic takeover time is directly correlated with BFD failure detection time. However, when using an overly aggressive BFD, the failover time may incur higher overhead as well may result in greater instability during traffic bursts. Arista recommends using the use default BFD interval which is currently 300 msec with a multiplier of 3.
  • the bfd source-interface used in Cloud HA configuration should not belong and/or routable via the route-tables controlled by the cloudeos router instance itself to avoid traffic looping issues.
  • If the Cloud HA is in an invalid configuration state due to erroneous/mismatched configuration in the provider's cloud, the administrator has to force update the Cloud HA configuration (for example, by shut/no shut under Cloud HA mode) after updating the provider's cloud configuration. In other words, by itself, the Cloud HA feature will not retry the back-end configuration check if it is found to be invalid at the time of configuration.

Cloud High Availability Commands

Cloud High Availability CLIs

 

the Cloud High Availability CLIs are divided into three separate configuration modes:

Cloud Proxy - For proxy related configuration such as http and https.

Cloud Provider - For cloud provider specific configuration such as region, credential, and proxy name.

Cloud High-Availability - For configurations such as route, next-hop, BFD source interface, and peer.

access-key-id (cloudeos aws)

 

the cloud provider aws command places the cloudeos in cloud-provider-aws configuration mode. This configuration mode allows user to configure cloud provider aws access-key-id command parameters. the no access-key-id command removes the configuration from the cloudeos running-config. the exit command returns the cloudeos to global configuration mode.

Note:Supported only on aws platform.

Command Mode

Cloud Provider aws Configuration

Command Syntax

access-key-id Password_Type

no access-key-id Password_Type

Parameters

Password_Type

  • 0 access-key-id the password is a clear-text string. Equivalent to no parameter.
  • 7 encrypted_key the password is an encrypted string.

  • Text

Example

the following example configures the aws access key to encrypted.

switch(config)#cloud provider aws
switch(config-cloud-aws)#access-key 0 565656 test 

Example

the following example removes the aws access key and returns the to Global configuration mode.

switch(config-cloud-aws)#access-key 0 565656 test 
switch(config-cloud-aws)#no access-key 0 565656 test
switch(config)# 

Example

the following example returns the to Global configuration mode.

switch(config-cloud-aws)#access-key 0 565656 test 
switch(config-cloud-aws)#exit
switch(config)#

active-directory credential email subscription-id (cloudeos Azure)

the active-directory credential email subscription-id command configures Azure's cloud provider azure active-directory credential parameters. the no active-directory command removes the configuration from the cloudeos running-config. the exit command returns the cloudeos to global configuration mode.

Note:Supported only on Azure platform.

Command Mode

Cloud Provider Azure Configuration

Command Syntax

active-directory credential email subscription-id ID

no active-directory credential email subscription-id

Parameters

  • ID Defines the active directory subscription ID.

Example

the following example places the cloud provider for Azure into the configuration mode.

switch(config)#cloud provider azure
switch(config-cloud-azure)#active-directory credential email subscription-id

Example

switch(config)#cloud provider azure
switch(config-cloud-azure)#active-directory credential email subscription-id

azure (cloudeos Azure)

 

the azure command in the cloud-ha-peer configuration sub-mode, accessible through the cloud-ha configuration mode, allows the user to configure cloud high-availability peer related parameters. the exitcommand returns the cloudeos to the to the cloud-ha-peer configuration mode.

Note:Supported on Azure platform only.

Command Mode

Global Cloud High Availability Peer Configuration Submode

Command Syntax

azure

Example

the following example configures the peer related information for Azure.

switch(config)#cloud high-availability 
switch(config-cloud-ha)#peer p
switch(config-cloud-ha-peer-veos2)#azure
switch(config-cloud-ha-peer-veos2-azure)#

Example

the following example returns the cloudeos to the cloud-ha configuration mode.

switch(config-cloud-ha-peer-veos2-azure)#exit
switch(config-cloud-ha-peer-veos2)#

backup-gateway (cloudeos Azure)

 

the cloud high-availability command in the cloud-ha submode assigns the backup gateway parameters for the Azure high availability peered cloud. the no backup-gateway command removes the configuration from the cloudeos running-config. the exit command returns the cloudeos to global configuration mode.

Command Mode

Cloud HA azure configuration submode

Command Syntax

backup-gateway [Azure Rt_Info] resource-group [Name]

no backup-gateway

Parameters

  • Azure Rt_Info
    • azure-rt-name the azure route name.
    • dest-ip-address/mask the destination IP address.
    • local-ip-address the local IP address.
  • resource-group
    • Name Azure resource group name.

Example

the following example configures the parameters for the Azure high availability peered cloud.

switch(config)#cloud high-availability 
switch(config-cloud-ha)#peer veos2
switch(config-cloud-ha-peer-veos2)#azure
switch(config-cloud-ha-peer-veos2-azure)#backup-gateway Rt1 10.10.1.1/10 1.1.1.1 resource-group test 

Example

the following example removes the backup-gateway parameters for the Azure high availability peered cloud.

switch(config-cloud-ha-peer-veos2-azure)#no backup-gateway Rt1 10.10.1.1/10 
switch(config-cloud-ha-peer-veos2-azure)#

bfd source-interface (cloudeos)

 

the bfd source-interface command in the cloud-ha configuration submode configures BFD source interface parameters for the high availability peer. the no bfd source-interface command removed the BFD configurations from the cloudeos running-config.

Command Mode

Global Cloud HA peer configuration mode

Command Syntax

bfd source-interface [Interface_Type] single-hop

no bfd source-interface

Parameters

  • Interface_Type
    • Ethernet Ethernet Port number <1-4>.
    • Loopback Loopback interface <0-1000>.
    • Tunnel Tunnel interface <0-255>.
  • Single-hop Single hop BFD . Default is multi-hop.

Example

the following example configures Ethernet 1 as the source interface for BFD and multi-hop set as the default .

switch(config)#cloud high-availability 
switch(config-cloud-ha)#peer veos2
switch(config-cloud-ha-peer-veos2)#bfd source-interface ethernet 1 

Example

the following example configures Tunnel 2 as a single hop the source interface for BFD.

switch(config)#cloud high-availability 
switch(config-cloud-ha)#peer veos2
switch(config-cloud-ha-peer-veos2)#bfd source-interface tunnel 2 single-hop 

Example

the following example removes the BFD configuration.

switch(config-cloud-ha-peer-veos2)#no bfd source-interface

cloud high-availability peer

 

Configures one peer at a time into high availability.

Command Mode

Global Cloud HA configuration submode

Command Syntax

config-cloud-ha-peer <peer name>

Example

the following example configures the peer and places it in the cloud high availability configuration mode.

switch(config)#cloud high-availability peer peer1
 switch(config-cloud-ha-peer-peer1)#

cloud high-availability shutdown (cloudeos)

 

the shutdown command in the cloud-ha configuration mode disables High Availability for virtual EOS instances running in the cloud environment.

Command Mode

Cloud High Availability configuration

Command Syntax

shutdown

Example

the following example configures the peer and places it in the cloud high availability configuration mode.

switch(config)#cloud high-availability 
switch(config-cloud-ha)#shutdown

cloud provider aws (cloudeos)

 

the cloud provider aws command places the cloudeos in cloud-provider-aws configuration mode. This configuration mode allows user to configure cloud provider aws command parameters. the exit command returns the cloudeos to global configuration mode.

Note:Supported on aws platform only.

Command Mode

Global Configuration

Command Syntax

cloud provider aws

Example

the following example places the cloud provider for aws into the configuration mode.

switch#config
switch(config)#cloud provider aws
switch(config-cloud-aws)#

Example

the following example returns to the global configuration mode.

switch(config-cloud-aws)#exit
switch(config)#

cloud provider azure (cloudeos)

 

the cloud provider azure command places the cloudeos in cloud-provider-azure configuration mode. This configuration mode allows user to configure cloud provider azurecommand parameters. the exit command returns the cloudeos to global configuration mode.

Note: Enabled for Azure platform only.

Command Mode

Global Configuration

Command Syntax

cloud provider azure

Example

the following example places the cloud provider for Azure into the configuration mode.
switch(config)#cloud provider azure
switch(config-cloud-azure)#

cloud proxy (cloudeos)

 

the cloud proxy command places the cloudeos in cloud-proxy configuration mode. This configuration mode allows user to configure the cloud proxy command parameters. the no cloud proxy command disables the named proxy and returns the cloudeos to global configuration mode.

Command mode

Global Configuration

Command Syntax

cloud proxy proxy_name

no cloud proxy proxy_name

Parameters

proxy_name the proxy name to configure.

Example

the following example configures the cloud proxy configuration setting for "test".

switch(config)#
switch(config)#cloud proxy test
switch(config-cloud-proxy-test)#

Example

This command disables the cloud proxy named "test" and returns to global configuration mode.

switch(config-cloud-proxy-test)# no cloud proxy test
switch(config)#

http (cloudeos)

 

the http command in the cloud-proxy configuration submode configures the IP, port, username, and password parameters. the no http command removes the configured cloud proxy information for HTTP from the running-config and returns the cloudeos and to the global configuration mode.

Command mode

Global Cloud Proxy Configuration

Command Syntax

http [proxy_IP_port] [username] [password]

no http [proxy_IP_port] [username] [password]

Parameters

  • proxy_IP_port Port number to be used for the HTTP server. Options include:
    • proxy-ip IP address used for the HTTPs proxy. Dotted decimal location.
    • proxy_port HTTPS proxy port. Value ranges from 1 to 65535.
  • username Name string.
  • password Password string.
    • 0 cleartext-passwd Indicates the cleartext password is in clear text. Equivalent to the no parameter case.
    • 7 encrypted_passwd Indicates encrypted password is md5 encrypted.

Example

the following example configures the cloud proxy IP, port and username and password for HTTP.

switch(config)#cloud proxy test
switch(config-cloud-proxy-test)# http 1.2.3.4 1234 username test password 7 075E731F1A 
switch(config-cloud-proxy-test)#

Example

the following example removes the configured cloud proxy information for HTTP from the running-config.

switch(config-cloud-proxy-test)# no http 1.2.3.4 1234 username test password 7 075E731F1A 
switch(config-cloud-proxy-test)#

https (cloudeos)

 

the https command in the command in the cloud-proxy configuration submode configures the IP, port, username and password parameters. the no https command removes the configured cloud proxy information for HTTPS from the running-config and returns the cloudeos to global configuration mode.

Command mode

Global Cloud Proxy Configuration

Command Syntax

https [[proxy_IP_port] [username]] [password]

no https [[proxy_IP_port] [username]] [password]

Parameters

  • proxy_IP_port Port number to be used for the HTTP server. Options include:
    • proxy-ipIP address used for the HTTPs proxy. Dotted decimal location.
    • proxy_port HTTPS proxy port. Value ranges from 1 to 65535.
  • username Name string.
  • password Password string.
    • 0 cleartext-passwd Indicates the cleartext password is in clear text. Equivalent to the no parameter case.
    • 7 encrypted_passwd Indicates encrypted password is md5 encrypted.

Example

the following example configures the cloud proxy IP and port for HTTPS.

switch(config)#
switch(config)#cloud proxy test
switch(config-cloud-proxy-test)#https 10.3.255.155 8888

Example

the following example removes the configured cloud proxy HTTPS information from the running-config.

switch(config-cloud-proxy-test)#no https 10.3.255.155 8888
switch(config-cloud-proxy-test)#

peer (cloudeos)

 

the peer command in the cloud-ha configuration mode identifies which peer to configure by name. the peer command in the cloud-ha configuration submode configures the cloud high-availability resource group peer related parameters. the no peer command removes the configuration from the cloudeos running-config. the exitcommand returns the cloudeos to the cloud-ha configuration mode.

Command Mode

Cloud High Availability Configuration

Cloud High Availability Configuration Submode

Command Syntax

peer ip-address

no peer ip-address

Parameters

  • IP-addressthe peer IP address.

Example

the following example configures the cloud high availability peer.

switch(config-cloud-ha)#peer veos2
switch(config-cloud-ha-peer-veos2)#

Example

the following example configures the peer IP address as 10.10.10.149.

switch(config)#cloud high-availability 
switch(config-cloud-ha)#peer veos2
switch(config-cloud-ha-peer-veos2)#peer 10.10.10.149 

Example

the following example removes the peer IP address from the running-config.

switch(config)#cloud high-availability 
switch(config-cloud-ha)#peer veos2
switch(config-cloud-ha-peer-veos2)#no peer 10.10.10.149

primary gateway (cloudeos Azure)

 

the primary-gateway command in the cloud-ha submode assigns the primary gateway parameters for the Azure high availability peered cloud. the no primary-gateway command removes the configuration from the cloudeos running-config.

Note: Supported on Azure platform only.

Command Mode

Cloud HA Azure Configuration Submode

Command Syntax

primary-gateway [Azure Rt_Info] resource-group [Name]

no primary-gateway [Azure Rt_Info]

Parameters

  • Azure Rt_Info
    • azure-rt-name the azure route name.
    • dest-ip-address/mask the destination IP address.
    • local-ip-address the local IP address.
  • resource-group
    • Name Azure resource group name.

Example

the following example configures the parameters for the Azure high availability peered cloud.

switch(config)#cloud high-availability 
switch(config-cloud-ha)#peer veos2
switch(config-cloud-ha-peer-veos2)#azure
switch(config-cloud-ha-peer-veos2-azure)#primary-gateway Rt1 10.10.1.1/10 1.1.1.1 resource-group test 

Example

the following example removes the primary-gateway parameters for the Azure high availability peered cloud.

switch(config-cloud-ha-peer-veos2-azure)#no primary-gateway Rt1 10.10.1.1/10 
switch(config-cloud-ha-peer-veos2-azure)#

proxy (cloudeos)

 

the proxy command configures the cloud provider aws proxy. the no proxy command removes the configuration from the running-config. the exit command returns the cloudeos to global configuration mode.

Note: Supported on aws platform only.

Command Mode

Global Cloud aws Configuration

Command Syntax

proxy <proxy_name>

no proxy <proxy_name>

Parameters

  • proxy_name Proxy name to configure.

Example

the following example configures the Azure cloud proxy named "test".

switch(config)#cloud provider aws
switch(config-cloud-aws)#proxy test

recovery (cloud HA peer)

 

the recovery wait-time command in the cloud-ha-peer configuration submode defines the amount of time, in seconds, to take control of the local route tables after failure recovery.

Command Mode

Cloud HA peer configuration

Command Syntax

recovery wait-time <time-in-secs>

no recovery wait-time <time-in-secs>

Parameters

  • time-in-secs the defined amount of time to take back control of local route tables after failure recovery. Default is 30 seconds.

Example

the following example configures the recovery wait time to 90 seconds.

switch(config)#cloud ha
switch(config-cloud-ha)#p1
switch(config-cloud-ha-p1)#recovery wait-time 90

recovery wait-time (cloudeos)

 

the recovery wait-time command in the cloud-ha configuration sub-mode allows takes back control of local route tables after failure recovery. the no recovery wait-time command removes the configuration from the cloudeos running-config. Default is set at 30 seconds.

Command Mode

Cloud High Availability peer configuration

Command Syntax

recovery wait-time <period>

no recovery wait-time <period>

default recovery wait-time <period>

Parameters

  • periodthe defined amount of time to take back control of local route tables after failure recovery. Default is 30 seconds.

Example

the following example shows the wait time is configured to 90 seconds.

switch(config-cloud-ha-peer1)#recovery wait-time 90

Example

the following example removes the configured the wait time.

switch(config-cloud-ha-peer1)#no recovery wait-time 

Example

the following example configures the wait time to the default of 30 seconds.

switch(config-cloud-ha-peer1)#default recovery wait-time

region (cloudeos aws)

 

the cloud provider aws command places the cloudeos in cloud-provider-aws configuration mode. This configuration mode allows user to configure aws cloud provider region command parameters. the no region command removes the configuration from the cloudeos running-config. the exit command returns the cloudeos to global configuration mode.

Note:Supported on aws platform only.

Command Mode

Global Cloud Provider aws Configuration

Command Syntax

region aws-region

no region aws-region

Parameters

  • aws-region Specifies the selected region.

Example

the following example configures the cloud provider aws region.

switch(config)#cloud provider aws
switch(config-cloud-aws)#region us-west-1 
switch(config-cloud-aws)#

Example

the following example removes the cloud provider aws region.

switch(config)#cloud provider aws
switch(config-cloud-aws)#no region us-west-1
switch(config-cloud-aws)#

secret-access_key (cloudeos aws)

 

the cloud provider aws command places the cloudeos in cloud-provider-aws configuration mode. This configuration mode allows user to configure cloud provider aws secret access-key command parameters. the no secret access-key command removes the configuration from the cloudeos running-config. the exit command returns the cloudeos to global configuration mode.

Note:Supported on aws platform only.

Command Mode

Global Cloud Provider aws configuration

Command Syntax

secret access-key password_type

no secret access-key password_type

Parameters

  • 0 access-key-idthe password is a clear-text string. Equivalent to no parameter.
  • 7 encrypted_key the password is an encrypted string.
  • Text

Example

the following example configures the aws secret access key.

switch(config)#cloud provider aws
switch(config-cloud-aws)#secret access-key 0 565656 test 
switch(config-cloud-aws)#

Example

the following example removes the secret access key from the cloudeos running-config.

switch(config-cloud-aws)#no secret access-key 0 565656 test
switch(config-cloud-aws)#

Example

the following example returns to Global configuration mode.

switch(config-cloud-aws)#secret access-key 0 565656 test 
switch(config-cloud-aws)#exit
switch(config)#

show cloud high-availability (cloudeos)

 

the show cloud high-availability command displays the high availability configured settings.

Command Mode

EXEC

Command Syntax

show cloud high-availability

Example

This command displays details and status of the cloud high-availability configuration.

switch#show cloud high-availability
Cloud HA Configuration:
Peer address: 10.2.201.149
Source interface: Ethernet1
Enabled : True
Failover recovery time: 5
Status: valid
State : ready
Last failover time: never
Last recovery time: never
Last config validation start time : 0:26:08 ago
Last config validation end time : 0:26:06 ago
Failovers : 0

show cloud high-availability routes

 

the show cloud high-availability routes command displays the configured local or peer route table, destination IP address and local Next Hop Interface.

Command Mode

EXEC

Command Syntax

show cloud high-availability routes

Example

the example below displays high availability routes information.

switch(config)#show cloud high-availability routes 
Peer RouteType RouteID Destination Next Hop Interface 
----------- ----------- -------------- ---------- ------------ 
veos6 primary rtb-1dc75679 0.0.0.0/0eni-e61d95e7 
veos6 primary rtb-a29617c6 0.0.0.0/0eni-69109868 
veos6 primary rtb-acc756c8 0.0.0.0/0eni-7f1d957e 
veos6 backuprtb-43c65727 0.0.0.0/0eni-7f1d957e 
veos6 backuprtb-71c65715 0.0.0.0/0eni-e61d95e7 
veos6 backuprtb-aca223c8 0.0.0.0/0eni-69109868

show cloud provider aws (cloudeos aws)

 

the show cloud provider aws command displays cloud provider information for the aws platform.

Command Mode

EXEC

Command Syntax

show cloud provider aws

Example

the following example displays the aws cloud configuration.

switch#show cloud provider aws
Cloud aws Configuration
Region : us-west-1
Access key ID:
Access secret key:
Proxy: test

Example

the following example displays the primary and backup gateway information for the aws cloud provider.

switch#show run section cloud 
cloud provider aws
 us-west-1
 proxy test
!
cloud high-availability
 no shutdown
 !
 peer veos12
aws
 backup-gatewayrtb-40b72d24 0.0.0.0/0 local-cloud-interface eni-26cb1d27
 backup-gatewayrtb-17b32973 0.0.0.0/0 local-cloud-interface eni-1589e714
 backup-gatewayrtb-54503330 0.0.0.0/0 local-cloud-interface eni-56cf1957
 primary-gateway rtb-a4be24c0 0.0.0.0/0 local-cloud-interface eni-26cb1d27
 primary-gateway rtb-e64b2882 0.0.0.0/0 local-cloud-interface eni-56cf1957
 primary-gateway rtb-63b02a07 0.0.0.0/0 local-cloud-interface eni-1589e714
peer address 10.2.201.149
 recovery wait-time 5
 bfd source-interface Ethernet1
!
cloud proxy test
 https 10.3.255.155 8888

show cloud provider azure (cloudeos Azure)

 

the show cloud provider azure command displays Azure cloud provider information.

Command Mode

EXEC

Command Syntax

show cloud provider azure

Example

the following example displays the Azure cloud configuration.

switch#show cloud provider azure
Cloud Azure Configuration: 
Active credentials : SDK authentication credential file
SDK auth credentials file: flash:
Proxy name : 
Active directory credentials : 

show cloud proxy

the show cloud proxy command displays cloud proxy information.

Command Mode

EXEC

Command Syntax

show cloud proxy [<proxy_name>]

Parameters

  • proxy_name Identifies the selected proxy by name.

Example

#show cloud proxy [<proxy_name>]
Proxy Name : MyProxyName
Http proxy : 1.2.3.4:8080
Https proxy : 1.2.3.4:4443
Proxy user : proxyuser1
Proxy password : obfuscatedpassword