Cloud High Availability
Amazon Web Services cloud and Microsoft Azure cloud resources are hosted in multiple locations worldwide. These locations are composed of Regions and Availability Zones. Each Region is a separate geographic area and each Region has multiple, isolated locations known as Availability Zones.
In the cloud, resources can be deployed across different regions or multiple locations within a region for fault tolerance reasons. AWS Availability Zones and Azure Availability Sets (or Fault Domains; Azure currently supports different resource groupings within a physical datacenter) are examples of cloud high availability offerings. When deploying CloudEOS and vEOS Routers to enhance your cloud's network capability, deploy the CloudEOS and vEOS Routers as a high availability pair using the CloudEOS and vEOS Cloud High Availability feature that fits your cloud's high availability design.
- CloudEOS and vEOS Router instance goes down due to underlying cloud infrastructure issues.
- CloudEOS and vEOS Router instance is unable to forward traffic due to connectivity issues in the cloud infrastructure.
- CloudEOS and vEOS Router experiences an internal issue leading to unavailability.
CloudEOS and vEOS Router HA pair with Cloud HA is an active-active deployment model for different cloud high availability design in a region. Each CloudEOS and vEOS Router in an HA pair provides enhanced routing capabilities as the gateway (or next-hop router for certain destinations) for the subnets to which the CloudEOS and vEOS routers connect. The two CloudEOS and vEOS Router peers monitor the liveliness of each other by using Bidirectional Forwarding Detection (BFD) between the router interfaces. In case of the cloud infrastructure issues or CloudEOS and vEOS router failure, the active CloudEOS and vEOS router takes over as the gateway or next-hop for the subnets that were connected to the peer router through cloud-specific API calls that modify the corresponding cloud route table(s) according to pre-configured information.
Cloud HA Topology
This diagram shows an example of a vEOS Router Cloud HA implementation.
In the diagram above, a virtual network is a collection of resources that are in the same cloud region. Within this virtual network, the resources, including vEOS routers, deploy into two cloud high availability zones (Availability Zones for AWS and Fault Domain for Azure) for fault tolerance reasons.
Within each availability zone, the hosts/VMs and vEOS interfaces are connected to their corresponding subnets when the network is operating normally. Each subnet associates to a route table within the cloud infrastructure. Static routes are configured in the cloud route tables so the traffic from the hosts/VMs are routed to vEOS Routers in the corresponding availability zone as gateway or next-hop to reach certain destinations. For example, configure a default route (0.0.0.0/0) in the cloud route table with the next-hop as vEOS Router's cloud interface ID or IP (varies depending on the cloud). The routing policy or protocol, such as BGP, on the vEOS Routers, are user configurable based on user's network design.
The two vEOS Routers in the diagram above are configured with the Cloud HA feature as HA peers. The Cloud HA on the vEOS routers would establish a BFD peering session between the two devices through ethernet or tunnel interfaces.
When BFD connectivity loss is detected by the active vEOS router, the existing routes in the backup route table in the cloud would be updated through cloud-specific API to use the active vEOS router as the next-hop. For example, if vEOS 2 detected BFD connectivity loss with its peer, vEOS 2 would update the routes in Route Table 1 so traffic from hosts in Subnet 1 and Subnet 2 for vEOS 1 would be forwarded to next-hop ID or IP owned by vEOS 2. Traffic from the hosts in availability zone 1 would first be forwarded to the corresponding subnet gateways in the cloud. After that, the subnet gateways in the cloud would forward the traffic toward the new next-hop interface ID or IP that exist on vEOS 2. When vEOS 2 received the traffic, it would forward the traffic on according to its routing table.
What about traffic going toward the hosts in availability zone 1 while connectivity to vEOS 1 is down? When connectivity to vEOS 1 is down, hosts behind Subnet 1 and Subnet 2 become unreachable to the other part of the network (routes being withdrawn by routing protocols like BGP). Since Subnet 1 and Subnet 2 are not directly connected to vEOS 2, a routing strategy for the two subnets as "backup" on vEOS 2 is to be considered as part of your network design. A typical design would be to use static routes for the subnets connected to the peer vEOS router and point them toward the cloud subnet gateways of the active vEOS router (for example, static route for peer subnet 10.1.1.0/24 would be configured on the active vEOS router as ip route10.1.1.0/24 10.2.1.1 255 where 10.2.1.1 is the gateway/next-hop for one of the ethernet interfaces) with a high administrative distance value (least preferred). The static routes would be redistributed or advertised when the original routes with better administrative distance are withdrawn or removed by dynamic routing protocol (such as BGP).
When BFD peering session is restored to UP state upon recovery, each active vEOS router would restore its locally controlled route table entries (per user configuration) to point to itself as primary gateway again.
Configuring the Cloud Proxy
Optional proxies can be configured if used in a deployment. The configuration is applicable for any cloud type. All web traffic for the underlying restful APIs for the Cloud provider SDK will use the configured proxies. Multiple proxies can be configured but only one can be used at any given time from the Cloud High-Availability configuration.
switch(config)#
switch(config)#cloud proxy test
switch(config-cloud-proxy-test)#
The following example configures the cloud proxy IP, port, username, and password for HTTP.
switch(config)#
switch(config)#cloud proxy test
switch(config-cloud-proxy-test)#http 1.2.3.4 1234 username test password 7 075E731F1A
switch(config-cloud-proxy-test)#Configuring the Cloud Provider
Cloud Configuration
To have access to the cloud services, the CloudEOS and vEOS Router must be provided with credentials. Additionally, a proxy may be configured for the connection to the cloud services to go through.
AWS Specific Cloud
Complete the following tasks to configure AWS Specific Cloud services.
- Configure Credentials
- Access to AWS Specific Cloud API Server
- If CloudEOS and vEOS is associated with a public IP address, no special configuration is required.
- If CloudEOS and vEOS is not associated with an public IP address, either use AWS Private Link or Proxy configuration
Configure Credentials
In the AWS Specific Cloud configuration, a region must be specified. It is recommended to authorize the CloudEOS and vEOS Router by assigning it an IAM role, but an explicit credential can also be specified.
- IAM Role Configuration - No credentials. See Cloud Provider Helpful Tips for additional information.
- Explicit Credential Configuration
AWS Specific Cloud IAM Role Configuration
The IAM role should be configured on the AWS Specific as shown below. This is the recommended configuration.
- "Trust Relationships" has "ec2.amazonaws.com" as trusted entities.
- "Policy" with "Permissions" for the network related EC2 actions.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:AssociateRouteTable",
"ec2:CreateRoute",
"ec2:CreateRouteTable",
"ec2:DeleteRoute",
"ec2:DeleteRouteTable",
"ec2:DescribeRouteTables",
"ec2:DescribeVpcs",
"ec2:ReplaceRoute",
"ec2:DisassociateRouteTable",
"ec2:ReplaceRouteTableAssociation",
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeInstances",
"ec2:DescribeSubnets"
],
"Resource": "*"
}
]
}
This is applicable only when running in AWS cloud environment and configures various aspects of Cloud HA feature to interact with AWS web services.
Note: The access-key-id and secret access-key commands are either both configured or both are omitted. If omitted, the Cloud HA Agent will try to use AWS IAM role for security tokens to access and control AWS route tables. Verify the IAM role for the CloudEOS and vEOS router Virtual Machine( VM ) is configured properly on the AWS cloud. Refer to AWS documentation to configure IAM role.
switch(config)#
switch(config)#cloud provider aws
switch(config-cloud-aws)#access-key 0 ATPAILIL5E982IPT7P3R
switch(config-cloud-aws)#secret access-key 0 M0RRUtAA8I8wYxJB8
switch(config-cloud-aws)#region us-west-1
switch(config-cloud-aws)#proxy testConfigure the backup-gateway, primary-gateway, Route Table ID(rtb) and local interface for AWS.
The Route Table ID specifies for AWS the backup-gateway and primary gateway, then the destination selects the individual route within the route table to control. The local-cloud-interface then points to the interface ID eni-867caa86 (from AWS perspective) of the vEOS router that the traffic should be directed.
switch(config)#cloud high-availability
switch(config-cloud-ha)#peer veos2
switch(config-cloud-ha-peer-veos2)#aws
switch(config-cloud-ha-peer-veos2-aws)#backup-gateway rtb-40b72d24
0.0.0.0/0 local-cloud-interface eni-867caa86
switch(config-cloud-ha-peer-veos2-aws)#primary-gateway rtb-2843124c
0.0.0.0/0 local-cloud-interface eni-867caa86Explicit Credential Configuration
The explicit credential should be configured as shown below.
switch(config)#cloud provider aws
switch(config-cloud-aws)#region us-west-1
switch(config-cloud-aws)#access-key 0 MYEXAMPLESECRETKEY
switch(config-cloud-aws)#secret access-key 0 MYEXAMPLESECRETKEY
switch(config-cloud-aws)#exit
switch(config-cloud)#exit
Azure
- SDK Auth Credentials
To generate SDK Auth Credentials, use the sdk authentication credential-file flash:startup-config command in the config-cloud-azure configuration mode.
switch(config)#cloud provider azure switch(config-cloud-azure)#sdk authentication credential-file flash:startup-config - Active Directory Credentials
The following example places the vEOS router into the config-cloud-azure configuration mode and sets the active directory credentials.
switch(config)#cloud provider azure switch(config-cloud-azure)#active-directory credential email subscription-id ef16892c-aa46-4aba-ae9a-d4fhsb1c612c
Configuring BFD
switch(config)#cloud high-availability
switch(config-cloud-ha)#peer veos2
switch(config-cloud-ha-peer-veos2)#bfd source-interface tunnel 2 single-hopConfiguring the Recovery Time
The recovery wait-time command in the cloud-ha configuration sub-mode configures the amount of time to take back control of local route tables after failure recovery. The following example shows the wait time is configured to 90 seconds.
switch(config-cloud-ha-peer-veos2)#recovery wait-time 90Cloud Provider Helpful Tips
The following are needed for Cloud High Availability but are not part of the CloudEOS and vEOS configuration on the CloudEOS and vEOS Router. These may change or can be another way to achieve the same effect without changing the CloudEOS and vEOS Router.
AWS VPN Specific Cloud PrivateLink
AWS VPN Specific Cloud PrivateLink allows a private (no public IP address) CloudEOS and vEOS instance to access services offered by AWS (without using proxy).
The interface VPC endpoints enables a private CloudEOS and vEOS instance to connect to AWS VPN Specific Cloud PrivateLink.
To configure Interface VPC Endpoints:
- Open the Amazon VPC console and choose Endpoints in the navigation panel.
- Select Create Endpoint.
- Choose the AWS Services and select service name com.amazonaws.<your-region>.ec2.
- Choose the VPC and the subnets in each availability zone for the Interface VPC endpoints.
- Enable private DNS name and set security group accordingly.
- Select Create Endpoint.
Once the Endpoint(s) is created, the EC2 API IP associated with the domain-name will be updated to the endpoint IP.
Additional interface VPC endpoints information can be found at: https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpce-interface.html
Configuring Cloud High Availability
To enable the Cloud HA and its parameters, use the following configurations.
Enable Cloud High Availability
The cloud high-availability command places the CloudEOS and vEOS in the cloud-ha configuration mode. This example enables cloud high-availability and configures the peer veos2.
switch(config)#cloud high-availability
switch(config-cloud-ha)#no shutdown
switch(config-cloud-ha)#peer veos2
switch(config-cloud-ha-peer-veos2)#AWS Specific for High Availability
This is applicable only when running in AWS cloud environment and configures various aspects of Cloud HA feature to interact with AWS web services.
Note: The access-key-id and secret access-key commands are either both configured or both are omitted. If omitted, the Cloud HA Agent will try to use AWS IAM role for security tokens to access and control AWS route tables. Verify the IAM role for the CloudEOS and vEOS router Virtual Machine( VM ) is configured properly on the AWS cloud. Refer to AWS documentation to configure IAM role.
switch(config)#
switch(config)#cloud provider aws
switch(config-cloud-aws)#access-key 0 ATPAILIL5E982IPT7P3R
switch(config-cloud-aws)#secret access-key 0 M0RRUtAA8I8wYxJB8
switch(config-cloud-aws)#region us-west-1
switch(config-cloud-aws)#proxy testConfigure the backup-gateway, primary-gateway, Route Table ID(rtb) and local interface for AWS.
The Route Table ID specifies for AWS the backup-gateway and primary gateway, then the destination selects the individual route within the route table to control. The local-cloud-interface then points to the interface ID eni-867caa86 (from AWS perspective) of the vEOS router that the traffic should be directed.
AWS
switch(config)#cloud high-availability
switch(config-cloud-ha)#peer veos2
switch(config-cloud-ha-peer-veos2)#aws
switch(config-cloud-ha-peer-veos2-aws)#backup-gateway rtb-40b72d24
0.0.0.0/0 local-cloud-interface eni-867caa86
switch(config-cloud-ha-peer-veos2-aws)#primary-gateway rtb-2843124c
0.0.0.0/0 local-cloud-interface eni-867caa86Azure Specific for High Availability
Azure
- SDK Auth Credentials
To generate SDK Auth Credentials, use the sdk authentication credential-file flash:startup-config command in the config-cloud-azure configuration mode.
switch(config)#cloud provider azure switch(config-cloud-azure)# sdk authentication credential-file flash:startup-config - Active Directory Credentials
The following example places the vEOS router into the config-cloud-azure configuration mode and sets the active directory credentials.
switch(config)#cloud provider azure switch(config-cloud-azure)#active-directory credential email subscription-id ef16892c-aa46-4aba-ae9a-d4fhsb1c612c
Configure the backup-gateway, primary-gateway, Route Table ID (rtb), resource-group and next-hop for Azure
The resource group specified is the one which contains the route table referenced beneath it. The nextHopIp is the IP of the vEOS Router interface that traffic should be directed.
Azure
switch(config)#cloud high-availability
switch(config-cloud-ha)#peer veos2
switch(config-cloud-ha-peer-veos2)#azure
switch(config-cloud-ha-peer-veos2-azure)#backup-gateway Subnet-2-vEOS-RouteTable 0.0.0.0/0 10.1.1.4 resource-group my_resource_group_64f86970ffe24ab GCP Specific for High Availability
In GCP specific cloud configuration, you must specify a Project. The GCP cloud uses either the Default Credentials, or a Service Account mode for authorization.
- Default Credentials: Cloud HA uses the Application Default Credentials for authorization and no configuration is required. Setup a GCP instance with a service account with the requisite permissions. For more information on creating an instance with a service account or changing the service account of an existing instance, refer: https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances
- Service Account: Specify a service account file. For more information on creating a service account file, refer:https://cloud.google.com/iam/docs/creating-managing-service-account-keys#creating_service_account_keys
{
"type": "service_account",
"project_id": "project-id",
"private_key_id": "key-id",
"private_key": "-----BEGIN PRIVATE KEY-----\nprivate-key\n-----END PRIVATE KEY-----\n",
"client_email": "service-account-email",
"client_id": "client-id",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://accounts.google.com/o/oauth2/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/service-account-email"
}For Default Credentials or Service Account authorization model, the role associated with the CloudEOS instance/service account must have the following permissions:
- compute.networks.get
- compute.networks.updatePolicy
- compute.projects.get
- compute.routes.create
- compute.routes.delete
- compute.routes.get
- compute.routes.list
Example
- Cloud HA GCP configuration example.
cloudEos(config)# cloudEos(config)#cloud provider gcp cloudEos(config-cloud-gcp)#project gcp-project-name cloudEos(config-cloud-gcp)#service-account file flash:.gcp_service_account.json cloudEos(config-cloud-gcp)#proxy test
Specify a destination prefix for Cloud HA routes on GCP to select the individual routes that we wish to control. Since route table is not present in GCP, you can specify an optional tag for each route to simulate the route table.
cloudEos(config)#cloud high-availability
cloudEos(config-cloud-ha)#peer veos2
cloudEos(config-cloud-ha-peer-veos2)#gcp
cloudEos(config-cloud-ha-peer-veos2-gcp)#backup-gateway 0.0.0.0/0 tag tag2
cloudEos(config-cloud-ha-peer-veos2-gcp)#primary-gateway 0.0.0.0/0 tag tag1Limitations
- For Cloud HA on AWS/Azure, pre-create the HA routes before configuring these routes on a CloudEOS instance. For GCP, the HA routes are created automatically by Cloud HA, and thus no need to pre-create any HA routes to avoid route conflicts.
- For GCP, though the HA routes are automatically created by Cloud HA, they do not get deleted automatically from GCP when removed from the Cloud HA configuration from CloudEOS instance. Hence, you need to delete these routes from GCP manually.
- Cloud HA on GCP allows adding routes only in the VPC network corresponding to the first interface (nic0) of the CloudEOS instance.
- Since routes in GCP are added at a per VPC level, specify the tags used to simulate route tables when adding a Cloud HA route using the primary-gateway / backup-gateway command. Use the same tag to apply manually on the Cloud HA routes to a subset of the instances in the VPC network.
CloudEOS HA GCP provider commands
- cloud provider gcp
- project
- service-account
- primary-gateway
- backup-gateway
- show cloud high-availability routes
- show cloud provider gcp
cloud provider gcp
The cloud provider gcp command places the switch in cloud-provider-gcp configuration mode, and allows you to configure cloud provider gcp command parameters. The exit command returns the CloudEOS to global configuration mode.
Global Configuration
Command Syntax
cloud provider gcp
- These commands places the cloud provider for GCP into the configuration mode.
cloudEos#config cloudEos(config)#cloud provider gcp cloudEos(config-cloud-gcp)# - The exit command returns to the global configuration mode.
cloudEos(config-cloud-gcp)#exit cloudEos(config)#
project
The project command specifies the GCP project name. The no project command removes the configuration from the CloudEOS running-config.
Global Cloud Provider GCP Configuration
Command Syntax
project name
no project name
- name Specifies the selected GCP project name.
- These commands configures the GCP project.
cloudEos(config)#cloud provider gcp cloudEos(config-cloud-gcp)#project test-project - The no project command removes the GCP project.
cloudEos(config)#cloud provider gcp cloudEos(config-cloud-gcp)#no project test-project
service-account
The service-account specifies the service account file when using the Service Account authorization model. The no service-account command removes the configuration from the CloudEOS running-config.
Global Cloud Provider GCP Configuration
Command Syntax
service-account file sa-file
no service-account file sa-file
- sa-file Specifies the path to the service account file.
- These commands configures the Service Account file used.
cloudEos(config)#cloud provider gcp cloudEos(config-cloud-gcp)#service-account file flash:.gcp_service_account.json cloudEos(config-cloud-gcp)# - The no service-account command removes the Service Account file configuration.
cloudEos(config)#cloud provider gcp cloudEos(config-cloud-gcp)#no service-account file flash:.gcp_service_account.json cloudEos(config-cloud-gcp)#
primary-gateway
The primary-gateway command in the cloud-ha submode adds a primary high availability route for GCP. The no primary-gateway command removes the route configuration from the CloudEOS and vEOS running-config.
Cloud HA GCP Configuration Submode
Command Syntax
primary-gateway dest-prefix [tag rt-tag]
no primary-gateway dest-prefix [tag rt-tag]
- dest-prefix Specifies the destination IP prefix.
- rt-tag Specifies the route tag.
- These commands configures a primary high availability route.
cloudEos(config)#cloud high-availability cloudEos(config-cloud-ha)#peer veos2 cloudEos(config-cloud-ha-peer-veos2)#gcp cloudEos(config-cloud-ha-peer-veos2-azure)#primary-gateway 10.1.0.0/16 tag tag1 - The no primary-gateway command removes the primary high availability route configuration.
cloudEos(config)#cloud high-availability cloudEos(config-cloud-ha)#peer veos2 cloudEos(config-cloud-ha-peer-veos2)#gcp cloudEos(config-cloud-ha-peer-veos2-azure)#no primary-gateway 10.1.0.0/16 tag tag1
backup-gateway
The backup-gateway command in the cloud-ha submode adds a backup high availability route for GCP. The no backup-gateway command removes the route configuration from the CloudEOS and vEOS running-config.
Cloud HA GCP Configuration Submode
Command Syntax
backup-gateway dest-prefix [tag rt-tag]
no backup-gateway dest-prefix [tag rt-tag]
- dest-prefix Specifies the destination IP prefix.
- rt-tag Specifies the route tag.
- These commands configures a backup high availability route.
cloudEos(config)#cloud high-availability cloudEos(config-cloud-ha)#peer veos2 cloudEos(config-cloud-ha-peer-veos2)#gcp cloudEos(config-cloud-ha-peer-veos2-azure)#backup-gateway 10.1.0.0/16 tag tag2 - The no primary-gateway command removes the backup high availability route configuration.
cloudEos(config)#cloud high-availability cloudEos(config-cloud-ha)#peer veos2 cloudEos(config-cloud-ha-peer-veos2)#gcp cloudEos(config-cloud-ha-peer-veos2-azure)#no backup-gateway 10.1.0.0/16 tag tag2
show cloud high-availability routes
The show cloud high-availability routes commanddisplays the configured local or peer route table, destination IP address and local Next Hop Interface.
EXEC
Command Syntax
show cloud high-availability routes
- The show cloud high-availability routes command displays high availability routes information.
cloudEos(config)#show cloud high-availability routes Peer Route TypeDestination Tag ------ ------------ -------------- ------ peer primary21.2.3.14/24 peer primary1.2.3.4/21 tag1 peer backup 1.2.3.4/21 tag2 peer backup11.2.3.4/21 az1
show cloud provider gcp
The show cloud provider gcp command displays cloud provider information for the GCP platform.
EXEC
Command Syntax
show cloud provider gcp
- The show cloud provider gcp command displays the GCP cloud configuration for Default Credentials authorization model. .
cloudEos(config)#show cloud provider gcp Project: project-123 Authentication mode: default credentials Service account file: Proxy: myProxyName - The following example displays the GCP cloud configuration for Service Account authorization model.
cloudEos(config)#show cloud provider gcp Project: project-123 Authentication mode: service account Service account file: flash:.gcp_service_account.json Proxy: myProxyName
Full Configurations
AWS VPN Specific Cloud Full Configuration
The following AWS configuration is valid for use with the IAM role.
cloud provider aws
region us-west-1
!
cloud high-availability
no shutdown
!
peer veos2
aws
backup-gatewayrtb-40b72d24 0.0.0.0/0 local-cloud-interface eni-26cb1d27
backup-gatewayrtb-17b32973 0.0.0.0/0 local-cloud-interface eni-1589e714
backup-gatewayrtb-54503330 0.0.0.0/0 local-cloud-interface eni-56cf1957
primary-gateway rtb-a4be24c0 0.0.0.0/0 local-cloud-interface eni-26cb1d27
primary-gateway rtb-40b72d24 0.0.0.0/0 local-cloud-interface eni-56cf1957
primary-gateway rtb-63b02a07 0.0.0.0/0 local-cloud-interface eni-1589e714
peer address 10.2.201.149
recovery wait-time 5
bfd source-interface Ethernet1
!
Azure Full Configuration
The following Azure configuration is valid for the MSI.
cloud high-availability
no shutdown
!
peer veos2
azure
backup-gateway Subnet-2-vEOS-RouteTable 0.0.0.0/0 10.1.2.4 resource-group CloudHaAzure
backup-gateway Subnet-2-vEOS-RouteTable 10.1.0.0/16 10.1.2.4 resource-group CloudHaAzure
backup-gateway Subnet-3-vEOS-RouteTable 10.1.0.0/16 10.1.3.4 resource-group CloudHaAzure
backup-gateway Subnet-3-vEOS-RouteTable 0.0.0.0/0 10.1.3.4 resource-group CloudHaAzure
primary-gateway Subnet-1-vEOS-RouteTable 10.1.0.0/16 10.1.1.4 resource-group CloudHaAzure
primary-gateway Subnet-1-vEOS-RouteTable 0.0.0.0/0 10.1.1.4 resource-group CloudHaAzure
peer address 10.1.0.5
recovery wait-time 10
bfd source-interface Ethernet1 Cloud HA GCP Full Configuration
- The following GCP configuration uses default credentials.
CloudEOS - 1
cloud provider gcp project gcp-project-name ! cloud high-availability no shutdown ! peer cloudEOS2 gcp backup-gateway 0.0.0.0/0 tag tag1 backup-gateway 0.0.0.0/0 tag tag2 primary-gateway 0.0.0.0/0 tag tag3 primary-gateway 0.0.0.0/0 tag tag4 peer address 10.3.0.59 recovery wait-time 5 bfd source-interface Ethernet1 ! - CloudEOS - 2
cloud provider gcp project gcp-project-name ! cloud high-availability no shutdown ! peer cloudEOS1 gcp backup-gateway 0.0.0.0/0 tag tag3 backup-gateway 0.0.0.0/0 tag tag4 primary-gateway 0.0.0.0/0 tag tag1 primary-gateway 0.0.0.0/0 tag tag2 peer address 10.3.0.59 recovery wait-time 5 bfd source-interface Ethernet1 !
General Troubleshooting Tips
If the Cloud HA feature is not working as expected, follow these tips for debugging.
- Make sure that the network connectivity is there and DNS server is setup correctly for this feature to work.
- If using Proxy and IAM role under AWS, make sure that the HTTP traffic (TCP port 80) is not proxied to allow for temporarily security credentials to be retrieved by CloudEOS and vEOS instance.
- Make sure to use a corresponding BFD source interface on the peer CloudEOS and vEOS instance. This makes sure that the BFD traffic ingress and egress are on the same interface on each instance.
- For an AWS Specific Cloud, if the IAM role does not work, Arista recommends temporarily using access-key id and secret access key with enough permissions to make sure the rest of the Cloud HA configuration is fine until you debug IAM role policy.
Caveats and Limitations
- This feature was introduced in EOS release 4.20.5F which uses /mnt/flash/cloud_ha_config.json file for Cloud HA configuration without any CLI support. Starting from release 4.20.5.A1 onwards, Cloud HA feature supports CLI based configuration only. Deployments using JSON based configurations are not supported and will not work when the image is upgraded or downgraded. To upgrade image, the administrator must configure Cloud HA feature manually by converting the JSON configuration to equivalent CLI configuration. Downgrading will work as long as the older JSON file is still present in /mnt/flash directory.
- Only a single resource-group is supported across all routing entries for Azure under Cloud specific config HA configuration.
- The Cloud HA feature currently supports only a single peer.
- The AWS IAM role or Azure MSI needs to be configured properly using cloud provider's management tools and should give sufficient permissions to CloudEOS and vEOS instance to access and update route table entries.
- The CloudEOS and vEOS instance should have connectivity to the cloud provider's web services. The access can also be via proxy or using feature like AWS private-link.
- The recovery wait-time should not be configured less than 10 seconds to avoid unnecessary route flapping when experiencing periodic instabilities.
- The Cloud HA feature completely validates all the provided cloud configuration to make sure it is consistent and has all required permissions. However, the administrator should not change the provider's network configuration afterwards to avoid any issues during fail-over.
- When there are BFD connectivity issues between the two CloudEOS and vEOS peers, each instance will take over the other's traffic. This cross traffic forwarding on provider's network should not have any adverse affect and still work as active-active even though both of the instance will report as fail-over. After the network connectivity is resolved, the traffic pattern reverts to the normal active-active mode.
- The user can adjust the BFD specific parameters for the session used by Cloud HA feature using normal BFD commands such as multiplier, tx/rx intervals, etc. The Cloud HA fail-over and traffic takeover time is directly correlated with BFD failure detection time. However, when using an overly aggressive BFD, the failover time may incur higher overhead as well may result in greater instability during traffic bursts. Arista recommends using the use default BFD interval which is currently 300 msec with a multiplier of 3.
- The bfd source-interface used in Cloud HA configuration should not belong and/or routable via the route-tables controlled by the CloudEOS and vEOS router instance itself to avoid traffic looping issues.
- If the Cloud HA is in an invalid configuration state due to erroneous/mismatched configuration in the provider's cloud, the administrator has to force update the Cloud HA configuration (for example, by shut/no shut under Cloud HA mode) after updating the provider's cloud configuration. In other words, by itself, the Cloud HA feature will not retry the back-end configuration check if it is found to be invalid at the time of configuration.
Cloud High Availability Commands
Global
Cloud Provider Commands
Global
Cloud Proxy Commands
Global
InterfaceShow Commands
EXEC
Cloud High Availability CLIs
The Cloud High Availability CLIs are divided into three separate configuration modes:
• Cloud Proxy - For proxy related configuration such as http and https.
• Cloud Provider - For cloud provider specific configuration such as region, credential, and proxy name.
• Cloud High-Availability - For configurations such as route, next-hop, BFD source interface, and peer.
access-key-id (CloudEOS and vEOS-AWS)
The cloud provider AWS command places the CloudEOS and vEOS in cloud-provider-aws configuration mode. This configuration mode allows user to configure cloud provider aws access-key-id command parameters. The no access-key-id command removes the configuration from the CloudEOS and vEOS running-config. The exit command returns the CloudEOS and vEOS to global configuration mode.
Command Mode
Cloud Provider AWS Configuration
Command Syntax
access-key-id Password_Type
no access-key-id Password_Type
Parameters
Password_Type
- 0 access-key-id The password is a clear-text string. Equivalent to no parameter.
-
7 encrypted_key The password is an encrypted string.
- Text
Example
The following example configures the AWS access key to encrypted.
switch(config)#cloud provider aws
switch(config-cloud-aws)#access-key 0 565656 test Example
The following example removes the AWS access key and returns the vEOS to Global configuration mode.
switch(config-cloud-aws)#access-key 0 565656 test
switch(config-cloud-aws)#no access-key 0 565656 test
switch(config)# Example
The following example returns the vEOS to Global configuration mode.
switch(config-cloud-aws)#access-key 0 565656 test
switch(config-cloud-aws)#exit
switch(config)#active-directory credential email subscription-id (CloudEOS and vEOS-Azure)
The active-directory credential email subscription-id command configures Azure's cloud provider azure active-directory credential parameters. The no active-directory command removes the configuration from the CloudEOS and vEOS running-config. The exit command returns the CloudEOS and vEOS to global configuration mode.
Command Mode
Cloud Provider Azure Configuration
Command Syntax
active-directory credential email subscription-id ID
no active-directory credential email subscription-id
Parameters
- ID Defines the active directory subscription ID.
Example
The following example places the cloud provider for Azure into the configuration mode.
switch(config)#cloud provider azure
switch(config-cloud-azure)#active-directory credential email subscription-idExample
switch(config)#cloud provider azure
switch(config-cloud-azure)#active-directory credential email subscription-idazure (CloudEOS and vEOS - Azure)
The azure command in the cloud-ha-peer configuration sub-mode, accessible through the cloud-ha configuration mode, allows the user to configure cloud high-availability peer related parameters. The exitcommand returns the CloudEOS and vEOS to the to the cloud-ha-peer configuration mode.
Command Mode
Global Cloud High Availability Peer Configuration Submode
Command Syntax
azure
Example
The following example configures the peer related information for Azure.
switch(config)#cloud high-availability
switch(config-cloud-ha)#peer p
switch(config-cloud-ha-peer-veos2)#azure
switch(config-cloud-ha-peer-veos2-azure)#Example
The following example returns the CloudEOS and vEOS to the cloud-ha configuration mode.
switch(config-cloud-ha-peer-veos2-azure)#exit
switch(config-cloud-ha-peer-veos2)#backup-gateway (CloudEOS and vEOS - Azure)
The cloud high-availability command in the cloud-ha submode assigns the backup gateway parameters for the Azure high availability peered cloud. The no backup-gateway command removes the configuration from the CloudEOS and vEOS running-config. The exit command returns the CloudEOS and vEOS to global configuration mode.
Command Mode
Cloud HA azure configuration submode
Command Syntax
backup-gateway [Azure Rt_Info] resource-group [Name]
no backup-gateway
Parameters
- Azure Rt_Info
- azure-rt-name The azure route name.
- dest-ip-address/mask The destination IP address.
- local-ip-address The local IP address.
- resource-group
- Name Azure resource group name.
Example
The following example configures the parameters for the Azure high availability peered cloud.
switch(config)#cloud high-availability
switch(config-cloud-ha)#peer veos2
switch(config-cloud-ha-peer-veos2)#azure
switch(config-cloud-ha-peer-veos2-azure)#backup-gateway Rt1 10.10.1.1/10 1.1.1.1 resource-group test Example
The following example removes the backup-gateway parameters for the Azure high availability peered cloud.
switch(config-cloud-ha-peer-veos2-azure)#no backup-gateway Rt1 10.10.1.1/10
switch(config-cloud-ha-peer-veos2-azure)#bfd source-interface (CloudEOS and vEOS)
The bfd source-interface command in the cloud-ha configuration submode configures BFD source interface parameters for the high availability peer. The no bfd source-interface command removed the BFD configurations from the CloudEOS and vEOS running-config.
Command Mode
Global Cloud HA peer configuration mode
Command Syntax
bfd source-interface [Interface_Type] single-hop
no bfd source-interface
Parameters
- Interface_Type
- Ethernet Ethernet Port number <1-4>.
- Loopback Loopback interface <0-1000>.
- Tunnel Tunnel interface <0-255>.
- Single-hop Single hop BFD . Default is multi-hop.
Example
The following example configures Ethernet 1 as the source interface for BFD and multi-hop set as the default .
switch(config)#cloud high-availability
switch(config-cloud-ha)#peer veos2
switch(config-cloud-ha-peer-veos2)#bfd source-interface ethernet 1 Example
The following example configures Tunnel 2 as a single hop the source interface for BFD.
switch(config)#cloud high-availability
switch(config-cloud-ha)#peer veos2
switch(config-cloud-ha-peer-veos2)#bfd source-interface tunnel 2 single-hop Example
The following example removes the BFD configuration.
switch(config-cloud-ha-peer-veos2)#no bfd source-interfacecloud high-availability peer
Configures one peer at a time into high availability.
Command Mode
Global Cloud HA configuration submode
Command Syntax
config-cloud-ha-peer <peer name>
Example
The following example configures the peer and places it in the cloud high availability configuration mode.
switch(config)#cloud high-availability peer peer1
switch(config-cloud-ha-peer-peer1)#cloud high-availability shutdown (CloudEOS and vEOS)
The shutdown command in the cloud-ha configuration mode disables High Availability for virtual EOS instances running in the cloud environment.
Command Mode
Cloud High Availability configuration
Command Syntax
shutdown
Example
The following example configures the peer and places it in the cloud high availability configuration mode.
switch(config)#cloud high-availability
switch(config-cloud-ha)#shutdowncloud provider aws (CloudEOS and vEOS)
The cloud provider aws command places the CloudEOS and vEOS in cloud-provider-aws configuration mode. This configuration mode allows user to configure cloud provider aws command parameters. The exit command returns the CloudEOS and vEOS to global configuration mode.
Command Mode
Global Configuration
Command Syntax
cloud provider aws
Example
The following example places the cloud provider for AWS into the configuration mode.
switch#config
switch(config)#cloud provider aws
switch(config-cloud-aws)#Example
The following example returns to the global configuration mode.
switch(config-cloud-aws)#exit
switch(config)#cloud provider azure (CloudEOS and vEOS)
The cloud provider azure command places the CloudEOS and vEOS in cloud-provider-azure configuration mode. This configuration mode allows user to configure cloud provider azurecommand parameters. The exit command returns the CloudEOS and vEOS to global configuration mode.
Command Mode
Global Configuration
Command Syntax
cloud provider azure
Example
switch(config)#cloud provider azure
switch(config-cloud-azure)#cloud proxy (CloudEOS and vEOS)
The cloud proxy command places the CloudEOS and vEOS in cloud-proxy configuration mode. This configuration mode allows user to configure the cloud proxy command parameters. The no cloud proxy command disables the named proxy and returns the CloudEOS and vEOS to global configuration mode.
Command mode
Global Configuration
Command Syntax
cloud proxy proxy_name
no cloud proxy proxy_name
Parameters
proxy_name The proxy name to configure.Example
The following example configures the cloud proxy configuration setting for "test".
switch(config)#
switch(config)#cloud proxy test
switch(config-cloud-proxy-test)#Example
This command disables the cloud proxy named "test" and returns the vEOS to global configuration mode.
switch(config-cloud-proxy-test)# no cloud proxy test
switch(config)#http (CloudEOS and vEOS)
The http command in the cloud-proxy configuration submode configures the IP, port, username, and password parameters. The no http command removes the configured cloud proxy information for HTTP from the running-config and returns the CloudEOS and vEOS to the global configuration mode.
Command mode
Global Cloud Proxy Configuration
Command Syntax
http [proxy_IP_port] [username] [password]
no http [proxy_IP_port] [username] [password]
Parameters
- proxy_IP_port Port number to be used for the HTTP server. Options include:
- proxy-ip IP address used for the HTTPs proxy. Dotted decimal location.
- proxy_port HTTPS proxy port. Value ranges from 1 to 65535.
- username Name string.
- password Password string.
- 0 cleartext-passwd Indicates the cleartext password is in clear text. Equivalent to the no parameter case.
- 7 encrypted_passwd Indicates encrypted password is md5 encrypted.
Example
The following example configures the cloud proxy IP, port and username and password for HTTP.
switch(config)#cloud proxy test
switch(config-cloud-proxy-test)# http 1.2.3.4 1234 username test password 7 075E731F1A
switch(config-cloud-proxy-test)#Example
The following example removes the configured cloud proxy information for HTTP from the running-config.
switch(config-cloud-proxy-test)# no http 1.2.3.4 1234 username test password 7 075E731F1A
switch(config-cloud-proxy-test)#https (CloudEOS and vEOS)
The https command in the command in the cloud-proxy configuration submode configures the IP, port, username and password parameters. The no https command removes the configured cloud proxy information for HTTPS from the running-config and returns the CloudEOS and vEOS to global configuration mode.
Command mode
Global Cloud Proxy Configuration
Command Syntax
https [[proxy_IP_port] [username]] [password]
no https [[proxy_IP_port] [username]] [password]
Parameters
- proxy_IP_port Port number to be used for the HTTP server. Options include:
- proxy-ipIP address used for the HTTPs proxy. Dotted decimal location.
- proxy_port HTTPS proxy port. Value ranges from 1 to 65535.
- username Name string.
- password Password string.
- 0 cleartext-passwd Indicates the cleartext password is in clear text. Equivalent to the no parameter case.
- 7 encrypted_passwd Indicates encrypted password is md5 encrypted.
Example
The following example configures the cloud proxy IP and port for HTTPS.
switch(config)#
switch(config)#cloud proxy test
switch(config-cloud-proxy-test)#https 10.3.255.155 8888Example
The following example removes the configured cloud proxy HTTPS information from the running-config.
switch(config-cloud-proxy-test)#no https 10.3.255.155 8888
switch(config-cloud-proxy-test)#peer (CloudEOS and vEOS)
The peer command in the cloud-ha configuration mode identifies which peer to configure by name. The peer command in the cloud-ha configuration submode configures the cloud high-availability resource group peer related parameters. The no peer command removes the configuration from the CloudEOS and vEOS running-config. The exitcommand returns the CloudEOS and vEOS to the cloud-ha configuration mode.
Command Mode
Cloud High Availability Configuration
Cloud High Availability Configuration Submode
Command Syntax
peer ip-address
no peer ip-address
Parameters
- IP-addressThe peer IP address.
Example
The following example configures the cloud high availability peer.
switch(config-cloud-ha)#peer veos2
switch(config-cloud-ha-peer-veos2)#Example
The following example configures the peer IP address as 10.10.10.149.
switch(config)#cloud high-availability
switch(config-cloud-ha)#peer veos2
switch(config-cloud-ha-peer-veos2)#peer 10.10.10.149 Example
The following example removes the peer IP address from the vEOS running-config.
switch(config)#cloud high-availability
switch(config-cloud-ha)#peer veos2
switch(config-cloud-ha-peer-veos2)#no peer 10.10.10.149primary gateway (CloudEOS and vEOS - Azure)
The primary-gateway command in the cloud-ha submode assigns the primary gateway parameters for the Azure high availability peered cloud. The no primary-gateway command removes the configuration from the CloudEOS and vEOS running-config.
Command Mode
Cloud HA Azure Configuration Submode
Command Syntax
primary-gateway [Azure Rt_Info] resource-group [Name]
no primary-gateway [Azure Rt_Info]
Parameters
- Azure Rt_Info
- azure-rt-name The azure route name.
- dest-ip-address/mask The destination IP address.
- local-ip-address The local IP address.
- resource-group
- Name Azure resource group name.
Example
The following example configures the parameters for the Azure high availability peered cloud.
switch(config)#cloud high-availability
switch(config-cloud-ha)#peer veos2
switch(config-cloud-ha-peer-veos2)#azure
switch(config-cloud-ha-peer-veos2-azure)#primary-gateway Rt1 10.10.1.1/10 1.1.1.1 resource-group test Example
The following example removes the primary-gateway parameters for the Azure high availability peered cloud.
switch(config-cloud-ha-peer-veos2-azure)#no primary-gateway Rt1 10.10.1.1/10
switch(config-cloud-ha-peer-veos2-azure)#proxy (CloudEOS and vEOS)
The proxy command configures the cloud provider aws proxy. The no proxy command removes the configuration from the running-config. The exit command returns the CloudEOS and vEOS to global configuration mode.
Command Mode
Global Cloud AWS Configuration
Command Syntax
proxy <proxy_name>
no proxy <proxy_name>
Parameters
- proxy_name Proxy name to configure.
Example
The following example configures the Azure cloud proxy named "test".
switch(config)#cloud provider aws
switch(config-cloud-aws)#proxy testrecovery (cloud HA peer)
The recovery wait-time command in the cloud-ha-peer configuration submode defines the amount of time, in seconds, to take control of the local route tables after failure recovery.
Command Mode
Cloud HA peer configuration
Command Syntax
recovery wait-time <time-in-secs>
no recovery wait-time <time-in-secs>
Parameters
- time-in-secs The defined amount of time to take back control of local route tables after failure recovery. Default is 30 seconds.
Example
The following example configures the recovery wait time to 90 seconds.
switch(config)#cloud ha
switch(config-cloud-ha)#p1
switch(config-cloud-ha-p1)#recovery wait-time 90recovery wait-time (CloudEOS and vEOS)
The recovery wait-time command in the cloud-ha configuration sub-mode allows takes back control of local route tables after failure recovery. The no recovery wait-time command removes the configuration from the CloudEOS and vEOS running-config. Default is set at 30 seconds.
Command Mode
Cloud High Availability peer configuration
Command Syntax
recovery wait-time <period>
no recovery wait-time <period>
default recovery wait-time <period>
Parameters
- periodThe defined amount of time to take back control of local route tables after failure recovery. Default is 30 seconds.
Example
The following example shows the wait time is configured to 90 seconds.
switch(config-cloud-ha-peer1)#recovery wait-time 90Example
The following example removes the configured the wait time.
switch(config-cloud-ha-peer1)#no recovery wait-time Example
The following example configures the wait time to the default of 30 seconds.
switch(config-cloud-ha-peer1)#default recovery wait-timeregion (CloudEOS and vEOS - AWS)
The cloud provider aws command places the CloudEOS and vEOS in cloud-provider-aws configuration mode. This configuration mode allows user to configure AWS cloud provider region command parameters. The no region command removes the configuration from the CloudEOS and vEOS running-config. The exit command returns the CloudEOS and vEOS to global configuration mode.
Command Mode
Global Cloud Provider AWS Configuration
Command Syntax
region aws-region
no region aws-region
Parameters
- aws-region Specifies the selected region.
Example
The following example configures the cloud provider AWS region.
switch(config)#cloud provider aws
switch(config-cloud-aws)#region us-west-1
switch(config-cloud-aws)#Example
The following example removes the cloud provider AWS region.
switch(config)#cloud provider aws
switch(config-cloud-aws)#no region us-west-1
switch(config-cloud-aws)#secret-access_key (CloudEOS and vEOS - AWS)
The cloud provider aws command places the CloudEOS and vEOS incloud-provider-aws configuration mode. This configuration mode allows user to configure cloud provider aws secret access-key command parameters. The no secret access-key command removes the configuration from the CloudEOS and vEOS running-config. The exit command returns the CloudEOS and vEOS to global configuration mode.
Command Mode
Global Cloud Provider AWS configuration
Command Syntax
secret access-key password_type
no secret access-key password_type
Parameters
- 0 access-key-idThe password is a clear-text string. Equivalent to no parameter.
- 7 encrypted_key The password is an encrypted string.
- Text
Example
The following example configures the AWS secret access key.
switch(config)#cloud provider aws
switch(config-cloud-aws)#secret access-key 0 565656 test
switch(config-cloud-aws)#Example
The following example removes the secret access key from the CloudEOS and vEOS running-config.
switch(config-cloud-aws)#no secret access-key 0 565656 test
switch(config-cloud-aws)#Example
The following example returns the vEOS to Global configuration mode.
switch(config-cloud-aws)#secret access-key 0 565656 test
switch(config-cloud-aws)#exit
switch(config)#show cloud high-availability (CloudEOS and vEOS)
The show cloud high-availability command displays the high availability configured settings.
Command Mode
EXEC
Command Syntax
show cloud high-availability
Example
This command displays details and status of the cloud high-availability configuration.
switch#show cloud high-availability
Cloud HA Configuration:
Peer address: 10.2.201.149
Source interface: Ethernet1
Enabled : True
Failover recovery time: 5
Status: valid
State : ready
Last failover time: never
Last recovery time: never
Last config validation start time : 0:26:08 ago
Last config validation end time : 0:26:06 ago
Failovers : 0show cloud high-availability routes
The show cloud high-availability routes command displays the configured local or peer route table, destination IP address and local Next Hop Interface.
Command Mode
EXEC
Command Syntax
show cloud high-availability routes
Example
The example below displays high availability routes information.
switch(config)#show cloud high-availability routes
Peer RouteType RouteID Destination Next Hop Interface
----------- ----------- -------------- ---------- ------------
veos6 primary rtb-1dc75679 0.0.0.0/0eni-e61d95e7
veos6 primary rtb-a29617c6 0.0.0.0/0eni-69109868
veos6 primary rtb-acc756c8 0.0.0.0/0eni-7f1d957e
veos6 backuprtb-43c65727 0.0.0.0/0eni-7f1d957e
veos6 backuprtb-71c65715 0.0.0.0/0eni-e61d95e7
veos6 backuprtb-aca223c8 0.0.0.0/0eni-69109868show cloud provider aws (CloudEOS and vEOS - AWS)
The show cloud provider aws command displays cloud provider information for the AWS platform.
Command Mode
EXEC
Command Syntax
show cloud provider aws
Example
The following example displays the AWS cloud configuration.
switch#show cloud provider aws
Cloud AWS Configuration
Region : us-west-1
Access key ID:
Access secret key:
Proxy: testExample
The following example displays the primary and backup gateway information for the AWS cloud provider.
switch#show run section cloud
cloud provider aws
us-west-1
proxy test
!
cloud high-availability
no shutdown
!
peer vEOS12
aws
backup-gatewayrtb-40b72d24 0.0.0.0/0 local-cloud-interface eni-26cb1d27
backup-gatewayrtb-17b32973 0.0.0.0/0 local-cloud-interface eni-1589e714
backup-gatewayrtb-54503330 0.0.0.0/0 local-cloud-interface eni-56cf1957
primary-gateway rtb-a4be24c0 0.0.0.0/0 local-cloud-interface eni-26cb1d27
primary-gateway rtb-e64b2882 0.0.0.0/0 local-cloud-interface eni-56cf1957
primary-gateway rtb-63b02a07 0.0.0.0/0 local-cloud-interface eni-1589e714
peer address 10.2.201.149
recovery wait-time 5
bfd source-interface Ethernet1
!
cloud proxy test
https 10.3.255.155 8888show cloud provider azure (CloudEOS and vEOS - Azure)
The show cloud provider azure command displays Azure cloud provider information.
Command Mode
EXEC
Command Syntax
show cloud provider azure
Example
The following example displays the Azure cloud configuration.
switch#show cloud provider azure
Cloud Azure Configuration:
Active credentials : SDK authentication credential file
SDK auth credentials file: flash:
Proxy name :
Active directory credentials : show cloud proxy
The show cloud proxy command displays cloud proxy information.
Command Mode
EXEC
Command Syntax
show cloud proxy [<proxy_name>]
Parameters
- proxy_name Identifies the selected proxy by name.
Example
#show cloud proxy [<proxy_name>]
Proxy Name : MyProxyName
Http proxy : 1.2.3.4:8080
Https proxy : 1.2.3.4:4443
Proxy user : proxyuser1
Proxy password : obfuscatedpassword