In this context, Pave refers to the automation of the first time installation of a DMF hardware appliance. This involves installing a DMF image on a hardware appliance and the completion of the firstboot configuration. Firstboot configuration uses Ansible playbook as a automation tool. In contrast, Repave refers to the automation of the re-installation of DMF images on supported DMF appliances. This involves the automated process of re-installing a DMF image on a DMF hardware appliance and the completion of the firstboot configuration.

There are two main steps involved in accomplishing automatic installation of an image on a supported DMF hardware appliance and the completion of the firstboot configuration. Auto installation of images using PXE and auto configuration of firstboot parameters to complete DMF HW appliance installation.

• Auto-installation of Images - Auto installation of DMF image uses well known services like DHCP, TFTP, NFS and PXE. DMF images are PXE bootable and using the aforementioned services, we can accomplish auto-installation of images on the DMF appliances.

  A high level procedure for auto-installation is given below.

  • User configures DHCP server to provide DHCP IP address:

    • Next-server IP address (This is the TFTP server IP address specified by Option 66 configuration on DHCP server).

      next-server <TFTP_SERVER_IP>;
      class "pxeclient" {
      match if substring (option vendor-class-identifier, 0, 9) = "PXEClient";
      filename "boot/grub/x86_64-efi/core.efi"; # x86 EFI
      }

    • It is recommended to bind a static IP to the hardware MAC address of the DMF appliance. Sample configuration using linux based DHCP service is shown below.

      group {
      host <DMF_APPLIANCE_HOSTNAME> {
      hardware ethernet <DMF_APPLIANCE_MAC>;
      fixed-address <DESIRED_IP_ADDRESS>;
      }

  • On a DMF HW appliance where you want PXE boot, enable PXE on the management interface NIC card.

    

  • For the initial PAVE action, press the F12 key during the initial boot process to manually trigger PXE boot. This is not required when the user intends to reinstall (REPAVE) the DMF appliance images.

    

    • Power cycle the DMF HW appliance.
    • DHCP client on NIC card sends DHCP discover and request and gets IP address and next-server (TFTP server) IP address.
    • Then management NIC gets the bootloader, via PXE boot using a TFTP server. A bootloader config file with a filename based on the MAC address of the appliance is configured and saved on the DHCP server.
    • The appliance also gets other configuration parameters like NFS mount where appliance ISO images are stored.
    • It boots from the boot loader and then gets the DMF appliance ISO image from the NFS server.
    • The appliance then boots the ISO image and starts the installation process without needing user input.
    • On reboot, the appliance again acquires a DHCP based management IP address, sets up initial default user credentials and waits for auto firstboot configuration via Ansible.
      Note: For repave, the procedure is the same except that the DMF appliance is already running a DMF image and needs to boot from PXE again for re-installation.
  • Auto Configuration of Firstboot - Auto configuring of firstboot parameters uses Ansible. This is accomplished by interacting with auto-firstboot-cloud-plugin available in hardware appliances.
    • Customer initiates initial configuration playbook in Ansible. A sample YAML based anisble playbook file is shown below:

      - name: Test Autofirstboot Properties Provider
      gather_facts: false
      hosts: Controllers
      connection: "local"
      run_once: true
      tasks:
      - name: Provide Autofirstboot Properties to Cluster
      arista.dmf.provisioner:
      config_json: "<Pave-Repave-config.json>"
      timeout: 3600
      log_dir: "logs"

    • The configuration playbook will get the JSON based configuration file from the server where the playbook is being executed. The JSON file should have specific sections for each DMF appliance that is being installed/re-installed. An example is shown below.

      {
      "<ACTIVE-CONTROLLER-IP>": {
      "initial-admin-password": "<ACTIVE-CONTROLLER-CLEAR-TEXT-PASSWD>",
      "initial-config-password": "pxe_temp_password",
      "dhcp-ip": "10.240.156.82",
      "appliance-type": "BMF",
      "firstboot-properties": {
      "admin-password": "<ACTIVE-CONTROLLER-CLEAR-TEXT-PASSWD>",
      "recovery-password": "<ACTIVE-CONTROLLER-CLEAR-TEXT-RECOVERY-PASSWD>",
      "hostname": "<ACTIVE-CONTROLLER-HOSTNAME>",
      "cluster-name": "<CLUSTER-NAME>",
      "cluster-description": "<CLUSTER-DESCRIPTION>",
      "ip-stack": "ipv4",
      "ntp-servers": ["<CLUSTER-NTP-SERVER-1>","<CLUSTER-NTP-SERVER-2>"],
      "dns-servers": ["<CLUSTER-DNS-SERVER-1>","<CLUSTER-DNS-SERVER-2>"]
      }
      },
      "<STANDBY-CONTROLLER-IP>": {
      "initial-config-password": "pxe_temp_password",
      "dhcp-ip": "<STANDBY-CONTROLLER-DHCP-IP>",
      "appliance-type": "BMF",
      "firstboot-properties": {
      "admin-password": "<STANDBY-CONTROLLER-CLEAR-TEXT-PASSWD>",
      "recovery-password": "<STANDBY-CONTROLLER-CLEAR-TEXT-RECOVERY-PASSWD>",
      "hostname": "<STANDBY-CONTROLLER-HOSTNAME>",
      "cluster-name": "<CLUSTER-NAME>",
      "cluster-to-join": "<ACTIVE-CONTROLLER-IP>",
      "ip-stack": "ipv4"
      }
      },
      "<MANAGED-APPLIANCE-IP>": {
      "initial-config-password": "pxe_temp_password",
      "dhcp-ip": "<STANDBY-CONTROLLER-DHCP-IP>",
      "appliance-type": "BMFSN",
      "firstboot-properties": {
      "admin-password": "<MANAGED-APPLIANCE-CLEAR-TEXT-PASSWD>",
      "recovery-password": "<MANAGED-APPLIANCE-CLEAR-TEXT-RECOVERY-PASSWD>",
      "hostname": "<MANAGED-APPLIANCE-HOSTNAME>",
      "ip-stack": "ipv4",
      "controller_ip": "<ACTIVE-CONTROLLER-IP>",
      "ntp-servers": ["<NTP-SERVER-1>","<NTP-SERVER-2>"],
      "dns-servers": ["<DNS-SERVER-1>","<DNS-SERVER-2>"]
      }

      Note: The initial-config-password is used by the ansible script to access the DMF appliance via SSH for the first time.
    • Ansible pushes initial configuration data to appliance via SSH, using default credential configured in the JSON file in Step 1 above.
    • Appliance completes initial configuration (firstboot).
    • Appliance confirms to Ansible that initial configuration is complete. An example run of the ansible-playbook is shown below.
    An example run of the ansible-playbook is shown below.

    ansible-playbook sample_playbook.yml --limit="10.240.156.82,10.240.156.84" -v
    [DEPRECATION WARNING]: Ansible will require Python 3.8 or newer on the controller starting
    with Ansible 2.12. Current version: 2.7.18 (default, Jul 1 2022, 12:27:04) [GCC 9.4.0].
    This feature will be removed from ansible-core in version 2.12. Deprecation warnings can be
    disabled by setting deprecation_warnings=False in ansible.cfg.
    /root/.cache/pypoetry/virtualenvs/arista-dmf-vQFV2Q4_-py2.7/lib/python2.7/site-packages/
    ansible/parsing/vault/__init__.py:44: CryptographyDeprecationWarning: Python 2 is no longer
    supported by the Python core team. Support for it is now deprecated in cryptography, and
    will be removed in the next release.
    from cryptography.exceptions import InvalidSignature
    Using /etc/ansible/ansible.cfg as config file
    PLAY [Test Autofirstboot Properties Provider]
    *************TASK [Provide Autofirstboot Properties to Cluster]
    *************Targeting all hosts specified in config [u'10.240.156.84', u'10.240.156.82']
    [10.240.156.82 BMF (active)]: Waiting for SSH connection.
    [10.240.156.84 BMF (standby)]: Waiting for SSH connection.
    [10.240.156.84 BMF (standby)]: Established SSH connection.
    [WARNING]: It is NOT recommended to disable SSL verification. Please upload a valid SSL
    certificate to the https://10.240.156.84:8443 API server.
    [10.240.156.82 BMF (active)]: Established SSH connection.
    [WARNING]: It is NOT recommended to disable SSL verification. Please upload a valid SSL
    certificate to the https://10.240.156.82:8443 API server.
    [WARNING]: [10.240.156.82 BMF (active)]: Detected BIOS firmware. The BIOS PXE boot trigger has
    been found to be unreliable. Upgrade to UEFI is recommended.
    [WARNING]: [10.240.156.84 BMF (standby)]: Appliance already configured. Attempting PXE boot to
    repave.
    [WARNING]: [10.240.156.82 BMF (active)]: Appliance already configured. Attempting PXE boot to
    repave.
    [10.240.156.84 BMF (standby)]: Expecting appliance to reboot with IP = 10.240.156.84.
    [10.240.156.82 BMF (active)]: Expecting appliance to reboot with IP = 10.240.156.82.
    [10.240.156.84 BMF (standby)]: Verified repave succeeded.
    [10.240.156.84 BMF (standby)]: Waiting for active to be configured.
    [10.240.156.82 BMF (active)]: Verified repave succeeded.
    [10.240.156.82 BMF (active)]: Successfully provided config to appliance.
    [10.240.156.82 BMF (active)]: Waiting for appliance to apply config. Expecting final IP = 10.
    240.156.82.
    [10.240.156.82 BMF (active)]: Successfully retrieved firstboot logs.
    [10.240.156.82 BMF (active)]: Config applied.
    [10.240.156.82 BMF (active)]: Attempting to reset recovery password.
    [10.240.156.82 BMF (active)]: Successfully reset recovery password.
    [10.240.156.84 BMF (standby)]: Active configured. Proceeding with configuration.
    [10.240.156.84 BMF (standby)]: Successfully provided config to appliance.
    [10.240.156.84 BMF (standby)]: Waiting for appliance to apply config. Expecting final IP = 10.
    240.156.84.
    [10.240.156.84 BMF (standby)]: Successfully retrieved firstboot logs.
    [10.240.156.84 BMF (standby)]: Config applied.
    [10.240.156.84 BMF (standby)]: Attempting to reset recovery password.
    [10.240.156.84 BMF (standby)]: Successfully reset recovery password.
    changed: [10.240.156.82] => {"changed": true, "success": true, "summary": {"10.240.156.82": {
    "appliance-type": "BMF", "expected-dhcp-ip": "10.240.156.82", "role": "active"}, "10.240.156.
    84": {"appliance-type": "BMF", "expected-dhcp-ip": "10.240.156.84", "role": "standby"}}}
    PLAY RECAP
    ******************56.82 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

PXE boot is fundamentally incompatible with zero-trust environments and as a result assumptions must be made in order to establish the trust required to authenticate the appliances. This section provides a summary of the assumptions which underpin the security this design provides.

• The management network is trusted, such that:
  • DHCP is secured.
    • The DHCP server (or DHCP relaying) is secure.
    • Rogue DHCP packets are dropped/blocked.
  • Impersonation by MAC or IP spoofing is not possible due to either:
    • Assumption: Machines on the same L2 network are trusted not to impersonate other machines by presenting false identities (MAC addresses or IP addresses).
    • Guarantee: Machines in the same L2 network cannot impersonate other machines by presenting themselves with false identities as enforced by network admins who may, for example:
      • Pin a MAC address to a specific switch and switch port.
      • Pin an IP address to a MAC address statically (i.e. static ARP entry).
  • Routers between the PXE TFTP/HTTP server and the target machine are secure/trusted to forward packets to the rightful owners (e.g. having correct routing tables and MAC address tables).
  • The PXE (TFTP/NFS) server is secure and cannot be compromised resulting in an attacker providing a malicious image.