Virtual Edge Deployment

To enable the SR-IOV mode on KVM, perform the following steps.

1. Enable SR-IOV in BIOS. This will be dependent on your BIOS. Login to the BIOS console and look for SR-IOV Support/DMA. You can verify support on the prompt by checking that Intel has the correct CPU flag.

   cat /proc/cpuinfo | grep vmx

2. Add the options on Boot (in /etc/default/grub).

   GRUB_CMDLINE_LINUX="intel_iommu=on"

   1. Run the following commands: update-grub and update-initramfs -u.
   2. Reboot
   3. Make sure iommu is enabled.

      velocloud@KVMperf3:~$ dmesg | grep -i IOMMU
       [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.13.0-107-generic root=/dev/mapper/qa--multiboot--002--vg-root ro intel_iommu=on splash quiet vt.handoff=7 
       [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.13.0-107-generic root=/dev/mapper/qa--multiboot--002--vg-root ro intel_iommu=on splash quiet vt.handoff=7 
       [ 0.000000] Intel-IOMMU: enabled
       ….
       velocloud@KVMperf3:~$ 

3. Based on the NIC chip set used, add a driver as follows:
   • For the Intel 82599/82599ES cards in SR-IOV mode:
     1. Download and install ixgbe driver from the Intel website.
     2. Configure ixgbe config (tar and sudo make install).

        velocloud@KVMperf1:~$ cat /etc/modprobe.d/ixgbe.conf

     3. If the ixgbe config file does not exist, you must create the file as follows.

        options ixgbe max_vfs=32,32
        options ixgbe allow_unsupported_sfp=1
        options ixgbe MDD=0,0
        blacklist ixgbevf

     4. Run the update-initramfs -u command and reboot the Server.
     5. Use the modinfo command to verify if the installation is successful.

        velocloud@KVMperf1:~$ modinfo ixgbe and ip link
         filename: /lib/modules/4.4.0-62-generic/updates/drivers/net/ethernet/intel/ixgbe/ixgbe.ko
         version: 5.0.4
         license: GPL
         description: Intel(R) 10GbE PCI Express Linux Network Driver
         author: Intel Corporation, <이 이메일 주소가 스팸봇으로부터 보호됩니다. 확인하려면 자바스크립트 활성화가 필요합니다.>
         srcversion: BA7E024DFE57A92C4F1DC93

   • For the Intel X710/XL710 cards in SR-IOV mode:
     1. Download and install i40e driver from the Intel website.
     2. Create the Virtual Functions (VFs).

        echo 4 > /sys/class/net/device name/device/sriov_numvfs

     3. To make the VFs persistent after a reboot, add the command from the previous step to the "/etc/rc.d/rc.local" file.
     4. Deactivate the VF driver.

        echo “blacklist i40evf” >> /etc/modprobe.d/blacklist.conf

     5. Run the update-initramfs -u command and reboot the Server.

lspci | grep -i Ethernet

01:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function(rev

Discusses how to install and activate the Virtual Edge on KVM using a cloud-init config file.

1. Use gunzip to extract the qcow2 file to the image location (for example, /var/lib/libvirt/images).
2. Create the Network pools that you are going to use for the device, using SR-IOV and OpenVswitch.

   Using SR-IOV:

   The following is a sample network interface template specific to Intel X710/XL710 NIC cards using SR-IOV.

   
   <interface type='hostdev' managed='yes'>
   <mac address='52:54:00:79:19:3d'/>
   <driver name='vfio'/>
   <source>
   <address type='pci' domain='0x0000' bus='0x83' slot='0x0a' function='0x0'/>
   </source>
   <model type='virtio'/>
   </interface>
   

   Using OpenVSwitch:

   <network>
   <name>passthrough</name>
   <model type='virtio'/>
   <forward mode="bridge"/>
   <bridge name="passthrough"/>
   <virtualport type='openvswitch'/>
   <vlan trunk='yes'>
   <tag id='33' nativeMode='untagged'/>
   <tag id='200'/>
   <tag id='201'/>
   <tag id='202'/>
   </vlan>
   </network>
   
   <network>
   <name>passthrough</name>
   <model type='virtio'/>
   <forward mode="bridge"/>
   </network>
   
   <domain type='kvm'>
   <name>vedge1</name>
   <memory unit='KiB'>4194304</memory>
   <currentMemory unit='KiB'>4194304</currentMemory>
   <vcpu placement='static'>2</vcpu>
   <resource>
   <partition>/machine</partition>
   </resource>
   <os>
   <type arch='x86_64' machine='pc-i440fx-trusty'>hvm</type>
   <boot dev='hd'/>
   </os>
   <features>
   <acpi/>
   <apic/>
   <pae/>
   </features>
   <!-- Set the CPU mode to host model to leverage all the available features on the host CPU -->
   <cpu mode='host-model'>
   <model fallback='allow'/>
   </cpu>
   <clock offset='utc'/>
   <on_poweroff>destroy</on_poweroff>
   <on_reboot>restart</on_reboot>
   <on_crash>restart</on_crash>
   <devices>
   <emulator>/usr/bin/kvm-spice</emulator>
   <!-- Below is the location of the qcow2 disk image -->
   <disk type='file' device='disk'>
   <driver name='qemu' type='qcow2'/>
   <source file='/var/lib/libvirt/images/edge-VC_KVM_GUEST-x86_64-2.3.0-18-R23-20161114-GA-updatable-ext4.qcow2'/>
   <target dev='sda' bus='sata'/>
   <address type='drive' controller='0' bus='0' target='0' unit='0'/>
   </disk>
   <!-- If using cloud-init to boot up virtual edge, attach the 2nd disk as CD-ROM -->
   <disk type='file' device='cdrom'>
   <driver name='qemu' type='raw'/>
   <source file='/home/vcadmin/cloud-init/vedge1/seed.iso'/>
   <target dev='sdb' bus='sata'/>
   <readonly/>
   <address type='drive' controller='1' bus='0' target='0' unit='0'/>
   </disk>
   <controller type='usb' index='0'>
   <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
   </controller>
   <controller type='pci' index='0' model='pci-root'/>
   <controller type='sata' index='0'>
   <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
   </controller>
   <controller type='ide' index='0'>
   <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
   </controller>
   <!-- The first two interfaces are for the default L2 interfaces, NOTE VLAN support just for SR-IOV and OpenvSwitch -->
   <interface type='network'>
   <model type='virtio'/>
   <source network='LAN1'/>
   <vlan><tag id='#hole2_vlan#'/></vlan>
   <alias name='LAN1'/>
   <address type='pci' domain='0x0000' bus='0x00' slot='0x12' function='0x0'/>
   </interface>
   <interface type='network'>
   <model type='virtio'/>
   <source network='LAN2'/>
   <vlan><tag id='#LAN2_VLAN#'/></vlan>
   <alias name='hostdev1'/>
   <address type='pci' domain='0x0000' bus='0x00' slot='0x13' function='0x0'/>
   </interface>
   <!-- The next two interfaces are for the default L3 interfaces. Note that additional 6 routed interfaces are supported for a combination of 8 interfaces total -->
   <interface type='network'>
   <model type='virtio'/>
   <source network='WAN1'/>
   <vlan><tag id='#hole2_vlan#'/></vlan>
   <alias name='LAN1'/>
   <address type='pci' domain='0x0000' bus='0x00' slot='0x12' function='0x0'/>
   </interface>
   <interface type='network'>
   <model type='virtio'/>
   <source network='LAN2'/>
   <vlan><tag id='#LAN2_VLAN#'/></vlan>
   <alias name='hostdev1'/>
   <address type='pci' domain='0x0000' bus='0x00' slot='0x13' function='0x0'/>
   </interface>
   <serial type='pty'>
   <target port='0'/>
   </serial>
   <console type='pty'>
   <target type='serial' port='0'/>
   </console>
   <input type='mouse' bus='ps2'/>
   <input type='keyboard' bus='ps2'/>
   <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1'>
   <listen type='address' address='127.0.0.1'/>
   </graphics>
   <sound model='ich6'>
   <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
   </sound>
   <video>
   <model type='cirrus' vram='9216' heads='1'/>
   <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
   </video>
   <memballoon model='virtio'>
   <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
   </memballoon>
   </devices>
   </domain>

3. Save the domain XML file that defines the VM (for example, vedge1.xml created in Step 2).
4. Launch the VM by performing the following steps:
   1. Create VM.

      virsh define vedge1.xml

   2. Start VM.

      virsh start vedge1

   Note: vedge1 is the name of the VM defined in the name element of the domain XML file. Replace vedge1 with the name you specify in the name element.
5. If you are using SR-IOV mode, after launching the VM, set the following on the Virtual Functions (VFs) used:
   1. Set the spoofchk off.

      ip link set eth1 vf 0 spoofchk off

   2. Set the Trusted mode on.

      ip link set dev eth1 vf 0 trust on

   3. Set the VLAN, if required.

      ip link set eth1 vf 0 vlan 3500

   Note: The Virtual Functions configuration step is not applicable for OpenVSwitch (OVS) mode.
6. Console into the VM.

   virsh list
   Id Name State
   ----------------------------------------------------
   25 test_vcg running
   velocloud@KVMperf2$ virsh console 25
   Connected to domain test_vcg
   Escape character is ^]