Virtual Edge Deployment

cat /proc/cpuinfo | grep vmx

GRUB_CMDLINE_LINUX="intel_iommu=on"

1. Run the following commands: update-grub and update-initramfs -u.
2. Reboot
3. Make sure iommu is enabled.

   velocloud@KVMperf3:~$ dmesg | grep -i IOMMU [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.13.0-107-generic root=/dev/mapper/qa--multiboot--002--vg-root ro intel_iommu=on splash quiet vt.handoff=7 [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.13.0-107-generic root=/dev/mapper/qa--multiboot--002--vg-root ro intel_iommu=on splash quiet vt.handoff=7 [ 0.000000] Intel-IOMMU: enabled …. velocloud@KVMperf3:~$

• For the Intel 82599/82599ES cards in SR-IOV mode:
  1. Download and install ixgbe driver from the Intel website.
  2. Configure ixgbe config (tar and sudo make install).

     velocloud@KVMperf1:~$ cat /etc/modprobe.d/ixgbe.conf

  3. If the ixgbe config file does not exist, you must create the file as follows.

     options ixgbe max_vfs=32,32 options ixgbe allow_unsupported_sfp=1 options ixgbe MDD=0,0 blacklist ixgbevf

  4. Run the update-initramfs -u command and reboot the Server.
  5. Use the modinfo command to verify if the installation is successful.

     velocloud@KVMperf1:~$ modinfo ixgbe and ip link filename: /lib/modules/4.4.0-62-generic/updates/drivers/net/ethernet/intel/ixgbe/ixgbe.ko version: 5.0.4 license: GPL description: Intel(R) 10GbE PCI Express Linux Network Driver author: Intel Corporation, <This email address is being protected from spambots. You need JavaScript enabled to view it.> srcversion: BA7E024DFE57A92C4F1DC93

• For the Intel X710/XL710 cards in SR-IOV mode:
  1. Download and install i40e driver from the Intel website.
  2. Create the Virtual Functions (VFs).

     echo 4 > /sys/class/net/device name/device/sriov_numvfs

  3. To make the VFs persistent after a reboot, add the command from the previous step to the "/etc/rc.d/rc.local" file.
  4. Deactivate the VF driver.

     echo “blacklist i40evf” >> /etc/modprobe.d/blacklist.conf

  5. Run the update-initramfs -u command and reboot the Server.

Using SR-IOV:

<interface type='hostdev' managed='yes'>
<mac address='52:54:00:79:19:3d'/>
<driver name='vfio'/>
<source>
<address type='pci' domain='0x0000' bus='0x83' slot='0x0a' function='0x0'/>
</source>
<model type='virtio'/>
</interface>

<network>
<name>passthrough</name>
<model type='virtio'/>
<forward mode="bridge"/>
<bridge name="passthrough"/>
<virtualport type='openvswitch'/>
<vlan trunk='yes'>
<tag id='33' nativeMode='untagged'/>
<tag id='200'/>
<tag id='201'/>
<tag id='202'/>
</vlan>
</network>

<network>
<name>passthrough</name>
<model type='virtio'/>
<forward mode="bridge"/>
</network>

<domain type='kvm'>
<name>vedge1</name>
<memory unit='KiB'>4194304</memory>
<currentMemory unit='KiB'>4194304</currentMemory>
<vcpu placement='static'>2</vcpu>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64' machine='pc-i440fx-trusty'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
</features>
<!-- Set the CPU mode to host model to leverage all the available features on the host CPU -->
<cpu mode='host-model'>
<model fallback='allow'/>
</cpu>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
<emulator>/usr/bin/kvm-spice</emulator>
<!-- Below is the location of the qcow2 disk image -->
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/edge-VC_KVM_GUEST-x86_64-2.3.0-18-R23-20161114-GA-updatable-ext4.qcow2'/>
<target dev='sda' bus='sata'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<!-- If using cloud-init to boot up virtual edge, attach the 2nd disk as CD-ROM -->
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<source file='/home/vcadmin/cloud-init/vedge1/seed.iso'/>
<target dev='sdb' bus='sata'/>
<readonly/>
<address type='drive' controller='1' bus='0' target='0' unit='0'/>
</disk>
<controller type='usb' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<controller type='sata' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</controller>
<controller type='ide' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<!-- The first two interfaces are for the default L2 interfaces, NOTE VLAN support just for SR-IOV and OpenvSwitch -->
<interface type='network'>
<model type='virtio'/>
<source network='LAN1'/>
<vlan><tag id='#hole2_vlan#'/></vlan>
<alias name='LAN1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x12' function='0x0'/>
</interface>
<interface type='network'>
<model type='virtio'/>
<source network='LAN2'/>
<vlan><tag id='#LAN2_VLAN#'/></vlan>
<alias name='hostdev1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x13' function='0x0'/>
</interface>
<!-- The next two interfaces are for the default L3 interfaces. Note that additional 6 routed interfaces are supported for a combination of 8 interfaces total -->
<interface type='network'>
<model type='virtio'/>
<source network='WAN1'/>
<vlan><tag id='#hole2_vlan#'/></vlan>
<alias name='LAN1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x12' function='0x0'/>
</interface>
<interface type='network'>
<model type='virtio'/>
<source network='LAN2'/>
<vlan><tag id='#LAN2_VLAN#'/></vlan>
<alias name='hostdev1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x13' function='0x0'/>
</interface>
<serial type='pty'>
<target port='0'/>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1'>
<listen type='address' address='127.0.0.1'/>
</graphics>
<sound model='ich6'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</sound>
<video>
<model type='cirrus' vram='9216' heads='1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</memballoon>
</devices>
</domain>

1. Create VM.

   virsh define vedge1.xml

2. Start VM.

   virsh start vedge1

1. Set the spoofchk off.

   ip link set eth1 vf 0 spoofchk off

2. Set the Trusted mode on.

   ip link set dev eth1 vf 0 trust on

3. Set the VLAN, if required.

   ip link set eth1 vf 0 vlan 3500

virsh list
Id Name State
----------------------------------------------------
25 test_vcg running
velocloud@KVMperf2$ virsh console 25
Connected to domain test_vcg
Escape character is ^]

• Brand Name: Intel
• I/O Device Type: Network
• Features: SR-IOV

The following KB article provides details of how to enable SR-IOV on the supported NIC: Enable SR-IOV on the supported NIC.

1. From the Orchestrator UI, retrieve the URL/IP Address. You will need this address for Step c below.
2. Create a new Virtual Edge for the Enterprise. Once the Edge is created, copy the Activation Key. You will need the Activation Key for Step c" below.

   

3. On the customize template page shown in the image below, type in the Activation Code that you retrieved in Step b above, and the Orchestrator URL/IP Address retrieved in Step a above, into the corresponding fields.