DPDK Mode
This chapter will focus on new DPDK based veos, while highlighting the changes from the existing kernel based veos, as needed.
the Arista veos router is a cloud-grade, feature-rich, multi-cloud and multi-hypervisor virtual router that empowers enterprises and cloud providers to build consistent, highly secure and scalable multi-cloud networks.
![](/assets/images/user-manual/cg-veos-router/Graphics/DPDK%20mode/DPDK_Topology.png)
the veos router can run in two modes : DPDK (high performance) and kernel, each with its own set of supported features. From veos-router-4.23.0FX new installations of veos router from the hypervisor image or on public cloud by default run in high performance DPDK mode. Going forward all the new features and development will be in the DPDK mode.
Both flavors of veos perform all the packet forwarding operations in software, but use different software components for the same. DPDK based veos which use DPDK to perform packet forwarding operations is much more efficient. Note, that while DPDK mode has more number of features than the kernel mode, there are certain features that are available only in the kernel mode like Zone Based Segmentation (ZSS) and sflow.
Platform Compatibility
the following platforms supports veos-DPDK mode.
- VMWare esxi 6.0+
- Supported NICs
- VMWare vmxnet3 ( para-virtualized )
- Intel x520/82599 PCIe passthrough and SRIOV mode
- Supported NICs
- Linux / kvm
- RHEL/CentOS 7.0+
- Ubuntu 18.04+
- Supported NICs
- Virtio-net ( para-virtualized )
- Intel x520/82599 PCIe passthrough and SRIOV mode
- Supported NICs
Hardware Resource Requirements
the veos router DPDK mode depends on the features available in modern CPUs, and thus it is important that veos VMs are deployed on only certain types of server platforms in order to meet performance benchmarks. Ideal server configuration should be similar to the following.
- Intel E5-26XX v4 or later class of CPU
- > 2.2 GHz core frequency for best performance
- Minimum 2 CPU cores reserved for veos
- At-least 2 additional cores will be used by hypervisor software
-
- Hyper-threading disabled for best performance
- non-NUMA or ensure veos resources are from single NUMA node
- Power saving and frequency scaling features disabled in BIOS/Firmware/Hypervisor
- High speed hard-drive (~7200 RPM) or SSD and minimum 8 GB of free disk space.
- At-least 4 GB for RAM reserved for veos.
- 4 GB RAM can only support 3 VRFs
- 8 GB required for supporting up-to 8 VRFs.
- Server should have more memory for use by hypervisor software.
In addition to the server requirements, user need to configure the hypervisor to provide memory and CPU reservation for veos VMs to ensure optimum performance is achieved. In addition to this, map each vCPU used by veos VM to a unique physical CPU core. these configuration and settings are based on the type of hypervisor used, and information is usually documented in configuration guide(s) provided by the hypervisor vendor.
Switching to DPDK Mode
All the new installations of CloudEOS and veos router from the hypervisor image or public cloud by default run in high performance DPDK mode from CloudEOS and veos-router-4.23.0FX.
If you have an existing instance that runs in the kernel mode that you want to switch to DPDK mode, please consider the following:
- Make sure you are not enabled features that DPDK mode does not support.
- Memory requirements of DPDK mode are listed below under “Hardware resource requirements”. DPDK requires more memory than the kernel mode.
- When upgrading an existing instance make sure the instance has sufficient memory to run in DPDK mode.
- DPDK mode runs the datapath cores in poll mode and therefore it runs the CPU at 100%.
switch#conf t
switch(config)#bash sudo su -
Arista Networks EOS shell
-bash-4.3# cat /mnt/flash/veos-config
# Use 'MODE' to set the forwarding plane for veos. If 'MODE' is set multiple times
# the last configuration takes effect.
# 'MODE=linux' runs veos with linux forwarding plane
MODE=linux
# 'MODE=sfe' runs veos with DPDK forwarding plane
#MODE=sfe
-bash-4.3# cat /mnt/flash/veos-config
# Use 'MODE' to set the forwarding plane for veos. If 'MODE' is set multiple times
# the last configuration takes effect.
# 'MODE=linux' runs veos with linux forwarding plane
#MODE=linux
# 'MODE=sfe' runs veos with DPDK forwarding plane
MODE=sfe
-bash-4.3# exit
logout
switch(config)# reload
After reload,veos router will boot up in DPDK mode.
Upgrading CloudEOS and veos-Kernel to CloudEOS and veos-DPDK
Upgrade an existing CloudEOS and veos installation from kernel mode to DPDK mode, by copying EOS.swi to /mnt/flash , and then follow procedure defined above in Installing CloudEOS and veos-DPDK, to switch to DPDK mode. Please note, this process requires a system reload.
CloudEOS and veos-DPDK Mode Verification
switch#show agent sfe ping
show agent sfe ping
Agent Name Last Ping Max PingMax Ping Response SeenLast Ping Response Seen
---------------------- ------------- ----------------------------------- ------------------------
Sfe1.571 ms 2209.819 ms 2019-11-15 11:14:05 2019-12-12 15:02:48
A system in DPDK mode uses 100% of CPU cycles for each datapath vCPU. This is normal and expected. To ensure that packet forwarding tasks, which are CPU intensive, do not starve control plane and management operations, EOS dedicates CPU cores for control/management functions.
Linux “top” command followed by typing “1” when “top” is running is used to get detailed CPU utilization. the below output shows “top” results for a CloudEOS and veos with 2 cores. Depending on the version either “Sfe” or “bessd” will show using the 100% of the datapath core.
veos-CLI(config)#bash top -n 1
Tasks: 236 total, 1 running, 235 sleeping, 0 stopped, 0 zombie
%Cpu0:1.6 us,0.7 sy,0.0 ni, 95.1 id,0.0 wa,2.6 hi,0.0 si,0.0 st
%Cpu1:100.0 us,0.0 sy,0.0 ni,0.0 id,0.0 wa,0.0 hi,0.0 si,0.0 st
KiB Mem: 8122156 total,4642632 used,3479524 free, 255624 buffers
KiB Swap:0 total,0 used,0 free,1857744 cached
PID USERPRNIVIRTRESSHR S%CPU %MEMTIME+COMMand
3355 root20 0 2186m 239m 201m S 100.13.039262:38 Sfe [or bessd]
2544 root20 0375m58m38m S 0.30.7 192:36.08 ProcMgr-worker
2705 root20 0403m 180m 142m S 0.32.3 135:53.49 Sysdb
3102 root20 0379m 111m95m S 0.31.4 8:06.90 Ira
3119 root20 0373m86m70m S 0.31.124:34.26 StpTxRx
veos vCPU Core Allocation
the veos-DPDK VM is supported only on a few standard CPU configurations. Such as:
- 2-core version
- 4-core version
- 8-core version
Depending on the CPU core count, a certain number of CPU cores are reserved for control and management operations, and the remaining are reserved for packet forwarding and data-plane operations.
- On 2-core and 4-core VMs, a single core is reserved for management and control plane and remaining core(s) is/are reserved for packet-forwarding or data-plane operations.
- On 8-core VMs, two cores are reserved for control/management functions and the remaining are reserved for data-plane functions.
Monitoring Datapath CPU utilization in DPDK Mode
switch#platform sfe bessctl
shType "help" for more information.
localhost:10514 $ show busy
Worker IDBusy PPS
0 0 0
1 0 0
2 0 0
localhost:10514 $ show busy
Worker IDBusy PPS
0 0 0
1 0 0
2 0 0
In addition to this, a syslog message is logged if the CPU utilization is over 80% for 60 seconds, all the while doing useful packet processing.
2019-10-28 23:44:11.7769098908 Log
0 %SFE-4-CORE_SUSTAINED_BUSY: Software Data Plane Forwarding service is experiencing heavy load as it has been80 percent busy over last 60 seconds
2019-10-29 15:54:53.3942848908 Log
0 %SFE-4-CORE_SUSTAINED_BUSY: Software Data Plane Forwarding service is experiencing heavy load as it has been80 percent busy over last 60 seconds
General Troubleshooting
To find information about packet forwarding agent, please check the following log file.
veos-CLI(config)# bash cat /var/log/agents/bessd.INFO
veos-CLI(config)# bash cat /var/log/agents/bessd.FATAL
In addition to the aforementioned log file(s), syslog and EOS show-tech are also a valuable source of troubleshooting information.