virtual grind thoughts from the virtual world


Dell Compellent 6.4 Write Latency

Recently, I started seeing abnormal write latency across a number of pods in our environment. After spending a few hours looking at best practices and firmware on the compute and storage side, I decided to look at the switching layer. As expected, things like IO, buffers, buffer credits, etc. all looked fine.

Thinking it may be a cosmetic issue in Dell's Enterprise Manager, I engaged Compellent Support to have a look. After a week or so of analysis, I finally reached a senior engineer, who on a whim, asked me to test disabling SNMP at the controller level. In an instant, the abnormal write latency disappeared. What is alarming is that this was not just cosmetic, but actually felt all the way down to the host level.

I will say that I do not see this behavior in 6.3 but the change made a huge difference in and You can disable SNMP in Storage Center Manager by navigating to Storage Management -> System -> Access -> Configure SNMP Server, then click on "Stop Agent". If SNMP is not running, the link will say "Start Agent".

If you are running any FluidOS version 6.4.1 to 6.4.10, you may want to upgrade or disable SNMP and feel the difference!

Please note the change was made around 11AM, screenshots attached (click to view):

Controller View

Host View

Virtual Machine View


Enabling VAAI Block Zeroing For Compellent Arrays

Recently, Compellent released VAAI (Block Zeroing) support for their arrays. Although we are still waiting on full VAAI support, block zeroing support is a great addition.

To enable this feature, you must install the .vib file from Compellent and the following changes need to be made on your hosts:

esxcli corestorage claimrule add --claimrule-class=Filter --plugin=VAAI_FILTER --type=vendor --vendor=COMPELNT --autoassign

esxcli corestorage claimrule add --claimrule-class=VAAI --plugin=VMW_VAAIP_T10 --type=vendor --vendor=COMPELNT --autoassign

esxcli corestorage claimrule load --claimrule-class=Filter

esxcli corestorage claimrule load --claimrule-class=VAAI

esxcli corestorage claimrule run --claimrule-class=Filter


Setting Default Path Selection Policy (PSP) – Round Robin

When using a storage technology such as Compellent, the ability to take advantage of multipathing is highly desirable. To utilize multipathing you must select the correct Path Selection Policy (PSP) on each ESX/ESXi host.

By default, the "Fixed (VMware)" path selection policy is selected on a new ESX/ESXi install. As you add volumes across multiple hosts in a cluster, this become a pain to change the path selection to "Round Robin (VMware)" on each volume on each host.

Fortunately, changing the default PSP is very easy with the following esxcli command:

esxcli nmp satp setdefaultpsp --satp="VMW_SATP_DEFAULT_AA" --psp="VMW_PSP_RR"

Please note that you want to first verify your "Storage Array Type" before setting the above policy. For Compellent, as of this post, the storage array type is "VMW_SATP_DEFAULT_AA". Other vendors may require a different type, such as "VMW_SATP_EVA" or "VMW_SATP_EQL".


Link To vSphere 4.0 iSCSI SAN Configuration Guide

When working with iSCSI arrays, this guide is worth gold.

Get the guide here.


Enabling ARP Redirect For Compellent Arrays

In order for controller failover to work correctly when using ESX and Compellent arrays, you must enable arp redirect on your ESX hosts. If, for instance, you have a vmhba numbered vhmba33, you can check to see if arp redirect is enabled by connecting to the console and typing:

esxcfg-hwiscsi -l vmhba33

If arp redirect is not enabled, you can simply enable it with the following command:

esxcfg-hwiscsi -a allow vmhba33