virtual grind thoughts from the virtual world


Dell Compellent 6.4 Write Latency

Recently, I started seeing abnormal write latency across a number of pods in our environment. After spending a few hours looking at best practices and firmware on the compute and storage side, I decided to look at the switching layer. As expected, things like IO, buffers, buffer credits, etc. all looked fine.

Thinking it may be a cosmetic issue in Dell's Enterprise Manager, I engaged Compellent Support to have a look. After a week or so of analysis, I finally reached a senior engineer, who on a whim, asked me to test disabling SNMP at the controller level. In an instant, the abnormal write latency disappeared. What is alarming is that this was not just cosmetic, but actually felt all the way down to the host level.

I will say that I do not see this behavior in 6.3 but the change made a huge difference in and You can disable SNMP in Storage Center Manager by navigating to Storage Management -> System -> Access -> Configure SNMP Server, then click on "Stop Agent". If SNMP is not running, the link will say "Start Agent".

If you are running any FluidOS version 6.4.1 to 6.4.10, you may want to upgrade or disable SNMP and feel the difference!

Please note the change was made around 11AM, screenshots attached (click to view):

Controller View

Host View

Virtual Machine View


vCloud Director Does Not Show Storage Policies

I have been experiencing this a lot lately with vCloud Director versions 5.x. What happens is that you will add tags in the web client and then apply to them to the appropriate datastores. After performing a "Refresh Storage Policies" in vCloud Director, your newly created polices do not show up.

I have found that this is a bug and does not clear unless you empty some tables in your vCloud Director database. The script below simply empties some inventory tables and when the services are restarted, re-syncs them from vCenter.

I will say that I always gracefully stop all cells, backup my vCloud database, then run this script. Please note that this script is not intrusive to any static data.


delete from task;
update jobs set status = 3 where status = 1;
update last_jobs set status = 3 where status = 1;
delete from busy_object;
delete from QRTZ_CALENDARS;
delete from QRTZ_TRIGGERS;
delete from QRTZ_JOB_DETAILS;
delete from compute_resource_inv;
delete from custom_field_manager_inv;
delete from cluster_compute_resource_inv;
delete from datacenter_inv;
delete from datacenter_network_inv;
delete from datastore_inv;
delete from dv_portgroup_inv;
delete from dv_switch_inv;
delete from folder_inv;
delete from managed_server_inv;
delete from managed_server_datastore_inv;
delete from managed_server_network_inv;
delete from network_inv;
delete from resource_pool_inv;
delete from storage_pod_inv;
delete from task_inv;
delete from vm_inv;
delete from property_map;


Cisco Champion

I am honored to be selected as a Cisco Champion. To find out more about the program, check it out here:

Cisco Champions

justin giardina, cisco champion

justin giardina, cisco champion


Using PowerCLI To Answer All CD-ROM Override Questions

I recently ran in to a scenario where multiple cd-rom drives were disconnected from 100's of virtual machines and there were questions generated for each. In order to quickly answer and clear the questions, you can simply use this one liner after connecting to vCenter (Connect-VIServer):

Get-VM | Get-VMQuestion | Set-VMQuestion -Option "Yes" -Confirm:$false

Note than the "-Confirm:$false" will not prompt you as the script runs, taking care of each question automatically. If you would rather see each confirmation, you can remove the the Confirm switch.


Openstack Hands-On Labs Now Available

It is great to see the contributions from VMware around the Openstack platform, especially with the new Havana release.

VMware also released new hands-on labs around Openstack and vSphere (HOL-SDC-1320):

VMware Blog Posting


Using PowerCLI to Answer VM Questions

I recently have been testing some vendor's storage solutions and fast provisioning in vCloud Director. During the testing, I create a load simulator to mimic 1000 virtual machines inflating disks, testing write patterns, etc. In any case, during the testing, I was able to completely obliterate the vendor's write cache and write IOPS, causing datastore issues. This also caused 1000 virtual machines to get stalled due to datastores being filled up, and having a question placed on them.

In order to quickly answer these 1000 questions, this PowerCLI example worked like a charm:

Get-VM LoadTest* | Get-VMQuestion | Set-VMQuestion --Option "Cancel"

In this example, I wanted to answer "Cancel" on all VM's connected named LoadTest*


VMware vExpert 2012

I am honored to be chosen as a VMware vExpert for 2012. To find out more about the vExpert program, check it out here:

Justin Giardina vExpert

Justin Giardina vExpert


Forcing VMware Tools To Cancel

I have run in to a few occasions recently where a customer will initiate a VMware Tools install via vCD and not complete the installer. If this virtual machine needs to be moved in vCenter due to something like HA, DRS, etc., vCenter will report that the machine cannot migrate because the .iso that is mounted to the virtual machine cannot unmount.

Fortunately, there is an easy way to cancel the install using vim-cmd:

First, you can get the list of all VM's on your host with the following command:

vim-cmd vmsvc/getallvms

You will see that each VM is identified with a number, which is the VM ID. To cancel the tools install, simply use vim-cmd again with the vm number you found above:

vim-cmd vmsvc/tools.cancelinstall (number from above)


vim-cmd vmsvc/tools.cancelinstall 777

Tagged as: , , 1 Comment

Manually Remove vCloud Director Agent From Hosts

I recently ran in to an issue in a lab where I had to manually remove the vCD agent from an ESXi host. The command to manually do this on an ESXi5 host is:

esxcli software vib remove -n vcloud-agent

Note that this is vCloud Director 1.5.


Using vim-cmd to Power On and Power Off Virtual Machines

To get a list of all virtual machines on a host:

vim-cmd vmsvc/getallvms

To power on a specific virtual machine (from above):

vim-cmd vmsvc/power.on vmnumber

To power off a specific virtual machine (from above):

vim-cmd vmsvc/ vmnumber