virtual grind thoughts from the virtual world


vCloud Director Catalog Sync Issues

Over the years, we have seen numerous issues with the catalog sync process in vCloud Director, even back in the 5.x days. The common scenario is that a catalog is made public from a source vCD instance and then a remote vCD instance will subscribe to the catalog.

During the process, you will see the export task happen from the source instance, the files placed on the source NFS mount, the synchronization task start on the remote instance, and the files successfully written to the NFS mount at the remote instance. Next, the import task will start in vCD, but there is never a corresponding import OVF task in vCenter.

What will immediately fix this is an advanced configuration option in vCD:

cell-management-tool manage-config -n vcloud.activities.forceSingleCellOperation -v true


vCloud Director Does Not Show Storage Policies

I have been experiencing this a lot lately with vCloud Director versions 5.x. What happens is that you will add tags in the web client and then apply to them to the appropriate datastores. After performing a "Refresh Storage Policies" in vCloud Director, your newly created polices do not show up.

I have found that this is a bug and does not clear unless you empty some tables in your vCloud Director database. The script below simply empties some inventory tables and when the services are restarted, re-syncs them from vCenter.

I will say that I always gracefully stop all cells, backup my vCloud database, then run this script. Please note that this script is not intrusive to any static data.


delete from task;
update jobs set status = 3 where status = 1;
update last_jobs set status = 3 where status = 1;
delete from busy_object;
delete from QRTZ_CALENDARS;
delete from QRTZ_TRIGGERS;
delete from QRTZ_JOB_DETAILS;
delete from compute_resource_inv;
delete from custom_field_manager_inv;
delete from cluster_compute_resource_inv;
delete from datacenter_inv;
delete from datacenter_network_inv;
delete from datastore_inv;
delete from dv_portgroup_inv;
delete from dv_switch_inv;
delete from folder_inv;
delete from managed_server_inv;
delete from managed_server_datastore_inv;
delete from managed_server_network_inv;
delete from network_inv;
delete from resource_pool_inv;
delete from storage_pod_inv;
delete from task_inv;
delete from vm_inv;
delete from property_map;


Openstack Hands-On Labs Now Available

It is great to see the contributions from VMware around the Openstack platform, especially with the new Havana release.

VMware also released new hands-on labs around Openstack and vSphere (HOL-SDC-1320):

VMware Blog Posting


Using PowerCLI to Answer VM Questions

I recently have been testing some vendor's storage solutions and fast provisioning in vCloud Director. During the testing, I create a load simulator to mimic 1000 virtual machines inflating disks, testing write patterns, etc. In any case, during the testing, I was able to completely obliterate the vendor's write cache and write IOPS, causing datastore issues. This also caused 1000 virtual machines to get stalled due to datastores being filled up, and having a question placed on them.

In order to quickly answer these 1000 questions, this PowerCLI example worked like a charm:

Get-VM LoadTest* | Get-VMQuestion | Set-VMQuestion --Option "Cancel"

In this example, I wanted to answer "Cancel" on all VM's connected named LoadTest*


Forcing VMware Tools To Cancel

I have run in to a few occasions recently where a customer will initiate a VMware Tools install via vCD and not complete the installer. If this virtual machine needs to be moved in vCenter due to something like HA, DRS, etc., vCenter will report that the machine cannot migrate because the .iso that is mounted to the virtual machine cannot unmount.

Fortunately, there is an easy way to cancel the install using vim-cmd:

First, you can get the list of all VM's on your host with the following command:

vim-cmd vmsvc/getallvms

You will see that each VM is identified with a number, which is the VM ID. To cancel the tools install, simply use vim-cmd again with the vm number you found above:

vim-cmd vmsvc/tools.cancelinstall (number from above)


vim-cmd vmsvc/tools.cancelinstall 777

Tagged as: , , 1 Comment

Manually Remove vCloud Director Agent From Hosts

I recently ran in to an issue in a lab where I had to manually remove the vCD agent from an ESXi host. The command to manually do this on an ESXi5 host is:

esxcli software vib remove -n vcloud-agent

Note that this is vCloud Director 1.5.


Unable to Create or Restart Networks After vCloud Director Upgrade to 1.5

I recently ran in to an issue with a vCloud Director upgrade. After the vCD cells and database were upgraded, I had to upgrade the existing vShield Manager to version 5. The update is simple, you basically provide an upgrade package (in .tgz format) via the VShield Manager UI. From here, the software uploads, installs, reboots, etc. Please note that this process takes a few minutes, the UI is not the best at letting you know exactly what is going on. I simply opened the console to the vShield Manager to watch the progress.

After rebooting, everything seemed okay, but I noticed that when I tried to create new networks or reset existing networks, the process failed. I kept getting the following message in vCD:

Cannot create vShield Edge Device for network: [Unique ID Number].
- edge error: Creating/configuring the VR failed: vShield Edge Device on network: [Unique ID Number] is not ready for initialization after 180 seconds.

After digging around a bit, it seems that even though the vSM upgrade went well, the version change was not recorded in the vCD database. This was confirmed as a bug with VMware Engineering and the workaround is very simple. Simply log in to vCD as an administrator, go to the Manage & Monitor tab, highlight the vCenter server in question, right click, and choose Properties. From there, choose the vShield Manager tab and re-enter only the username that is specified. Once you clear out and re-enter only the username, click OK.

You can see my response on this issue here:

Unable to create or reset networks in vCD 1.5 after upgrade from 1.0


Restarting vShield Manager Web Interface

Sometimes vCD cells lose connectivity to vShield Manager. Instead of rebooting the vShield Manager virtual machine, the web service of vShield Manager can simply be restarted.

To accomplish this, you can open the console of your vShield Manager virtual machine, log in, and enter enable mode. From there, enter configure mode and issue the command "no web-manager" and then "web-manager".

manager# configure terminal
manager(config)# no web-manager
manager(config)# web-manager

This will restart the web and hopefully clear any web service connectivity issues.


vCloud Director Cell Firewall Settings – Cisco ASA

In a vCloud Director environment, vCD cells are usually placed in a DMZ network. Based on best practices, a load balancer is also used in multi-cell environments, which is placed in front of the vCD cells.

Access to/from the vCD cells should be restricted not only from the public side, but also internally. For instance, vCD cells do need to communicate with a database vlan where a database server lies and a management vlan where services such as vCenter live.

When configuring multiple vlans, certain access lists are placed between the vlans for communication. An example of this would be an access list that allows your vCD cells to communicate with the database vlan. For example, you may have an access list that allows tcp port 1521 (Oracle) from your vCD cells to your database server.

Another issue that may come up are keepalives for tcp streams between your vCD cells on one vlan and your esxi hosts on another vlan. vCloud Director will also email messages such as:

"The Cloud Director Server cannot communicate with the Cloud Director agent on host "hostname". When the agent starts responding to the Cloud Director Server, Cloud Director Server will send an email alert.

If you are using a Cisco ASA environment, this issue can be fixed easily with a feature called Dead Connection Detection.

The following config will allow you to do this:

1. Create an access-list that allows the ip addresses or subnet of your vCD cells:

access-list vcd_dcd extended permit ip host any
access-list vcd_dcd extended permit ip host any


access-list vcd_dcd extended permit ip any

These access lists would allow your vCD cells on and .11 or Note that you can also make this access-list more specific by defining the destination, which would be your esxi hosts or subnet. An example of this would be:

access-list vcd_dcd extended permit ip host host
access-list vcd_dcd extended permit ip host host


access-list vcd_dcd extended permit ip

2. Next, you need to make a class-map:

class-map vcd_keepalive_class
match access-list vcd_dcd

3. Create a policy-map that defines your timeout and dcd settings:

policy-map vcd_keepalive_policy
class vcd_keepalive_class
set connection timeout idle 2:00:00 dcd 0:10:00 3

4. Finally, create a service policy for the interface where your vCD cells reside:

service-policy vcd_keepalive_policy interface INTNAME

* Note that you will change "INTNAME" with the ASA interface (nameif) name.

For reference, this Cisco article covers DCD in detail:

Configuring Connection Limits and Timeouts


Video – Using VMware vCloud Connector

A coworker, Jack Bailey, recently made a great video demonstrating VMware's vCloud Connector with iland vCloud Services. The video covers connecting to a vCloud Service Provider from a local (private) vSphere environment and shows the functionality of the product.

vCloud Connector Video