virtual grind thoughts from the virtual world


vCloud Director Catalog Sync Issues

Over the years, we have seen numerous issues with the catalog sync process in vCloud Director, even back in the 5.x days. The common scenario is that a catalog is made public from a source vCD instance and then a remote vCD instance will subscribe to the catalog.

During the process, you will see the export task happen from the source instance, the files placed on the source NFS mount, the synchronization task start on the remote instance, and the files successfully written to the NFS mount at the remote instance. Next, the import task will start in vCD, but there is never a corresponding import OVF task in vCenter.

What will immediately fix this is an advanced configuration option in vCD:

cell-management-tool manage-config -n vcloud.activities.forceSingleCellOperation -v true


Cisco Vulnerability – CVE-2016-1287

Recently, Cisco published an advisory highlighting a critical buffer overflow vulnerability that affects the ASA, ASAv, Firepower ASA, and ISA products. This vulnerability is remotely exploitable and the SANS ISC is reporting large spikes in UDP 500 scanning.

The vulnerability affects many different ASA versions and some older 8.x versions are not able to be patched, without doing a major version upgrade. This could be challenging for some organizations since there are numerous operational changes within the ASA firmware and how the ASA deals with NAT, for instance. Fortunately, Stack8 has released a workaround for older versions if an organization cannot upgrade. This also may be a good idea for organizations that need time to plan their upgrade or wait for a change window.

I would urge all organizations to take this vulnerability very seriously and patch as soon as possible. I already see exploits in the wild that can do things like reboot the ASA, download/upload config, and also install an ASA rootkit.

For the technical readers, a full explanation of the vulnerability is here.


Dell Compellent 6.4 Write Latency

Recently, I started seeing abnormal write latency across a number of pods in our environment. After spending a few hours looking at best practices and firmware on the compute and storage side, I decided to look at the switching layer. As expected, things like IO, buffers, buffer credits, etc. all looked fine.

Thinking it may be a cosmetic issue in Dell's Enterprise Manager, I engaged Compellent Support to have a look. After a week or so of analysis, I finally reached a senior engineer, who on a whim, asked me to test disabling SNMP at the controller level. In an instant, the abnormal write latency disappeared. What is alarming is that this was not just cosmetic, but actually felt all the way down to the host level.

I will say that I do not see this behavior in 6.3 but the change made a huge difference in and You can disable SNMP in Storage Center Manager by navigating to Storage Management -> System -> Access -> Configure SNMP Server, then click on "Stop Agent". If SNMP is not running, the link will say "Start Agent".

If you are running any FluidOS version 6.4.1 to 6.4.10, you may want to upgrade or disable SNMP and feel the difference!

Please note the change was made around 11AM, screenshots attached (click to view):

Controller View

Host View

Virtual Machine View


vCloud Director Does Not Show Storage Policies

I have been experiencing this a lot lately with vCloud Director versions 5.x. What happens is that you will add tags in the web client and then apply to them to the appropriate datastores. After performing a "Refresh Storage Policies" in vCloud Director, your newly created polices do not show up.

I have found that this is a bug and does not clear unless you empty some tables in your vCloud Director database. The script below simply empties some inventory tables and when the services are restarted, re-syncs them from vCenter.

I will say that I always gracefully stop all cells, backup my vCloud database, then run this script. Please note that this script is not intrusive to any static data.


delete from task;
update jobs set status = 3 where status = 1;
update last_jobs set status = 3 where status = 1;
delete from busy_object;
delete from QRTZ_CALENDARS;
delete from QRTZ_TRIGGERS;
delete from QRTZ_JOB_DETAILS;
delete from compute_resource_inv;
delete from custom_field_manager_inv;
delete from cluster_compute_resource_inv;
delete from datacenter_inv;
delete from datacenter_network_inv;
delete from datastore_inv;
delete from dv_portgroup_inv;
delete from dv_switch_inv;
delete from folder_inv;
delete from managed_server_inv;
delete from managed_server_datastore_inv;
delete from managed_server_network_inv;
delete from network_inv;
delete from resource_pool_inv;
delete from storage_pod_inv;
delete from task_inv;
delete from vm_inv;
delete from property_map;


Cisco Champion

I am honored to be selected as a Cisco Champion. To find out more about the program, check it out here:

Cisco Champions

justin giardina, cisco champion

justin giardina, cisco champion


The Insecurity of Everything

As we all know, it is becoming a standard to have every electronic device that you own connected to the Internet. In the past, we thought of our networking gear, servers, and workstations as our main points to protect from unwanted access and malicious code. As technology progresses, everything from your watch to home appliances are now connected to the Internet. This opens up a whole new attack vector.

I recently ran in to an article by Proofpoint that outlines a recent hack that infected over 100,000 hosts to run a SPAM attack. The research shows that more than 25% of the devices used in the attacks were consumer based products, even including a refrigerator! This is just one of many stories I hear and I believe that we will start seeing a lot more of this.

Let's face it, security is not always the focus of non-security geeks. With more devices coming online, is there an industry for refrigerator security software or maybe we sill start seeing the old 90's hacking pranks start to surface again!

Link to Proofpoint article - Proofpoint Uncovers Internet of Things (IoT) Cyberattack

Filed under: Security Comments Off

Using PowerCLI To Answer All CD-ROM Override Questions

I recently ran in to a scenario where multiple cd-rom drives were disconnected from 100's of virtual machines and there were questions generated for each. In order to quickly answer and clear the questions, you can simply use this one liner after connecting to vCenter (Connect-VIServer):

Get-VM | Get-VMQuestion | Set-VMQuestion -Option "Yes" -Confirm:$false

Note than the "-Confirm:$false" will not prompt you as the script runs, taking care of each question automatically. If you would rather see each confirmation, you can remove the the Confirm switch.


Renesys Internet Traffic Misdirection Report

Just ran in to an interesting article around Internet traffic redirection. I saw this a few times being presented at Defcon, but these things usually happen quietly in the world. This is some pretty scary stuff.

Check out here.


Openstack Hands-On Labs Now Available

It is great to see the contributions from VMware around the Openstack platform, especially with the new Havana release.

VMware also released new hands-on labs around Openstack and vSphere (HOL-SDC-1320):

VMware Blog Posting


Using PowerCLI to Answer VM Questions

I recently have been testing some vendor's storage solutions and fast provisioning in vCloud Director. During the testing, I create a load simulator to mimic 1000 virtual machines inflating disks, testing write patterns, etc. In any case, during the testing, I was able to completely obliterate the vendor's write cache and write IOPS, causing datastore issues. This also caused 1000 virtual machines to get stalled due to datastores being filled up, and having a question placed on them.

In order to quickly answer these 1000 questions, this PowerCLI example worked like a charm:

Get-VM LoadTest* | Get-VMQuestion | Set-VMQuestion --Option "Cancel"

In this example, I wanted to answer "Cancel" on all VM's connected named LoadTest*