ESXi 5.x – VM reset stuck at 95%

Few weeks ago we had an issue with freezing VM, and the only way of getting out of it was to perform hard reset to the VM. The procedure is well known – right click on the VM > Power > Reset.

tasks

The command took quite a while to actually initiate the reset. I really wondered what happened there, so I started investigating it.

vmkernel log didn’t show anything. The VM’s vmware.log log on the other hand, was quite interesting:


2014-09-25T03:44:35.209Z| vmx| VMMon_VSCSIStopVports: Invalid handle
2014-09-25T03:44:35.209Z| vmx| VMMon_VSCSIDestroyDev: Not found
2014-09-25T04:13:15.305Z| vmx| ide1:0: Command TEST UNIT READY took 2264.916 seconds (ok)
2014-09-25T04:13:15.305Z| vmx| SOCKET 9 (98) disconnecting VNC backend by request of remote manager
2014-09-25T04:13:15.307Z| vmx| MKS local poweroff

It looked like TEST UNIT READY took 37min to complete! Only after this command finished, the VM reset preparation continued. So what is this mysterious command, and what took it so long to complete?
TEST UNIT READY is a SCSI command sent to the target in order to get a response, and a status of the device. In the log file, ide1:0 is referred, but CD-ROM drive in this VM isn’t mounted according to vSphere client –

Edit-Settings

Just to make sure, I checked out the vmx file, which showed me something I didn’t see from the vSphere client side –

vmx-editor

vmx file indicates of an iso mounted to this VM. Not just an iso, but the VMware tools iso. Checking again from the guest OS side – and the disk isn’t mounted. So what just happened here?

A bug.

http://kb.vmware.com/kb/2076512

This KB has a low rating of 2 stars right now, probably since the solution didn’t include PowerCLI, right? Here it is.

Mapping of the VMs with ghosted CD-ROM:

Get-VM | Get-CDDrive | Where {$_.ExtensionData.Backing.DeviceName -like "*iso"} |  Select Parent,
@{N="DeviceName"; E={($_).ExtensionData.Backing.DeviceName}} | Export-Csv F:\Scripts\Temp\CD-Drive-mapping.csv

The actual removal of the CD-ROM from all VMs:

Get-VM | Get-CDDrive | Where {$_.ExtensionData.Backing.DeviceName -like "*iso"} | Set-CDDrive -NoMedia -Confirm:$false

Works best on 64Bit PowerShell console.

Enjoy.

iLO inventory in Multi-Domain ESXi environment

On enterprise companies, you might find a vCenter managing few DNS zones (a.serv.corp, b.serv.corp etc.). There might be a situation where there are few vCenters, each manages a different domain. Keeping track on HP iLO* IPs is easier when you register the iLO IP in the DNS, and use “ilo” as a prefix to each server – for example, esx01 iLO address will be ilo-esx01 and so on. It all works until it doesn’t, since host name changes, and maintaining it for more than just a few ESXi hosts can be demanding. Enters PowerShell.

I’ll assume you have some PowerShell / PowerCLI experience, and you know how to add the right snapin, and connect to the relevant vCenter(s).

First, add the Get-VMHostWSManInstance function, with prerequisites described here:

http://blogs.vmware.com/PowerCLI/2009/03/monitoring-esx-hardware-with-powershell.html

 function Get-VMHostWSManInstance {
            param (
            [Parameter(Mandatory=$TRUE,HelpMessage="VMHosts to probe")]
            [VMware.VimAutomation.Client20.VMHostImpl[]]
            $VMHost,
     
            [Parameter(Mandatory=$TRUE,HelpMessage="Class Name")]
            [string]
            $class,
     
            [switch]
            $ignoreCertFailures,
     
            [System.Management.Automation.PSCredential]
            $credential=$null
            )
     
            $omcBase = "http://schema.omc-project.org/wbem/wscim/1/cim-schema/2/"
            $dmtfBase = "http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/"
            $vmwareBase = "http://schemas.vmware.com/wbem/wscim/1/cim-schema/2/"
     
            if ($ignoreCertFailures) {
                    $option = New-WSManSessionOption -SkipCACheck -SkipCNCheck -SkipRevocationCheck
            } else {
                    $option = New-WSManSessionOption
            }
            foreach ($H in $VMHost) {
                    if ($credential -eq $null) {
                            $hView = $H | Get-View -property Value
                            $ticket = $hView.AcquireCimServicesTicket()
                            $password = convertto-securestring $ticket.SessionId -asplaintext -force
                            $credential = new-object -typename System.Management.Automation.PSCredential -argumentlist $ticket.SessionId, $password
                    }
                    $uri = "https`://" + $h.Name + "/wsman"
                    if ($class -cmatch "^CIM") {
                            $baseUrl = $dmtfBase
                    } elseif ($class -cmatch "^OMC") {
                            $baseUrl = $omcBase
                    } elseif ($class -cmatch "^VMware") {
                            $baseUrl = $vmwareBase
                    } else {
                            throw "Unrecognized class"
                    }
                    Get-WSManInstance -Authentication basic -ConnectionURI $uri -Credential $credential -Enumerate -Port 443 -UseSSL -SessionOption $option -ResourceURI "$baseUrl/$class"
            }
    }

 

Using System.Net.DNS .Net class isn’t enough for us, since we want to query few domain DNS servers.

You’ll need the DNSShell module installed in order to reverse lookup in each of the domains:

http://dnsshell.codeplex.com

Import-Module DnsShell

Please note that this module might not work well on 32Bit PowerShell version, so use 64Bit PowerShell console.

From this point, the script itself is straight forward, gathers all the data required including: vCenter Name, VMHost Name, Domain, iLO-IP, iLO-DNS.

Get-VMHost | Select `
@{n="vCenter"; e={($_.uid.split("@")[1]).split(":")[0] }},
@{n="VMHostName"; e={$_.Name}},
@{n="Domain"; e={$script:domain = ($_ | Get-VMHostNetwork).DomainName; $domain}},
@{n="iLOIP"; e={$script:ip = (Get-VMHostWSManInstance -VMHost ($_) -ignoreCertFailures -class OMC_IPMIIPProtocolEndpoint).IPv4Address; $ip}},
@{n="iLODNS"; e={((Get-Dns -Server $domain -Tcp $ip).Answer | Select -ExpandProperty RecordData) -join ","}} | Export-Csv -Path F:\Scripts\Output\iLO-inventory.csv F -NoTypeInformation

 

Output will look like:

excel-output-ilo

 

* HP iLO is referenced here, script will work for any other IPMI technology such as Cisco CIMC, Dell DRAC, etc.