Check root password with PowerCLI (Multi-threading!)

Keeping your root password similar in all of your ESXi hosts, is one of the virtual environment key methods to control and maintain large environments. It will make it easier to connect directly to a host in case of vCenter failure, access SSH for troubleshooting and control it from DCUI.

There is a great one-liner by Kelvin Wong, that allows you to get a list of all of the VMhosts that have different password than the standard one.

In this post, I’ll try to:

  1. Make it simpler for PowerCLI beginners to use this script
  2. Provide advanced users with methods of multi-threading in PowerShell 2.0

First, the script for getting a list of hosts with non standard password :

$vCenterName = "vcenter.company.corp"  # ChangeME
$ExportFileLocation = "F:\Scripts\Harel_CheckrootPassword\PasswordNotMatch.txt" # ChangeME
$rootpassword = Read-host -assecurestring -prompt "Please enter local root password"
 
if (!(get-pssnapin -name VMware.VimAutomation.Core -erroraction silentlycontinue)) {
    add-pssnapin VMware.VimAutomation.Core}
 
Connect-VIServer $vCenterName

$vmhostsView = get-view -ViewType HostSystem -Property Name,Summary.runtime.ConnectionState `
| Where {$_.Summary.runtime.ConnectionState -eq "connected"} | %{$_.Name}

if (Test-Path  ($ExportFileLocation) -pathtype leaf)
{Remove-Item $ExportFileLocation -Confirm:$False}

$vmhostsView | %{ $err = @() ; 
connect-viserver $_ -user root -password $rootpassword -EA silentlycontinue -EV err ; 
if ($err.count -gt 0) { $_ | out-file $ExportFileLocation -append }
else {disconnect-viserver $_ -force -confirm:$false} }

The file created by this script, contains a list of hosts with root password different than the one typed as input.

BUT, the script will test connection to each of the hosts one-by-one, which may take a while if you have more than 10 hosts. In an environment of 74 hosts, for example, it took the script 10:30 min to run (calculated with measure-command of course):

serial-10.30

We will reduce the run time of the script, by using parallelism of the hosts check. Multi-threading in PowerShell.

The script will start the same way as the one above –

$vCenterName = "vcenter.company.corp"  # ChangeME
$ExportFileLocation = "F:\Scripts\Harel_CheckrootPassword\PasswordNotMatch.txt" # ChangeME
$rootpassword = Read-host -assecurestring -prompt "Please enter local root password"
 
if (!(get-pssnapin -name VMware.VimAutomation.Core -erroraction silentlycontinue)) {
    add-pssnapin VMware.VimAutomation.Core}
 
Connect-VIServer $vCenterName

$vmhostsView = get-view -ViewType HostSystem -Property Name,Summary.runtime.ConnectionState `
| Where {$_.Summary.runtime.ConnectionState -eq "connected"} | %{$_.Name}

if (Test-Path  ($ExportFileLocation) -pathtype leaf)
{Remove-Item $ExportFileLocation -Confirm:$False}

And now, adding the interesting part –

function CanWeAddJob(){
$Alljobs=(Get-Job -State "Running" | measure-Object).count
if ($alljobs -lt 10) {return $true}
else {return $false}
}

Foreach ($vmhostname in $vmhostsView){
Do {Start-Sleep -Milliseconds 500}
while (!(CanWeAddJob))

$args = ($vmhostname,$credentials,$ExportFileLocation)

Start-Job -Name $vmhostname -ArgumentList $args -InitializationScript {Add-PSSnapin VMware.VimAutomation.Core} –Scriptblock {$err = @() ;
Connect-VIServer $args[0] -Credential $args[1] -EA silentlycontinue -EV err ; 
if ($err.count -gt 0) { $args[0] | out-file $args[2] -append } 
else {disconnect-viserver $_ -force -confirm:$false} } -RunAs32
}

Explanation:

Function CanWeAddJob() is checking what is the running job count in your PowerShell session, and determine whether to add more job, or not. In this example, it does it to a maximum of 10 parallel jobs.

Foreach is here to split the long VMHosts list to many separated tasks. It will only add task to the running job queue, if the queue have less than 10 jobs.
PSSnapin was added to each of the new powershell.exe instances created, with RunAs32 parameter, to make it take less RAM of your server / workstation.

While the script is running, you should see something like this in your task manager:

task-manager

Can you guess what was the run time of the multi-threaded script?

parallel-2.42

2:42 min, which saved me 75% of the original run time.

ESXi 5.x – VM reset stuck at 95%

Few weeks ago we had an issue with freezing VM, and the only way of getting out of it was to perform hard reset to the VM. The procedure is well known – right click on the VM > Power > Reset.

tasks

The command took quite a while to actually initiate the reset. I really wondered what happened there, so I started investigating it.

vmkernel log didn’t show anything. The VM’s vmware.log log on the other hand, was quite interesting:


2014-09-25T03:44:35.209Z| vmx| VMMon_VSCSIStopVports: Invalid handle
2014-09-25T03:44:35.209Z| vmx| VMMon_VSCSIDestroyDev: Not found
2014-09-25T04:13:15.305Z| vmx| ide1:0: Command TEST UNIT READY took 2264.916 seconds (ok)
2014-09-25T04:13:15.305Z| vmx| SOCKET 9 (98) disconnecting VNC backend by request of remote manager
2014-09-25T04:13:15.307Z| vmx| MKS local poweroff

It looked like TEST UNIT READY took 37min to complete! Only after this command finished, the VM reset preparation continued. So what is this mysterious command, and what took it so long to complete?
TEST UNIT READY is a SCSI command sent to the target in order to get a response, and a status of the device. In the log file, ide1:0 is referred, but CD-ROM drive in this VM isn’t mounted according to vSphere client –

Edit-Settings

Just to make sure, I checked out the vmx file, which showed me something I didn’t see from the vSphere client side –

vmx-editor

vmx file indicates of an iso mounted to this VM. Not just an iso, but the VMware tools iso. Checking again from the guest OS side – and the disk isn’t mounted. So what just happened here?

A bug.

http://kb.vmware.com/kb/2076512

This KB has a low rating of 2 stars right now, probably since the solution didn’t include PowerCLI, right? Here it is.

Mapping of the VMs with ghosted CD-ROM:

Get-VM | Get-CDDrive | Where {$_.ExtensionData.Backing.DeviceName -like "*iso"} |  Select Parent,
@{N="DeviceName"; E={($_).ExtensionData.Backing.DeviceName}} | Export-Csv F:\Scripts\Temp\CD-Drive-mapping.csv

The actual removal of the CD-ROM from all VMs:

Get-VM | Get-CDDrive | Where {$_.ExtensionData.Backing.DeviceName -like "*iso"} | Set-CDDrive -NoMedia -Confirm:$false

Works best on 64Bit PowerShell console.

Enjoy.