Find Restarted VMs After a VMware HA Event

On several occasions I've had the privilege of finding a VMware ESXi host hung with a Purple Screen of Death (PSOD).  Assuming you have your VMware cluster setup properly, VMware HA will automatically restart the VMs that were running on the crashed host on other hosts in the cluster.

In my experience, it's fairly easy to identify the host with the PSOD in vCenter.  But it's a bit more tricky to identify what VMs were restarted by VMware HA.  (Aka - find the victims)

I've found that PowerCLI can help with this task.

Below is a quick PowerCLI one-liner to find the VMs that were restarted.

get-cluster "Cluster Name" | get-vm | Get-VIEvent | where {$_.FullFormattedMessage -match "vSphere HA restarted virtual machine"} | select ObjectName, CreatedTime, FullFormattedMessage

You can easily add an export-csv to the end if you want to export the list of VM's into something where you edit the data.

vCenter 5 Upgrade Error - SQL Server Agent Not Running

When upgrading our vCenter server from 4.1 to 5.0 I would get a SQL Server Agent error when going through the vCenter 5.0 installer.  The error said:

"Please make sure SQL Server Agent service is running on the database server"

Our vCenter environment is comprised of a dedicated vm for vCenter and a separate vm for SQL.

Obviously the first thing I checked was to ensure the SQL Server Agent server was running on our SQL server.  It was and I even restarted it for good measure.  But that did not seem to make the error go away.

Then I did a quick search of some VMware KB’s and I found KB1036518 which matched by symptoms, but did not resolve my issue.  It basically said make sure the SQL Agent is running and that you’ve got the correct media downloaded.  Helpful!  [sarcasm]

Next I turned to the install log

VMware VirtualCenter-build-455964: 10/12/11 22:25:52 SqlState: <42000>, NativeError: <14262>, msg: <[Microsoft][SQL Server Native Client 10.0][SQL Server]The specified @job_name ('Past Day stats rollup') does not exist.>, msgLen: <119>
VMware VirtualCenter-build-455964: 10/12/11 22:25:52 ODBC Error: [Microsoft][SQL Server Native Client 10.0][SQL Server]The specified @job_name ('Past Day stats rollup') does not exist.
VMware VirtualCenter-build-455964: 10/12/11 22:25:52 SQL Server Agent is not running or query error.
VMware VirtualCenter-build-455964: 10/12/11 22:25:52 Getting Property DB_DSN_SERVER_REMOTE = 1
VMware VirtualCenter-build-455964: 10/12/11 22:25:52 errorMessage is <Please make sure SQL Server Agent service is running on the database server.

I noticed the installer was complaining about the job “Past Day stats rollup” saying it doesn’t exist.

That reminded me that we had to rebuild those jobs (VMwareKB 1004382).  We were having troubles with some of our update stat's jobs failing in the early days of vCenter 4.1

I checked with our DBA and the account the DSN uses (VMVCenter_App in screenshot above) did not have rights to that job.  He then updated the jobs to give the user in the DSN owner rights on the stats jobs.  The installer was then able to continue with the upgrade.

Long story short, make sure the account used in your DSN has rights to the SQL Agent jobs.

“The volume you have selected may not be extended" Error in DiskPart.exe on Windows 2003 Server SP2

I got an email from a colleague saying that he was having trouble increasing the size of a disk on a Windows 2003 SP2 server.  This server is a VMware virtual machine and he had already changed the size of the disk in VMware from 100 GB to 150 GB.

Problem

When trying to extend a partition on a Windows 2003 Server using diskpart.exe he got the error:

“The volume you have selected may not be extended.
Please select another volume and try again.”

Troubleshooting

I remembered seeing this error back in the VMware 3.0 days when resizing disks and then extending the partitions.  So without doing much other research I quickly found Microsoft KB 841650 and VMware KB 1007266 which both reference a bug in DiskPart.exe and a hotfix that is needed.

Well… It turns out that the Microsoft hotfix was only for pre-SP2 for Windows 2003 systems.  Since this server was running Windows 2003 SP2 the hotfix did not apply.

After looking closely at the screenshot I was sent, I noticed that the “Extended Partition” and “Free Space” sections did not look correct.  I wasn’t used to seeing those green colors and tried to figure out why it was showing like that.  I’m used to seeing show the unallocated in Disk Manager.

So took a look at a test Windows 2003 virtual machine to see if could reproduce the issue.  When I initially resized the disk in VMware and rescanned by disks in Disk Manager, I noticed the new space was showing "Unallocated" and not like the screenshot I was sent.

I then needed to try and get the disk to look like the screenshot that he sent me.  I was able to do this by right-clicking on unallocated space, selecting New Partition, Extended partition, and then selecting all of the new space.

Then my test machine was appearing like the screenshot I was sent.  I was then able to confirm that I got the same error when I’d try to run diskpart.exe extend on that volume.

Resolution

To fix I had him simply delete the new “Free Space” partition so that the space was showing unallocated.  At that point we were able to successfully extend the volume on that
disk.

Once the space was showing "Unallocated" I was able to sucessfully extend the volume with DiskPart.exe and now the partition is using all of the new space.

The lesson learned was that the space needs to show "Unallocated" if you want to extend a volume (partition) on a basic disk in Windows.

I passed this information back to my colleague and he was able to sucessfully delete the free space partition and extend his volume.

VMware Reporting Tool

This week someone told me about a slick little tool for looking at your VMware configurations. It's called RVTools and so far I have liked what I've seen.

Basically it's a lightweight .NET app that connects to your vCenter server(s) and displays your config.  It easily lets you export the config pages to a CSV where you can massage the data any way you want it.

I especially like the vHealth tab which does some checking on common issues found in VMware environments.

You can find RVTools here - http://www.robware.net/