Upgrading from vCenter Server 4.1 to 5.0 fails with the Error: Exception Thrown while executing SQL script

Upgrading from vCenter Server 4.1 to 5.0 fails with the below error.
"Exception Thrown while executing SQL script"

In the %temp%\VCDatabaseUpgrade.txt log file, you see entries similar to:

[8/18/2013 5:55:27 AM] Error: Failed to execute command: if exists(select 1 from information_schema.columns
 where TABLE_NAME='VPX_HIST_STAT1'
 AND COLUMN_NAME='TIME_ID'
 AND DATA_TYPE='numeric')
 ALTER TABLE VPX_HIST_STAT1 ALTER COLUMN TIME_ID BIGINT NOT NULL;
[8/18/2013 5:55:27 AM] Got exception: ERROR [42000] [Microsoft][SQL Server Native Client 10.0][SQL Server]The index 'X_VPX_HIST_STAT1' is dependent on column 'TIME_ID'.
ERROR [42000] [Microsoft][SQL Server Native Client 10.0][SQL Server]ALTER TABLE ALTER COLUMN TIME_ID failed because one or more objects access this column.
[8/18/2013 5:55:27 AM] Error while upgrading: ERROR [42000] [Microsoft][SQL Server Native Client 10.0][SQL Server]The index 'X_VPX_HIST_STAT1' is dependent on column 'TIME_ID'.
ERROR [42000] [Microsoft][SQL Server Native Client 10.0][SQL Server]ALTER TABLE ALTER COLUMN TIME_ID failed because one or more objects access this column.
[8/18/2013 5:55:27 AM] Info: Exiting Upgrade Wizard
[8/18/2013 5:55:29 AM] Successfully run: alter database [VCDB] set recovery SIMPLE


This issue is documented in the VMware KB - 2011533

However the steps provided do not work since the resolution steps in the KB talk about dropping the index named 'INDX_TIME_ID'. In my case, the index was 'X_VPX_HIST_STAT1' and therefore I had to alter the commands accordingly.

DROP INDEX VPX_HIST_STAT1.X_VPX_HIST_STAT1

And most IMPORTANTLY, perform the steps mentioned in the above KB, after you restore the vCenter Database from a backup.
If you were to perform the steps on the same DB on which the upgrade failed, you will run into the issue mentioned in VMware kb 2020504

VMware Converter - P2V of a physical server with multiple disks fails

I had the opportunity to work on a issue recently where the system administrator was trying to convert a physical server to a VM and the conversion was failing at the initial task of collecting the source machine details.

The default logs did not provide any information and therefore I went ahead and enabled verbose logging. Following this I was able to see the following in the logs.

2013-08-06T13:48:32.561+02:00 [04248 verbose 'Default'] Mapping a recognized volume \\?\Volume{132d9331-1f9f-441f-a37d-1283440beccb}\ with ID \WindowsBitmapDriverVolumeId=[44-4D-49-4F-3A-49-44-3A-31-93-2D-13-9F-1F-1F-44-A3-7D-12-83-44-0B-EC-CB]
2013-08-06T13:48:32.561+02:00 [04248 verbose 'Default'] [PopulateVolumeProperties] volume \\?\Volume{0b323bcd-7f6f-4473-9acf-7699b247f96e}\  corresponds to \Device\HarddiskDmVolumes\Ampsv01960pjtDg0\Volume31 device name.
2013-08-06T13:48:32.561+02:00 [04248 verbose 'Default'] [,0] error [Windisk_CheckNonFixedMediaDevice] Failed to retrieve media type for \\?\Volume{0b323bcd-7f6f-4473-9acf-7699b247f96e}\: Incorrect function (1)
2013-08-06T13:48:32.561+02:00 [04248 verbose 'Default'] [GetDriveLetterOrMountPoint] Mount point for volume \\?\Volume{0b323bcd-7f6f-4473-9acf-7699b247f96e}\ is V:\TlogTW2\
2013-08-06T13:48:32.561+02:00 [04248 verbose 'Default'] [,0] Info[Windisk_GetVolumeDeviceNumber]: IOCTL_GET_STORAGE_DEVICE_NUMBER for volume \\?\Volume{0b323bcd-7f6f-4473-9acf-7699b247f96e}\ is not supported .... Trying NT method.
2013-08-06T13:48:32.561+02:00 [04248 verbose 'Default'] [,0] Error[Windisk_GetVolumeDeviceNumber]:Failed to get \\?\Volume{0b323bcd-7f6f-4473-9acf-7699b247f96e}\ Volume's disk number
2013-08-06T13:48:32.561+02:00 [04248 verbose 'Default'] [,0] Info[Windisk_GetVolumePartitionInfo]:Number of disk extents is 6. Volume \\?\Volume{0b323bcd-7f6f-4473-9acf-7699b247f96e}\ is a dynamic volume. No meaningful partition information can be associated with it
2013-08-06T13:48:32.561+02:00 [04248 verbose 'Default'] [,0] Info[Windisk_GetVolumePartitionInfo]:Number of disk extents is 6. Volume \\?\Volume{0b323bcd-7f6f-4473-9acf-7699b247f96e}\ is a dynamic volume. No meaningful partition information can be associated with it
2013-08-06T13:48:32.561+02:00 [04248 verbose 'Default'] [PopulateVolumeProperties] Unable to get \\?\Volume{0b323bcd-7f6f-4473-9acf-7699b247f96e}\ volume's active flag info.
2013-08-06T13:48:32.561+02:00 [04248 verbose 'Default'] [GetVolumeProperties] finished.

Checking the system revealed that it had nearly 120+ disks mounted from the SAN. Checking the VMware community did not show any maximums for the number of disks that can be connected to the source server that is being converted.
Anyways, we unmapped the SAN disks and then initiated the conversions which went ahead smoothly. This solution was acceptable to the user, luckily.

If anybody has any information on what is the maximum number of disks supported by VMware converter, please share the information.

VMware tools Upgrade/Install fails with ERROR 1304

Recently I was asked for assistance with a VMware Tools upgrade issue. The tools upgrade on a couple of VMs (one running Windows Server 2000 and the other running Windows Server 2003) was failing with the below messages.

Error 1304
Error writing to file C:\\WINNT\system32\mfcm80.dll. Verify that you have access to that directory.
(Or )
Error 1304
Error writing to file C:\\WINNT\system32\wbem\vmStatsProvider.mof. Verify that you have access to that directory.

Resolution

It appeared to be a simple issue and I expected to find a solution on the web. But to my surprise there was not one site which talked about this issue. I narrowed it down to the McAfee Antivirus software.
To fix this issue, I had to disable (I repeat - DISABLE) the ACCESS PROTECTION in the VirusScan Console (shown below) during the upgrade.

It is a simple fix, but I wanted to blog about it since it was not documented on the web.

Update:
VMware has indeed released an article this week for the above issue.
VMware Tools installation fails when McAfee Antivirus is in Maximum Protection mode

Controlling logging (file retention and rotation) in VMware Servers

The other day one of my colleagues asked me a question on configuring logging with ESXi and I was not able to provide a complete answer. I try to find the necessary information and then realized that that there was no single page that clearly explained the logging behavior of both ESX and ESXi in their different versions.

I did some digging and my findings are as follows.
Hope this is of some help!!

ESX 4.x (classic version):

(All log files other than hostd and vpxa log file)

The default configuration for the various log files is controlled by "/etc/logrotate.conf" file.

Log file rotation is daily, weekly, monthly or yearly.

Log file retention is controlled by a numerical value - 'rotate 4' , etc.

So if we need to have 4 weeks of log files, the settings would be 'weekly' and 'rotate 4'.

In addition to this file, we have specific config files located in "/etc/logrotate.d" folder. Some of them are as follows.

syslog

vmkernel

vmkiscsid

vmkproxy

vmksummary

vmkwarning

We can control the logging for these log files individually. For example if we need vmkernel log files to be rotated on a daily basis and retention to be for 30

days, the settings in the corresponding file would be as follows.

--daily

--rotate 30

In addition to this we can also control the size of the file, by specifying the max size.

--size 200k

For log files not located in this folder, "/etc/logrotate.d", they are controlled by the settings in "/etc/logrotate.conf"

hostd log file - ESX & ESXi - all versions (4.x & 5.x)

Necessary changes have to be made in the below file.

/etc/vmware/hostd/config.xml

<directory>/var/log/vmware/</directory>

<level>verbose</level> -------trivia / info / verbose / .....

<maxFileSize>5242880</maxFileSize>

<maxFileNum>50</maxFileNum>

vpxa log file - ESX & ESXi - all versions (4.x & 5.x)

Necessary changes have to be made in the below file.

/etc/opt/vmware/vpxa/vpxa.cfg

<maxFileNum>50</maxFileNum> -- Maximum number of log files

<maxFileSize>5242880</maxFileSize> -- Maximum size of each log file, in bytes.

<level>verbose</level> -- Log Level (The default level is "verbose". To increase it, change it to "trivia".)

ESXi 4.x - messages log file

By default, only 24 hours of logs are retained. Logs are lost on reboot. (as per vSphere hardening guide).

To change the defaults, log into the busybox console and configure the syslogd.

#busybox syslogd --help

Ref: http://www.virtuallyghetto.com/2010/06/increase-syslog-count-in-esxi-using.html

ESXi 5.x

Note:

The logging in ESXi 5.x is redirected to syslog by default, which is not the case in older versions where you need to configure syslog only if you are redirecting the logs to a syslog server. In ESXi 5.x, the syslog service is responsible for sending the log files to the scratch partition.

If you dont want to use syslog, and want to log files directly to disk, then you need to make a change in "/etc/vmware/hostd/config.xml".

Change the below value to true.

<!-- If true, log to files on disk -->

<outputToFiles>false</outputToFiles>

VMware vSphere ESXi 5.0 hosts run a syslog service (vmsyslogd) that provides a standard mechanism for logging messages from the VMkernel and other system components.

Local and Remote syslog functionality can be configured for a host using the esxcli command line utility.

esxcli system syslog config set --help

--default-rotate=<long>

Default number of rotated logs to keep

--default-size=<long> Default size of logs before rotation, in KiB

--logdir=<str> The directory to output logs to

--logdir-unique Place logs in a unique subdirectory of logdir, based on hostname

--loghost=<str> The remote host to output logs to

--reset=<str> Reset values to default

esxcli system syslog config logger list

-- Show the currently configured sub-loggers - different log files

Notes:

Logs sent to Syslog server - KB 2003322 - Retention, rotation and splitting of logs received and managed by a syslog server are fully controlled by that syslog server. ESXi 5.0 cannot configure or control log management on a remote syslog server.

Regardless of the additional syslog configuration specified using these options, logs continue to be placed on the default locations on the ESXi host.