We have been running virtual servers for over 8 years and virtual desktops for over 1 1/2 years. Our CPU & memory usage has always looked great and have not had hardly any CPU contention. Our VDI (virtual desktop infrastructure) has had performance issues from day one. We contributed most of this to our anti-virus that is installed on the VDIs. We have over 1600 VDIs and about 1300 are in use at any one time. Recently, our VDI farm was having performance issues, which we could not trace down. Even vCops (vcenter operations manager) showed green across the board. We decided to contact VMware support to assist with the issue.
The VMware support tech had me SSH into an ESXi host and look at esxtop stats. Everything looked great, except for maybe 5-8 VMs with CPU contention, but the contention kept switching between different groups of VMs. The tech thought this was strange so he wanted to take a look at the power consumption of the CPUs. This is where it got interesting. The power consumption and the CPU utilizing was not matching up. The VMware tech said that he had seen this once before and it had to do with the power save mode setting in the BIOS. After changing the setting in the BIOS from power save to power performance, our VDI environment got a new life. Wow! We got about a 50%+ performance gain. Now our VDIs are performing inline with a normal desktop. Because of this, we checked our server farm and found the same power save setting. We are in the process of changing this now. Getting that much gain by changing the power settting is crazy. There is a VMware article that states these findings:
Before Cloud Director can perform guest customization on virtual machines with pre-vista Windows guest operating systems, you must create a Microsoft Sysprep deployment package on each Cloud cell in your installation.
1. Copy the Sysprep binary files for each operating system to a convenient location on a Cloud Director server host, such as /root/sysprep
Each operating system requires its own folder, MAKE SURE you use lower case for each folder, i.e. win2000 not Win2000.
Windows 2000 SysprepBinariesDirectory /win2000
Windows 2003 (32-bit) SysprepBinariesDirectory /win2k3
Windows 2003 (64-bit) SysprepBinariesDirectory /win2k3_64
Windows XP (32-bit) SysprepBinariesDirectory /winxp
Windows XP (64-bit) SysprepBinariesDirectory /winxp_64
SysprepBinariesDirectory represents a location you choose to which to copy the binaries.
Guest OS Copy Destination
1. stop the vcd services, service vmware-vcd stop
2. Run the /opt/vmware/vcloud director/deploymentPackageCreator/createSysprepPackage.sh SysprepBinariesDirectory command.
3. Use the service vmware-vcd restart command to restart the Cloud cell.
My company is gradually moving away from EMC Recover Point and over to IBM SVC. I migrated my first server on 4/3/2011 and it went exceedingly well. I was very sceptical about the IBM SVC, but after moving 600GB from EMC CX4 to the SVC, I was astounded how easy and quick it was. After migrating over to the SVC, I then extended the volume by 400GB, making my total volume size 1TB. Normally to extend this amount of space on CX4 would take about 3 days (migrating the 600GB LUN to a 1TB LUN), but on the SVC it took 1hr! Wow is all I have to say! I am impressed. I am gradually going to start moving my ESX hosts over to SVC, but will be taking it slow.
Had an interesting thing happen today. A colleague of mine cloned a VM from our base image onto our test cluster. When he powered it on it would not connect to the network. After a little while troubleshooting the issue with the Networking team, it was found that the VM had a duplicate MAC of another VM in our production cluster. My coworker removed the NIC on the test system and added it back. This caused the VM to assign another MAC, fixing the immediate issue.
My concerns are:
- How did this happen in the first place
- How can I prevent it in the future?
We currently have two vCenters, one configured for our test hosts and the other configured for our production hosts. Within those two environments we have two distributed virtual switches with the same vlans and attached to the same physical switches. The test environment is not isolated from production. Both vCenters are linked so that they can be seen and managed in one vSphere client. All of the VMs can talk to each other along with all the hosts. The test VM and production VM were on two totally separate vSphere hosts on two different dvSwitches. I was not able to duplicate the issue, so opened an SR with VMware.
Response from VMware:
After opening the SR with VMware and after shipping them the logs and waiting for them to analyze them, they came back with this statement “The possibility of having a VM get a duplicate MAC is very low; however, it could happen. Since we cannot reproduce this issue in our labs unless we hard code the MACs on the VMs and clone them, there is not much we can do” I let the technician know that we do not hard code MAC addresses for this exact reason and for the simple fact that it would be a nightmare to maintain. The ticket was closed with no true resolution.
Virtual machine MAC address conflicts
Each vCenter Server system has a vCenter Server instance ID. This ID is a number between 0 and 63 that is randomly generated at installation time but can be reconfigured after installation.
vCenter Server uses the vCenter instance ID to generate MAC addresses and UUIDs for virtual machines. If two vCenter Server systems have the same vCenter instance ID, they might generate identical MAC addresses for virtual machines. This can cause conflicts if the virtual machines are on the same network, leading to packet loss and other problems.
Workaround: If you deploy virtual machines from multiple vCenter Server systems to the same network, you must ensure that these vCenter Server systems have unique instance IDs.
To view or change the vCenter Server instance ID:
- Log in to vCenter Server using the vSphere Client, and select Administration > vCenter Server Settings.
- Select Runtime Settings.
The vCenter Server Unique ID text box displays the current vCenter Server instance ID.
- If this ID is not unique, type a new unique value between 0 and 63 in the vCenter Unique ID text box and click OK.
- If you changed the vCenter Server instance ID, restart vCenter Server for the change to take effect.
If you have existing virtual machines with conflicting MAC addresses, edit the MAC addresses to make them unique:
- Ensure that the virtual machine is powered off.
- In the vSphere Client inventory, right-click the virtual machine and select Edit Settings.
- On the Hardware tab, select the virtual network adapter for the virtual machine.
- Under MAC Address, select Manual and specify a unique MAC address.
- Click OK.
Alternatively, you can force vCenter Server to generate a new MAC address for the virtual network adapter by configuring the virtual network adapter to use a Manual MAC address, and then reconfiguring it to Automatic.
I am in the process of moving all of my vSphere hosts to ESXi hosts. As I need to refresh hardware or rebuild a host, I use ESXi. I had downloaded ESXi a couple of months ago and was using that iso to do the installs, but started having an issue with the CD on one of my particular hosts. I kept getting hard disk errors with the ESXi CD I had created. The error I was seeing was due to a CD-ROM issue and one that I did not want to troubleshoot. I decided to create an ESXi install from USB thumb drive instead.
So searched the internet on several how-to ESXi USB installation guides and after piecing several of them together, was able to get one working. Below are the steps I used to create the ESXi install USB drive.
Steps to creating an ESXi install USB thumb drive:
- Download Syslinux (I used Syslinux 3.8 becuase 4.0.3 kept throwing errors).
- Insert the USB flash drive that you plan to use. It will require about 400 MB free space for the ESXi install files and should be formatted as FAT32. If you’re using Windows make a note of the drive letter that is assigned to the drive.
- For Windows run the following command from a CMD prompt ..\win32\syslinux.exe -mfa <drive letter> . This command will force the install (-f), alter the boot partition on the USB (-m), and mark the partition active (-a). It will also copy over the file ldlinux.sys to the root of the USB. Make sure you run the CMD window in Administrative Mode or you will not be able to create the MBR on the USB drive.
- Extract the contents of the ESXi install CD to the USB flash drive. I used Daemon Tools to mount the ISO and then copy the contents to the USB.
- On the USB flash drive rename the file isolinux.cfg to SYSlinux.cfg.
It took me several tries to get the USB drive to boot, as I forgot to run the CMD prompt in Administrative mode when creating the USB boot drive. It wouldn’t write the MBR to the USB stick so kept saying that there was no bootable device. Got the USB drive to boot, but kept getting the error when it was booting “menu.c32: not a COM32R .3age”. Searched the web for the “menu.c32: not at COM32 Rimage” and found that running the latest Syslinux was detrimental to this project, so used an older version of Syslinux, version 3.8. This resolved the error I was seeing.
After getting the USB to boot, the error “Total number of sectors not a multiple of sectors per track! Add mtools_skip_check=1 to your .mtoolsrc file to skip this test” started showing up and I could not stop it. Researched this particular error and found that the version of ESXi I was trying to run was riddled with bugs that caused the sector errors. Downloaded VMware-VMvisor-Installer-4.1.0.update1-348481.x86_64.iso and overwrote the files on the USB with the downloaded ones, this fixed the error.
One website I researched, included a kickstart file to pre populate the ESXi information. I incorporated it into my ESXi USB boot drive. Here are the files which you can just do a copy paste.
Modify the syslinux.cfg file on the USB drive and add the ks=usb parameter. This tells it to use kickstart and that the kickstart file is found on the USB drive.
menu title VMware VMvisor Boot Menu
label ESXi Installer
menu label ^ESXi Installer
append vmkboot.gz ks=usb – - vmkernel.gz – - sys.vgz – - cim.vgz – - ienviron.vgz – - install.vgz
label ^Boot from local disk
menu label ^Boot from local disk
Kickstart file (ks.cfg)
A simple kickstart file (ks.cfg) looks like this (all the bold fields, you should change for your own environment):
autopart –firstdisk –overwritevmfs
network –bootproto=static – -ip=192.168.70.76 - -gateway=192.168.70.1 – -hostname=esxihost – -device=vmnic0 – -nameserver=192.168.70.1 –netmask=255.255.255.0
The following websites are where I got some of the information posted here: