PowerCLI – Rescan HBAs for a Cluster on a Host by Host Basis
After having some bad luck with rescanning HBAs for entire clusters or datacenters all at the same time (Cliff's Notes: the LUNs ended up looking like array based snapshots and therefore unusable), it was decided that any rescans should be done on an individual host basis. Below is the script I created to achieve this goal.
When you run the script, it will ask for the Cluster name so you don't have to modify the code.
#Rescan HBAs for a cluster on a host by host basis
$InputCluster = Read-Host "Cluster Name"
$vmhosts = get-cluster $InputCluster | get-vmhost
foreach ($i in $vmhosts) {
Write-Host "Starting rescan of all HBAs on $i"
$i | Get-VMHostStorage -RescanAllHba | Out-Null
Write-Host "Rescan of all HBAs on $i is complete"
Write-Host ""
}
Click here to download a text file containing the script.
Note: this was a script that worked in my environment. There is no warranty or support with this script, please use at your own risk.
PowerCLI – Analyze a Cluster to Move VMs to a Resource Pool
After someone deployed a bunch of VMs, we let them know about the Resource Pool they were supposed to be deployed to. Oops. To correct this, and to avoid a couple hours of dragging and dropping VMs into a resource pool, I was able to create a script that detects if a VM is outside of a Resource Pool and then move it to the specified Resource Pool.
When you run the script, it will ask for the Cluster name and the Resource Pool name so you don't have to modify the code.
#Analyze the Clusters and check for systems outside of Resource Pools
$InputCluster = Read-Host "Cluster Name"
$InputRP = Read-Host "Resource Pool Name"
#Cluster which will be analyzed
$cluster = Get-Cluster $InputCluster
#Resource Pool where VMs should be moved
$rp = Get-ResourcePool $InputRP
#Detection of VMs not in Resource Pools
$norp = Get-Cluster $cluster | Get-VM | where {$_.ResourcePool.Name -eq "Resources"}
foreach ($i in $norp) {
Write-Host "Moving $i to Resource Pool $rp"
Get-VM $i -Location $cluster | Move-VM -Destination $rp | Out-Null
Write-Host "Move complete"
Write-Host ""
}
Click here to download a text file containing the script.
Note: this was a script that worked in my environment. There is no warranty or support with this script, please use at your own risk.
Fusion-IO Caching Tests…
After some issues with the setup and configuration of the Fusion-IO ioCache cards we picked up, I finally got to dig in and do some basic testing with IOMeter.
To setup the test, I configured a new 20GB VMDK on it's own paravirtualized SCSI Controller. The drive was formatted NTFS in a full/non-quick method as the F:\ drive. The IOMeter test was run on a single worker against the entire F:\ drive. The All-in-one test was selected and run for 20 minutes.
First up, I tested the drive all by itself with no caching enabled:
| IOps: | 1348.69 | Read IOps: | 670.1738 | Write IOps: | 678.5159 |
Next, I tested Volume based caching. I started off by making the following modifications to the Fusion-IO tab within vCenter as follows to add only the F:\ drive to the Volume Caching Filter:
Then I reset the F:\ drive by formatting it again as NTFS in a full/non-quick method. Once the format was complete, I reran the IOMeter test and received these results:
| IOps: | 1486.163 | Read IOps: | 737.5608 | Write IOps: | 748.6018 |
Lastly, I tested the Drive based caching. I went back to the Fusion-IO tab within vCenter and removed the Volume Caching Filter on the F:\ drive and then set the Drive Caching Filter to Drive1 (Drive0 was the drive the OS was installed on, Drive2 was the drive which is attached by FusionIO automatically):
Then I reset the F:\ drive by formatting it again as NTFS in a full/non-quick method. Once the format was complete, I reran the IOMeter test and received these results:
| IOps: | 1509.644 | Read IOps: | 748.7889 | Write IOps: | 760.8555 |
I also managed to grab a shot of the Performance graphs for the disk during the tests via vSphere client: (pardon the lapse between 2PM and 3PM on the graph)
So to review and put the results all on the same table:
| No Caching Enabled | IOps: | 1348.69 | Read IOps: | 670.1738 | Write IOps: | 678.5159 |
| Volume Caching Enabled | IOps: | 1486.163 | Read IOps: | 737.5608 | Write IOps: | 748.6018 |
| Drive Caching Enabled | IOps: | 1509.644 | Read IOps: | 748.7889 | Write IOps: | 760.8555 |
Remember, these are just initial results with nothing but having the card installed, drivers installed, firmware upgraded, ioTurbine installed, and the guest package installed. While some of the results weren't exactly what I was expecting, I'm pretty excited to dig in deeper to see what kind of performance we can gain out of these cards.
Small update...
While this particular blog post is about caching, since that's how these cards will be used in this environment, I couldn't help but go back, mount the Fusion-IO card as VMFS storage, SvMotion the F:\ drive over to the Fusion-IO VMFS datastore and run the test again. So once again, the F:\ drive was formatted as NTFS in a full/non-quick method. Once the format was complete, I reran the IOMeter test and received these results:
| IOps: | 5443.904 | Read IOps: | 2700.201 | Write IOps: | 2743.703 |
Updating the Fusion-IO firmware…
We were fortunate enough to pick up some Fusion-IO IOcache cards to be implemented in our new infrastructure.
Installed the Fusion-IO drivers on the ESXi box and went to check the cards out. We do a "fio-status" command vis SSH connection to pull the information on the cards in each system.
Towards the bottom, you can see the Firmware listed as v5.0.7 and at the time the current release was v7.1.13. So head out to the Fusion-IO support site (http://support.fusionio.com/), log in with your support account credentials, click on the "Downloads" button, select the card installed, select the OS being used, and look for the firmware menu. Grab the firmware file, should be a file ending with an FFF extension.
Once it's downloaded, upload the firmware to a place the ESXi host can read from. In this case, I uploaded the firmware to a VMFS datastore so I can use the same file for every server.
At this point, we can get the update installing. To do that, SSH into the desired host (remember to have the SSH service running), and issue a "fio-update-iodrive *firmware location*" command.
After about 12 minutes, the install was complete and a reboot was required.
Once the ESXi host was back up, SSH in to the ESXi host and run another "fio-status" command and show that the Firmware is now indeed v7.1.13
If you happen to run into the same issue where the "Active Warnings" indicate a message of: The ioMemory is currently running in a minimal state. The problem I found was that the driver version was incompatible. Check out my other posting on how to update the driver: http://www.thatcouldbeaproblem.com/?p=602
Installing and/or Updating the Fusion-IO Drivers on an ESXi Host…
We were fortunate enough to pick up some Fusion-IO IOcache cards to be implemented in our new infrastructure.
In this case, Fusion-IO drivers on the ESXi box were slipstreamed into the install image. To check the driver version we do a "fio-status" command via SSH connection to pull the information on the cards in each system. If the drivers are not yet installed, move on the the next step.
Grab the drivers from the Fusion-IO support site (http://support.fusionio.com/), log in with your support account credentials, click on the "Downloads" button, select the card installed, select the OS being used, and look for the "Software Binaries" section and download the offline_bundle zip file. Once it's downloaded, upload it to somewhere the ESXi host can read it and run the following command: esxcli software vib install -d *full path to offload_bundle zip*
Once the install process has completed, reboot the host. Once it comes back up, SSH back into the ESXi host and run the "fio-status" command. It should now be updated to the version you just installed. In this case, version 3.2.2.
Using the Cold Clone 3.0.3 ISO with vSphere 5
As many know, the Cold Clone ISOs have been discontinued and support has been removed from VMware. This is quite unfortunate especially when you get into the business of P2V’ing items like domain controllers, SQL servers, and other pesky boxes.
I understand the converters from VMware have improved by leaps and bounds over what it was in version 3, but there’s still a reasonable amount of security in P2V’ing a box while it’s completely down with no running services whatsoever.
First issue I ran into while attempting to P2V an old SQL server was finding the ISO. It has almost been removed from the internet as a whole. So I've uploaded it here: Cold Clone 3.0.3 Link
Second issue, boot time. It took me literally 15 minutes from the point of “hitting a button to boot to disc” to the point of accepting the EULA.
Third issue, drivers. I was lucky enough for the internal NICs to be found but the add-on NICs were not. To put this into perspective, the server I was working on was an HP G5. I tested with an HP G6 and it did not find any NICs at all.
Fourth issue, disappearing network settings. Every time I went to the networking settings, they would be cleared. The settings held as long as I clicked apply and then ok, but the second you went back to the network config menu I had to renter all the info.
Fifth issue, vCenter integration. It’s not exactly shocking that it didn’t work with vCenter, but I was hopeful. The converter would go through and recognize everything out of vCenter, but then ran into a bunch of issues as soon as the clone actually starts. Such as:
Couldn’t find the Distributed vSwitch vNIC:
[managedVMCreator,2657] No network named "SYSMGMT" was found
[imageProcessingTaskStep,194] VmiImportTask::task{9} step "create VM" destroyed
[vmiImportTask,439] Error during creation of target VM
[imageProcessingTaskStep,194] VmiImportTask::task{9} step "create and clone to VM" destroyed
[imageProcessingTaskStep,194] VmiImportTask::task{9} step "Clone VM" destroyed
[imageProcessingTaskImpl,552] VmiImportTask::task{9}: Image processing task has failed with MethodFault::Exception: vim.fault.NotFound
[imageProcessingTaskImpl,154] VmiImportTask::task{9}: SetState to error
Couldn’t find the FC attached storage on a host:
[NFC ERROR] NfcNewAuthdConnectionEx: Failed to connect to peer. Error: Host address lookup for server esx01 failed: The requested name is valid, but no data of the requested type was found
NBD_ClientOpen: Couldn't connect to esx01:902 Host address lookup for server esx01 failed: The requested name is valid, but no data of the requested type was found
DISKLIB-DSCPTR: : "vpxa-nfc://[VMFS_015] VM01/VM01.vmdk@esx01:902!52 f1 8e a1 39 1c c1 f8-9c d4 05 71 1a 4f ae c6" : Failed to open NBD extent.
DISKLIB-LINK : "vpxa-nfc://[VMFS_015] VM01/VM01.vmdk@esx01:902!52 f1 8e a1 39 1c c1 f8-9c d4 05 71 1a 4f ae c6" : failed to open (NBD_ERR_NETWORK_CONNECT).
DISKLIB-CHAIN : "vpxa-nfc://[VMFS_015] VM01/VM01.vmdk@esx01:902!52 f1 8e a1 39 1c c1 f8-9c d4 05 71 1a 4f ae c6" : failed to open (NBD_ERR_NETWORK_CONNECT).
DISKLIB-LIB : Failed to open 'vpxa-nfc://[VMFS_015] VM01/VM01.vmdk@esx01:902!52 f1 8e a1 39 1c c1 f8-9c d4 05 71 1a 4f ae c6' with flags 0x2 (NBD_ERR_NETWORK_CONNECT).
[diskHandleWrapper,87] DiskLib_Open failed on vpxa-nfc://[VMFS_015] VM01/VM01.vmdk@esx01:902!52 f1 8e a1 39 1c c1 f8-9c d4 05 71 1a 4f ae c6 with error NBD_ERR_NETWORK_CONNECT.
[imageProcessingTaskImpl,552] BlockLevelCloning::task{21}: Image processing task has failed with MethodFault::Exception: sysimage.fault.DiskLibConnectionFault
[imageProcessingTaskImpl,154] BlockLevelCloning::task{21}: SetState to error
How I actually got it to work was by sending the P2V directly to a specific ESXi 5 host. Once I did that, I could P2V the system to the locally attached storage or the FC attached storage. I attached the vNIC to a standard vSwitch.
Playing with ESXi and Synology in the home lab…
So I finally bit the bullet and bought a NAS for my home lab. Ended up with a Synology DS411 due to some good timing from a NewEgg sale.
I went with Synology for two reasons: 1. I could supply my own disks (I don't want the "Green" drives that came with most other NAS units) 2. I always hear REALLY good things about them, they're always highly recommended.
I happened to already have a set of 4 Hitachi Deskstars sitting around so I tossed them in and started running some tests with IOMeter (Run all tests). I did a test on a RAID5 grouping of all 4 disks which were presented to the ESXi host as an iSCSI target and then I did the same test with a RAID 5 grouping of all 4 disks which were then presented to the ESXi host via NFS. I was amazed at the results.
From a personal standpoint, I fully expected iSCSI to blow away NFS. Wow, was I wrong. iSCSI pulled a whopping 270 IOPS from the IOMeter test. Then I ran the exact same test only via an NFS share... NFS blew iSCSI out of the water with a 1,070 IOPS. My jaw is still on the floor from that result. Here's some of the rest of the results:
I'm also seeing some excellent regular performance as well. 45MBps worth of write during an SvMotion.
So to say the least, I'm quite impressed at what this little guy can do with 4 SATA 7200RPM drives and a single NIC.
One other tidbit to add in an unrelated fashion is how much cooler the new Hitachi Deskstar runs compared to the older ones. A full 12F cooler. That was another surprise...
Standalone ESXi 5 Host Upgrade
Have an ESXi host which is a standalone box? No VMware Update Manager? No vMA?
Well, they still require patches. Luckily enough, you can still use the stripped down version of the console which is included in ESXi to update it.
Start by heading out to the VMware Patches portal http://www.vmware.com/patchmgr/download.portal and download the neccessary patches for the server that needs patched.
Upload the patch zip file to a datastore that the server can talk to via either SCP or the datastore browser
Next, make sure the SSH service has been started.
To do this while in the vSphere Client, click on the desired host, and click on the "Configuration" tab followed by the "Security Profile" link in the "Software" box, then click on "Properties" in the top right side.
Highlight "SSH" and then click "Options", after the SSH Options screen pops up, click on "Start", then click "OK" twice to get back to the Configuration tab.
After getting connected to the ESXi host, run the command: esxcli software vib install -d *full path to uploaded zip*
Example: esxcli software vib install -d /vmfs/volumes/VMO-01 Datastore/Temp/update-from-esxi5.0-5.0_update01.zip
There should be a message showing that the update was completed successfully and that the system needs to be rebooted.
If ready to reboot, type in "reboot" and the system will reboot. Just remember to check to make sure that the SSH service has been stopped when it boots back up.
One error that I ran into, if you don't give the full path to the zip file containing the update, the patching will fail with a "MetadataDownloadError" reading:
Could not download from depot at zip:/var/log/vmware/*update name*.zip?index.xml, skipping (('zip:/var/log/vmware/*update name*.zip?index.xml', '', "Error extracting index.xml from :/var/log/vmware/*update name*.zip: [Errno 2] No such file or directory: '/var/log/vmware/*update name*.zip?index.xml'"))
url = zip:/var/log/vmware/*update name*.zip?index.xml
Please refer to the log file for more details.
Once I put in the full path, it worked just fine.
Registering the EQL HIT Kit to a New vCenter
Instead of the planned upgrade we were going to perform, we decided to start from scratch and do a full reinstall of our environment. So that entailed registering the Dell EqualLogic HIT Kit to a new VirtualCenter.
Start off by opening up the console on the VM and logging in. (Default Username: root Default Password: eql) Once logged in, select Option 8 to unregister it from the old vCenter.
From there, select Option 4 to configure vCenter. Enter in the credentials for the new vCenter (IP, admin account, password, EQL HIT Kit Appliance IP, and an admin email addres), confirm the credentials and the appliance should connect to vCenter and be successful.
Once back to the main screen, select Option 7 to register the appliance with vCenter and then reboot the appliance
After the appliance is back at the login prompt, check back to the vCenter "Solutions and Applications" section and make sure that the EqualLogic utilities are there. For good measure, login to one of the utilities and ensure the configuration is correct.
Generating RSA Key & CSR for use with VMware Solutions…
Ever received a Security Warning while logging into either you ESX/i host and/or vCenter?
That's due to the SSL certificate being untrusted with your machine. You can always click the "Ignore" button or check the "Install this certificate..." box and then "Ignore" and move on, however you can improve the security by replacing the certificates with certificates signed by a commercial certificate authority (CA).
To generate an RSA Key and certificate signing request (CSR), we'll start by downloading the OpenSSL-Light application on the system you'll be installing or have already installed a VMware application. The application is available from the following site: http://www.slproweb.com/products/Win32OpenSSL.html
Download the "Win32 OpenSSL v1.0.0e Light" application along with the "Visual C++ 2008 Redistributables". Once downloaded, run the Visual C++ file (in this case, "vcredist_x86.exe"). Click "Next", check the "I have read and accept the license terms." box and click "Install", wait a couple seconds and click "Finish".
Now it's time to install OpenSSL by running the "Win32OpenSSL_Light-1_0_0e.exe" and installing it to your desired location. Click "Next", accept the agreement and click "Next", choose an install location (default is the root of C:\, but I don't like cluttering up the root of C:\) and click "Next", click "Next", change the option so that the OpenSSL DLLs are copied to the OpenSSL binaries (/bin) directory and click "Next", then click "Install", once the installer is finished click "Finish".
From this point, open up a command prompt and navigate to the bin folder within the location of the installation of OpenSSL. To generate the key, run the following command: openssl genrsa 1024 > rui.key Once that is complete, generate the CSR by running this command: openssl req -new -key rui.key > rui.csr After running the command, you'll be asked to populate some information regarding your country name, state, city, organization name and unit, common name and email address.
If you happen to receive the error: "WARNING: can’t open config file: /usr/local/ssl/openssl.cnf" this is due to OpenSSL being unable to find the openssl.cnf file. To correct this error, run the following command: set OPENSSL_CONF=c:\[PATH TO OPENSSL DIRECTORY]\bin\openssl.cfg
After creating the CSR, submit it to either the admin of your Microsoft Certificate Services CA or to whomever handles the certificates from a commercial CA.
Upcoming Events
Recent Posts
- PowerCLI – Copying PortGroups from a Standard vSwitch to a Distributed vSwitch
- PowerCLI – Rescan HBAs for a Cluster on a Host by Host Basis
- PowerCLI – Analyze a Cluster to Move VMs to a Resource Pool
- Fusion-IO Caching Tests…
- Updating the Fusion-IO firmware…
Certifications
My Home Lab vMetrics
10 GHz Total CPU 
16 GB Total RAM 
7,578 GB Total Disk
1 Host(s)
1 RPs
8 VMs
0 vMotions
(4) 
(4) 
(0)
3 Physical NICs
3 Virtual PGs
Tags
5.1.1 2008 R2 2008R2 6510E 8024F Data Recovery Dell EQL Equallogic ESXi ESXi 5 firmware Fusion-IO HIT HIT/VE HIT Kit install iSCSI MEM Microsoft Multipathing OVA OVF plugin PowerCLI PowerConnect Round Robin Server snapshot Update Manager Upgrade vCenter vDR VirtualCenter Virtual Center vMotion VMUG VMware VMworld vSphere vSphere 5 VSS vSwitch VUM Windows
WP Cumulus Flash tag cloud by Roy Tanck requires Flash Player 9 or better.