That… could be a problem…

19Feb/130

Installing and/or Updating the Fusion-IO Drivers on an ESXi Host…

We were fortunate enough to pick up some Fusion-IO IOcache cards to be implemented in our new infrastructure.

In this case, Fusion-IO drivers on the ESXi box were slipstreamed into the install image. To check the driver version we do a "fio-status" command via SSH connection to pull the information on the cards in each system. If the drivers are not yet installed, move on the the next step.

Grab the drivers from the Fusion-IO support site (http://support.fusionio.com/), log in with your support account credentials, click on the "Downloads" button, select the card installed, select the OS being used, and look for the "Software Binaries" section and download the offline_bundle zip file. Once it's downloaded, upload it to somewhere the ESXi host can read it and run the following command: esxcli software vib install -d *full path to offload_bundle zip*

Once the install process has completed, reboot the host. Once it comes back up, SSH back into the ESXi host and run the "fio-status" command. It should now be updated to the version you just installed. In this case, version 3.2.2.

3Nov/123

Using the Cold Clone 3.0.3 ISO with vSphere 5

As many know, the Cold Clone ISOs have been discontinued and support has been removed from VMware. This is quite unfortunate especially when you get into the business of P2V’ing items like domain controllers, SQL servers, and other pesky boxes.

I understand the converters from VMware have improved by leaps and bounds over what it was in version 3, but there’s still a reasonable amount of security in P2V’ing a box while it’s completely down with no running services whatsoever.

First issue I ran into while attempting to P2V an old SQL server was finding the ISO. It has almost been removed from the internet as a whole. So I've uploaded it here: Cold Clone 3.0.3 Link

Second issue, boot time. It took me literally 15 minutes from the point of “hitting a button to boot to disc” to the point of accepting the EULA.

Third issue, drivers. I was lucky enough for the internal NICs to be found but the add-on NICs were not. To put this into perspective, the server I was working on was an HP G5. I tested with an HP G6 and it did not find any NICs at all.

Fourth issue, disappearing network settings. Every time I went to the networking settings, they would be cleared. The settings held as long as I clicked apply and then ok, but the second you went back to the network config menu I had to renter all the info.

Fifth issue, vCenter integration. It’s not exactly shocking that it didn’t work with vCenter, but I was hopeful. The converter would go through and recognize everything out of vCenter, but then ran into a bunch of issues as soon as the clone actually starts. Such as:

Couldn’t find the Distributed vSwitch vNIC:
[managedVMCreator,2657] No network named "SYSMGMT" was found
[imageProcessingTaskStep,194] VmiImportTask::task{9} step "create VM" destroyed
[vmiImportTask,439] Error during creation of target VM
[imageProcessingTaskStep,194] VmiImportTask::task{9} step "create and clone to VM" destroyed
[imageProcessingTaskStep,194] VmiImportTask::task{9} step "Clone VM" destroyed
[imageProcessingTaskImpl,552] VmiImportTask::task{9}: Image processing task has failed with MethodFault::Exception: vim.fault.NotFound
[imageProcessingTaskImpl,154] VmiImportTask::task{9}: SetState to error

Couldn’t find the FC attached storage on a host:
[NFC ERROR] NfcNewAuthdConnectionEx: Failed to connect to peer. Error: Host address lookup for server esx01 failed: The requested name is valid, but no data of the requested type was found
NBD_ClientOpen: Couldn't connect to esx01:902 Host address lookup for server esx01 failed: The requested name is valid, but no data of the requested type was found
DISKLIB-DSCPTR: : "vpxa-nfc://[VMFS_015] VM01/VM01.vmdk@esx01:902!52 f1 8e a1 39 1c c1 f8-9c d4 05 71 1a 4f ae c6" : Failed to open NBD extent.
DISKLIB-LINK : "vpxa-nfc://[VMFS_015] VM01/VM01.vmdk@esx01:902!52 f1 8e a1 39 1c c1 f8-9c d4 05 71 1a 4f ae c6" : failed to open (NBD_ERR_NETWORK_CONNECT).
DISKLIB-CHAIN : "vpxa-nfc://[VMFS_015] VM01/VM01.vmdk@esx01:902!52 f1 8e a1 39 1c c1 f8-9c d4 05 71 1a 4f ae c6" : failed to open (NBD_ERR_NETWORK_CONNECT).
DISKLIB-LIB : Failed to open 'vpxa-nfc://[VMFS_015] VM01/VM01.vmdk@esx01:902!52 f1 8e a1 39 1c c1 f8-9c d4 05 71 1a 4f ae c6' with flags 0x2 (NBD_ERR_NETWORK_CONNECT).
[diskHandleWrapper,87] DiskLib_Open failed on vpxa-nfc://[VMFS_015] VM01/VM01.vmdk@esx01:902!52 f1 8e a1 39 1c c1 f8-9c d4 05 71 1a 4f ae c6 with error NBD_ERR_NETWORK_CONNECT.
[imageProcessingTaskImpl,552] BlockLevelCloning::task{21}: Image processing task has failed with MethodFault::Exception: sysimage.fault.DiskLibConnectionFault
[imageProcessingTaskImpl,154] BlockLevelCloning::task{21}: SetState to error

How I actually got it to work was by sending the P2V directly to a specific ESXi 5 host. Once I did that, I could P2V the system to the locally attached storage or the FC attached storage. I attached the vNIC to a standard vSwitch.

5Sep/120

VMworld 2012: VMware vExpert Hour – Tuesday

VMworld 2012 - VMware vExpert Hour - Tuesday - Discussing the 2nd Keynote

Host: Mike Letschin - TheSolutionsArchitect.net - @mletschin
vExpert #1: Antone Heyward - TheHyperadvisor.com - @thehyperadvisor
vExpert #2: Kyle Ruddy (me) - ThatCouldBeAProblem.com - @RuddyVCP
vExpert #3: Chuck Mills - vChuck.com - @vChuckmills

Note: The video really starts around the 6:30 mark

Watch live streaming video from vmwarecommunitytv at livestream.com

Check out the rest of the Community Tech Talks including the Unsupported sessions, vBrownBag talks, vExpert talks, and so forth: VMware Community TV

3Sep/120

VMworld 2012 In Review

After a nice long week away, it's nice to finally be back home from VMworld...

Let's cover some of the good stuff:

      Overall friendliness of people. Truly amazing how available people were and how they didn't mind talking anywhere from a couple minutes to all night.
      My top three sessions: Definitely grab these sessions once they're available on the VMworld site.
      NET2207 - vDS Deep Dive with Jason Nash from Varrow - Very in depth, Jason had a lot of tips and demos which even features new functionality for vSphere 5.1
      VSP1504 - Ask the Expert vBloggers with Chad Sakac, Scott Lowe, and Rick Scherer from EMC, also with Duncan Epping and Frank Denneman from VMware - Interactive sessions are always great, these guys sat up on stage and answered question after question, basically being turned into a makeshift helpdesk, and was quite entertaining
      VSP1353 - vCenter Deep Dive with Justin King, Ameet Jani and Deep Bhattacharjee all from VMware - this session had a lot of information to cover, especially with all the new features in vCenter 5.1 and the speakers covered it all with great depth and also featured a Q&A portion at the end with several good questions being brought up from the audience
      Hang Space with the vBrownBag and theCUBE spaces were terrific. It was amazing how many speakers, bloggers, and other on-demand people were hanging out freely in the Hang Space. The Community Tech Talks from the vBrownBag guys were terrific and as soon as they're posted I'll provide links. The Hang Space also featured a charging center which worked as a coat check for your portable devices, this was brilliant.
      Certificate Lounge and the Fast Track VCAP:DCA samples. This was a terrific idea and a bit of a hidden gem in the Marriott Marquis. I must have spent all of Thursday afternoon doing the samples. Some great information and awesome discussions with folks coming in from cert tests or even VCDX Defenses
      Parties. There were some terrific parties this year. My favorites were definitely the VMunderground, HP Customer Appreciation, EMC Customer Appreciation, VMware CTO, and then the VMware VMworld parties. So thanks to all those people who sponsored the parties as well as attended to make it such a great time.

There wasn't a lot of negative stuff, but there were a couple items that stuck out:

      Speakers that phoned it in from home. If you went through the time to make the slides, at least learn the content so the session attendees aren't stuck going through the slides with you
      Hands on Labs. These have been terrific in years past, but they just couldn't keep them up and error free.
      Food. Breakfast was served in Moscone West with minimal tables and chairs. Lunch was prepackaged and normally soggy by the time it was served. Lunch was also served outside in the garden area. In the past lunch usually consisted of a self-serve buffet with table seating. The lack of proper seating wasn't conducive to networking and a lack of shade is never a good thing. I will say, the VMworld blanket they handed out was very nice.
      Solutions Exchange being open on Sunday and closing on Wednesday Having the Solutions Expo open on Sunday instead of being open on Thursday was a poor choice. Personally, I couldn't make it to the Solutions Exchange or the welcome reception due to other obligations and that took away a whole day from being able to visit with the vendors.

Some of the other notable items from VMworld this year:

Took some pictures as well:
Fixing the lack of Twitter handle on the Attendee Badges:
Fixed Badge

Kicking things off with the VMunderground Party:
VMunderground Line

VMworld Keynote Intro:
Keynote Intro

Stumping the "Expert vBloggers" (VSP1504 - Chad Sakac, Frank Denneman, Duncan Epping, Scott Lowe, a guest from the audience and Rick Scherer) on stage:
Stumping the Expert vBloggers

Paul Maritz speaking to the VMUG Leader group during the luncheon:

Stumping the Expert vBloggers

Pat Gelsinger taking the stage in front of the VMUG Leaders:
Stumping the Expert vBloggers

Pat Gelsinger, Paul Maritz, and Steve Herrod talking to our group:
CxO Group at the VMUG Leader Luncheon

William Lam setting up for his Unsupported vInception session in the vBrownBag area:
William Lam at vBrownBag

Jason Nash (Varrow) getting mobbed after his vDS session (NET2207):
vDS Session Aftermath

Steve Herrod at the CTO Party:
Herrod at the CTO Party

Geeky Tat found while in the Thirsty Bear (yes, it really is binary):
Binary Tat

Post VMworld Meat Coma thanks to the WPAVMUG folks:
Brazillian Buffett

Just to wrap things up, thanks to everyone I met out there and thanks to all the people that came up and introduced themselves. It was a terrific week, lots of information was learned, tons of connections were made and it was one of the best conferences I've been to yet... Thanks again!

18Aug/120

VMworld Prep!

So you got your VMworld ticket... you got your hotel reservation... you got your flight booked... You're ready to go right?

You're only cracking the surface!

  • Build up your calendar with sessions: VMworld Session Builder
    With the way the sessions are being done, I'm taking the position of registering for sessions I would go to at time slots I have available. I would much rather be registered for a session and miss it, than miss it due to it being full and unavailable. You'll also find that if you export your calendar, it will tell you what rooms your sessions are in so that you can plan accordingly.
  • Build up your calendar with after hours events: VMworld Gatherings
    Please note that many of these events are RSVP only, so definitely make sure to get registered and/or talk to your favorite company's reps to get you registered.
  • Make time for the Hands on Labs
    The Hands on Labs are always very well organized and provide great content, don't miss out on an opportunity to learn in an environment you're not responsible for!
  • Check out the Solution Expo
    Yes, the vendors are going to want to scan your badge. However, there are always some diamonds in the rough. Either some amazing technologies with capabilities I didn't realize existed, or some awesome prizes.
  • Stop by the Hangspace
    The Hangspace will be providing an area for the VMTN Community TechTalks and other interviews as well as blogger tables and space for general networking.
  • Stay in the know by following VMworld related hashtags on twitter: VMworld Twitter Community

A map of the layout around the Moscone Center has already been released:
VMworld Map

If you fill your schedule like I do, chances are you aren't seeing the hotel for any reason but to sleep for a couple hours. So let's talk about the accessories that can keep you on the go...

  • Don't worry about a backpack. You'll get one at registration.
    They're always quite nice and very durable. Not to mention that's one less thing to lug on the plane on the way there or find space for on the way back. A sneak peek has already been released:
    VMworld Backpack Sneak Peek
  • Bring some comfortable shoes
    You'll be walking or standing a majority of the day, you don't want to miss out on something because your feet hurt
  • Bring spare batteries or external chargers
    I cant stress this one enough. With 15,000+ attendees there aren't enough plugs for everyone. Don't miss out on a session or activity because you couldn't get to the VMworld app or your calendar. Personally, I picked up a New Trent iPulse IMP100P. Two USB outputs and 10,000 mAh in the size of an iPhone should be good enough to get me through a full day.
  • A tablet to take notes on.
    In years past I'd have my iPad and the Pen Ultimate App, but this year I'm going with the Nexus 7 and the Evernote App. I've outfitted the Nexus 7 with the Poetic Leather case and an amCase amPen stylus. So far has been quite a combo for during business meetings and various local user groups.
  • Business Cards
    VMworld is an amazing resource to gain knowledge and information and the networking opportunities are incredibly strong.

Some other tips to those that have never been:

  • Dress code is normally said to be Business Casual, however you'll see everything from high dollar business suits down to shorts and sandals. Be comfortable, but don't look like a mess.
  • Wireless is readily available throughout. It may not be the fastest, but it's available and free. In years past, the AT&T networks (my carrier) have been quite overloaded so use whatever has the best connection out.
  • Drinks (soda, water, juice) are normally readily available. Snacks are available at different times during the day as well, everything from granola down to candy bars.
  • Breakfast and lunch are provided, make sure you know how to get there. There are other options around the Moscone, but none are free and some aren't exactly quick.

Not going to VMworld?

Well, watch the Keynotes and get access to quite a bit of other information live via VMware NOW: http://bit.ly/VMwareNOW and keep up with action by way of the Community Tech Talks featuring myself and many other vExperts: http://www.vmworld.com/docs/DOC-6032. There is also a large list of bloggers that will be onsite and providing content as well: VMworld Blogger Coverage.

11Aug/120

VMworld Session Selection…

I must start off by saying there is an amazing amount of knowledge and a great selection of sessions at VMworld this year. I'm sure the labs will be pretty spectacular as well, so definitely leave some room to take advantage of those.

With that said, I've spent quite a bit of time trying to whittle down the sessions into something that resembles a schedule.

Subject Start Date Start Time End Date End Time
Keynote: Join Paul Maritz as he shares how VMware is helping customers and partners thrive in the Cloud era. And Steve Herrod will discuss and demo technology at the heart of the software defined datacenter. 8/27/2012 8:30 AM 8/27/2012 10:00 AM
INF-VSP1168 - Architecting a Cloud Infrastructure 8/27/2012 10:30 AM 8/27/2012 11:30 AM
INF-STO1198 - vSphere Storage Features & Enhancements 8/27/2012 12:30 PM 8/27/2012 1:30 PM
INF-VSP1504 - Ask the Expert vBloggers 8/27/2012 2:30 PM 8/27/2012 3:30 PM
INF-NET2207 - VMware vSphere Distributed Switch - Technical Deep Dive 8/27/2012 4:00 PM 8/27/2012 5:00 PM
INF-STO2980 - vSphere 5 Storage Best Practices 8/27/2012 5:00 PM 8/27/2012 6:00 PM
Keynote: Steve Herrod and VMware partners take the stage to demonstrate state of the art technology that is transforming IT and enabling the mobile workforce. 8/28/2012 8:30 AM 8/28/2012 10:00 AM
INF-VSP1475 - VMware vSphere 5 Design Discussions 8/28/2012 12:00 PM 8/28/2012 1:00 PM
INF-VSP1365 - Understanding VMware ESXi Security 8/28/2012 1:30 PM 8/28/2012 2:30 PM
INF-VSP1683 - VMware vSphere Cluster Resource Pools Best Practices 8/28/2012 3:00 PM 8/28/2012 4:00 PM
OPS-CIM1645 - Cross-Domain Performance and Capacity Management of VNX Storage Using EMC VNX Connector and VMware vCenter Operations Manager 8/29/2012 8:00 AM 8/29/2012 9:00 AM
INF-VSP1252 - What's New with vSphere Automation 8/29/2012 9:30 AM 8/29/2012 10:30 AM
OPS-CSM2975 - Orchestrating the Cloud: From API to Workflow 8/29/2012 11:00 AM 8/29/2012 12:00 PM
INF-NET2161 - VMware Networking 2012: Enabling the Software Defined Network 8/29/2012 1:00 PM 8/29/2012 2:00 PM
OPS-CIM1564 - Troubleshooting Using vCenter Operations Manager 8/29/2012 2:00 PM 8/29/2012 3:00 PM
OPS-CIM1274 - Building Cloud Services using VMware vCenter Orchestrator 8/29/2012 3:30 PM 8/29/2012 4:30 PM
General Session: Watch three extraordinary researchers: Kevin Slavin, Co-Founder, Area/Code; Dr. Dennis Hong, Director, RoMeLa (Robotics and Mechanisms Lab), Georgia Tech University; and Chris Urmson, Lead, Self-Driving Car Project, Google - who are working at the frontier of machine technology. 8/30/2012 9:00 AM 8/30/2012 10:00 AM
INF-BCO2982 - Stretched Clusters and VMware vCenter Site Recovery Manager: How and When to Choose One, the Other, or Both 8/30/2012 11:00 AM 8/30/2012 12:00 PM
INF-VSP2749 - Get Up and Running Quickly with the Latest vSphere Web Client 8/30/2012 12:00 PM 8/30/2012 1:00 PM
OPS-CIM1718 - Configuration Management for Your Cloud using vCenter Operations Suite 8/30/2012 2:00 PM 8/30/2012 3:00 PM

I still have quite the extensive list of backup sessions, you know... in case a room fills up or I end up running late... Which is something that should be planned for.

Subject Start Date Start Time End Date End Time
GD27 - PowerCLI with Alan Renouf 8/27/2012 10:30 AM 8/27/2012 11:30 AM
INF-STO2428 - Flash as a Cache – Rethinking Virtualization 8/27/2012 12:30 PM 8/27/2012 1:30 PM
GD32 - Capacity Management with Monica Sharma 8/27/2012 1:00 PM 8/27/2012 2:00 PM
INF-STO1807 - Just Because You Could, Doesn't Mean You Should - Lessons Learned in Storage Best Practices 8/27/2012 2:00 PM 8/27/2012 3:00 PM
OPS-CIM1645 - Cross-Domain Performance and Capacity Management of VNX Storage Using EMC VNX Connector and VMware vCenter Operations Manager 8/27/2012 2:30 PM 8/27/2012 3:30 PM
INF-STO2239 - Deciphering the Mystical World of Storage Performance 8/27/2012 3:30 PM 8/27/2012 4:30 PM
APP-BCA1516 - Virtualizing SQL 2012 : Doing It Right 8/27/2012 4:00 PM 8/27/2012 5:00 PM
INF-VSP1856 - Become a Rock Star with PowerCLI and vCenter Orchestrator 8/27/2012 4:00 PM 8/27/2012 5:00 PM
GD22 - Resource Management (DRS/SDRS) with Frank Denneman 8/27/2012 4:00 PM 8/27/2012 5:00 PM
OPS-CIM1274 - Building Cloud Services using VMware vCenter Orchestrator 8/27/2012 5:30 PM 8/27/2012 6:30 PM
INF-VSP2825 - DRS: Advanced Concepts, Best Practices and Future Directions 8/27/2012 5:30 PM 8/27/2012 6:30 PM
INF-VSP1353 - vCenter: A Technical Deep Dive 8/28/2012 10:30 AM 8/28/2012 11:30 AM
INF-BCO1159 - Architecting and Operating a VMware vSphere Metro Storage Cluster 8/28/2012 11:00 AM 8/28/2012 12:00 PM
INF-VSP1295 - vCAT 3.0 – Architecture to Implementation in 5 Easy Steps 8/28/2012 12:00 PM 8/28/2012 1:00 PM
INF-SEC2813 - Beyond the Hypervisor: Three Key Areas to Consider When Securing Your Cloud Infrastructure Platform 8/28/2012 12:30 PM 8/28/2012 1:30 PM
INF-SEC1282 - Automating Security and Compliance with Disaster Recovery 8/28/2012 3:30 PM 8/28/2012 4:30 PM
GD10 - vSphere Storage Best Practices with Cormac Hogan 8/28/2012 3:30 PM 8/28/2012 4:30 PM
INF-BCO2807 - vSphere HA and Datastore Access Outages – Current-Capabilities Deep-Dive and Tech Preview 8/28/2012 5:00 PM 8/28/2012 6:00 PM
OPS-CIM1645 - Cross-Domain Performance and Capacity Management of VNX Storage Using EMC VNX Connector and VMware vCenter Operations Manager 8/29/2012 8:00 AM 8/29/2012 9:00 AM
INF-VSP1353 - vCenter: A Technical Deep Dive 8/29/2012 11:00 AM 8/29/2012 12:00 PM
ITT3241 - Operating a More Reliable Cloud Through Proactive Incident and Problem Management 8/29/2012 11:30 AM 8/29/2012 12:30 PM
OPS-CIM1309 - vCAT 3.0: Operating a VMware vCloud 8/29/2012 12:30 PM 8/29/2012 1:30 PM
INF-BCO2807 - vSphere HA and Datastore Access Outages – Current-Capabilities Deep-Dive and Tech Preview 8/29/2012 1:00 PM 8/29/2012 2:00 PM
TEX2815 - vCenter Orchestrator Plug-in for Self-Scaling Data Center 8/29/2012 2:00 PM 8/29/2012 3:00 PM
INF-BCO1159 - Architecting and Operating a VMware vSphere Metro Storage Cluster 8/29/2012 2:30 PM 8/29/2012 3:30 PM
INF-STO2564 - Supercharged SAN: Fine-tune Your VAAI Enabled vSphere SAN with This Collection of Configuration and Performance Best Practices 8/29/2012 2:30 PM 8/29/2012 3:30 PM
INF-SEC2850 - The Four Must-Haves for a Secure Cloud Infrastructure 8/29/2012 2:30 PM 8/29/2012 3:30 PM
OPS-CIM1274 - Building Cloud Services using VMware vCenter Orchestrator 8/29/2012 3:30 PM 8/29/2012 4:30 PM
OPS-CSM1209 - Securing the Virtual Environment: How to Defend the Enterprise 8/29/2012 3:30 PM 8/29/2012 4:30 PM
INF-SEC1840 - VMware vSphere Hardening to Achieve Regulatory Compliance: Better Faster Stronger 8/30/2012 10:30 AM 8/30/2012 11:30 AM
INF-BCO1883 - Deploying an Active/Active Datacenter with SRM 5 8/30/2012 11:00 AM 8/30/2012 12:00 PM
APP-CAP2956 - Inside the Hadoop Machine 8/30/2012 12:00 PM 8/30/2012 1:00 PM
APP-BCA1516 - Virtualizing SQL 2012 : Doing It Right 8/30/2012 12:00 PM 8/30/2012 1:00 PM
OPS-CSM1209 - Securing the Virtual Environment: How to Defend the Enterprise 8/30/2012 12:30 PM 8/30/2012 1:30 PM
Tagged as: , No Comments
24May/1212

Playing with ESXi and Synology in the home lab…

So I finally bit the bullet and bought a NAS for my home lab. Ended up with a Synology DS411 due to some good timing from a NewEgg sale.

I went with Synology for two reasons: 1. I could supply my own disks (I don't want the "Green" drives that came with most other NAS units) 2. I always hear REALLY good things about them, they're always highly recommended.

I happened to already have a set of 4 Hitachi Deskstars sitting around so I tossed them in and started running some tests with IOMeter (Run all tests). I did a test on a RAID5 grouping of all 4 disks which were presented to the ESXi host as an iSCSI target and then I did the same test with a RAID 5 grouping of all 4 disks which were then presented to the ESXi host via NFS. I was amazed at the results.

From a personal standpoint, I fully expected iSCSI to blow away NFS. Wow, was I wrong. iSCSI pulled a whopping 270 IOPS from the IOMeter test. Then I ran the exact same test only via an NFS share... NFS blew iSCSI out of the water with a 1,070 IOPS. My jaw is still on the floor from that result. Here's some of the rest of the results:
NFS vs iSCSI

I'm also seeing some excellent regular performance as well. 45MBps worth of write during an SvMotion.
Cooler

So to say the least, I'm quite impressed at what this little guy can do with 4 SATA 7200RPM drives and a single NIC.

One other tidbit to add in an unrelated fashion is how much cooler the new Hitachi Deskstar runs compared to the older ones. A full 12F cooler. That was another surprise...
Cooler

11Apr/121

vDR – causing problems…

For those new to vSphere 5's GUI, there's a new column that's been added to the Virtual Machine view by the name of "Needs Consolidation".
Needs Consolidation

This option was put in due to the occasional problem when Snapshots did not delete properly and would leave the delta files remaining in the VM's folder while the Snapshot Manager would show no snapshots existing.

With this option added to the columns, you should also take note of the option within the Snapshot options for each VM which will now allow a user to select the "Consolidate" function
Consolidate Snapshot

As noticed with the first screenshot, we had a couple systems which were requiring some consolidation to them. So another admin went through and hit the consolidated button and got hit with a "Unable to access file since it is locked" error. Normally, you can go through and figure out which file is being locked with some command line work or by rebooting the host (via: VMware KB: 10051) however our VM is running so there's something else going on.
Locked File

I still decided to dive into the CLI and check it out. I was stunned...
Deltas!

18 deltas... 18! Regardless of the vmsn file in there, there was no record of there being any snapshots.

In this case, that system probably hasn't even been rebooted 18 times much less been snapshot that many times... Except, vDR (VMware Data Recovery) is setup on it to do daily snaps. So I checked the vDR appliance settings and I found 8 disks too many attached.
Locked File

After removing all of those extra hard disks, the consolidations would succeed. Note, it took a while, but they did succeed.
Locked File

Just another reminder of while vDR is a great tool to have on hand, it should definitely not be the one and only method of backup

3Apr/121

SRM: vSphere Replicated VMs stuck in a “Sync” status

Here recently I've noticed that there is an occasional time where the VMs I have replicating using the vSphere Replication system are stuck in a "Sync" status for an overly long time.

After pulling the logs, I was able to figure out what was happening... Timeouts, lots of them. The log file vmware-dr.log pulled from the remote site was full of lines like the following: (local is the SRM server, peer is the vCenter server)

2012-04-02T07:35:04.077-04:00 [02784 verbose 'Default'] Timed out reading between HTTP requests. : Read timeout after approximately 50000ms. Closing stream TCPStreamWin32(socket=TCP(fd=2596) local=10.xx.xx.xxx:9085, peer=10.xx.xx.xxx:55039)

2012-04-02T11:54:34.159-04:00 [02744 verbose 'Licensing'] Asset in sync.
2012-04-02T11:58:12.527-04:00 [02868 info 'LocalVC' opID=ac2d1cb] [PCM] Received NULL results from PropertyCollector::WaitForUpdatesEx due to timeout of 900 seconds
2012-04-02T11:58:12.723-04:00 [02860 info 'LocalVC' opID=596971f7] [PCM] Received NULL results from PropertyCollector::WaitForUpdatesEx due to timeout of 900 seconds

After a brief discussion with our network engineers, it was believed that there was no problem with the connection between the local and remote site. So I took a "when in doubt, reboot" approach. I restarted the SRM service on the remote SRM server. No luck. After that, I did a "Restart Guest" on the VRS system at the remote site. After about 5 minutes, the systems started to connect and replicate again.

I've noticed it a lot, and I've heard from other people whom also manage their own SRM deployments that a reboot is a pretty good first step in troubleshooting. So keep that in mind as issues arise and troubleshooting is required.

30Mar/1210

Standalone ESXi 5 Host Upgrade

Have an ESXi host which is a standalone box? No VMware Update Manager? No vMA?

Well, they still require patches. Luckily enough, you can still use the stripped down version of the console which is included in ESXi to update it.

Start by heading out to the VMware Patches portal http://www.vmware.com/patchmgr/download.portal and download the neccessary patches for the server that needs patched.

Upload the patch zip file to a datastore that the server can talk to via either SCP or the datastore browser
Upload the patch

Next, make sure the SSH service has been started.

To do this while in the vSphere Client, click on the desired host, and click on the "Configuration" tab followed by the "Security Profile" link in the "Software" box, then click on "Properties" in the top right side.
Get to SSH Service

Highlight "SSH" and then click "Options", after the SSH Options screen pops up, click on "Start", then click "OK" twice to get back to the Configuration tab.
Start SSH Service
Start SSH Service

After getting connected to the ESXi host, run the command: esxcli software vib install -d *full path to uploaded zip*
Example: esxcli software vib install -d /vmfs/volumes/VMO-01 Datastore/Temp/update-from-esxi5.0-5.0_update01.zip

There should be a message showing that the update was completed successfully and that the system needs to be rebooted.
Upload the patch

If ready to reboot, type in "reboot" and the system will reboot. Just remember to check to make sure that the SSH service has been stopped when it boots back up.

One error that I ran into, if you don't give the full path to the zip file containing the update, the patching will fail with a "MetadataDownloadError" reading:
Could not download from depot at zip:/var/log/vmware/*update name*.zip?index.xml, skipping (('zip:/var/log/vmware/*update name*.zip?index.xml', '', "Error extracting index.xml from :/var/log/vmware/*update name*.zip: [Errno 2] No such file or directory: '/var/log/vmware/*update name*.zip?index.xml'"))
url = zip:/var/log/vmware/*update name*.zip?index.xml
Please refer to the log file for more details.
Error Message

Once I put in the full path, it worked just fine.