More on Locked files in VMware vCenter and ESX

Posted on 25th February, 2013 by BigChaps

Back in February, I posted a story on Fixing Invalid VM’s in VMware vCenter and ESX and it became one of my two most popular stories and still remains so even after 12 months.

Now, I can bring you some more on this topic, thanks to Bmonroe on Experts-Exchange for posting his answers. It was this article that pulled the thread that untied the knot.

After doing an upgrade of vCenter from 4.1 to 5.1, we found that we had a phantom Server. In fact, this server had become Schrodingers Server. It was not visible from either the vCenter Server nor was it visible using the client software to connect directly to the host. However, the Server was alive providing all Services (hence Shrodinger – The Server was both alive and dead)

This was an untenable situation. It couldn’t be managed or controlled yet it was interfering with our other production services causing all sorts of problems. Any attempt to delete the files direct from the datastore resulted in a “device or resource busy” error

Here is how I fixed it…

1.       Logon to the ESX host where the VM was last known to be running.

2.      type cmd:  vmkfstools -D /vmfs/volumes/path/to/file to dump information on the file into /var/log/vmkernel

3.      type cmd:  less /var/log/vmkernel and scroll to the bottom, you will see output like below:

Feb 25 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)FS3: 130:

Feb 2515:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)Lock [type 10c00001 offset 30439424 v 21, hb offset 4154368

Feb 25 15:49:17 vm22 vmkernel: gen 66493, mode 1, owner 46c60a7c-94813bcf-4273-0017a44c7727 mtime 8781867] 

Feb 25 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)Addr <4, 588, 7>, gen 20, links 1, type reg, flags 0x0, uid 0, gid 0, mode 644

Feb 25 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)len 23973, nb 1 tbz 0, zla 2, bs 65536

Feb 25 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)FS3: 132:

4.      The owner of the lock is on the third line, the last part is all you need, in this case 0017a44c7727 

5.      Type cmd: esxcfg-info | grep -i ‘system uuid’ | awk -F ‘-‘ ‘{print $NF}’ This will display the system uuid of the ESX server. You need to run the esxcfg-info command on each ESX server in the cluster to discover the owner. Of course, for me, it was on Host number 5 of a 6 Host Cluster.

6.      When you find the ESX server that matches the uuid owner, logon to that ESX server and run the command: ps -elf|grep vmname where vmname is the problem vm. Example output below:

4 S root 13254 1 0 65 -10 – 435 schedu Feb 25 ? 00:00:02 /usr/lib/vmware/bin/vmkload_app /usr/lib/vmware/bin/vmware-vmx -ssched.group=host/user/pool2 -@ pipe=/tmp/vmhsdaemon-0/vmxf7fb85ef5d8b3522;vm=f7fb85ef5d8b3522 /vmfs/volumes/470e25b6-37016b37-a2b3-001b78bedd4c/iu-lsps-vstest/iu-lsps-vstest.vmx0

7.      Since there is a process running, pid 13254 in the example, you need to kill it by following steps 5-12 on stopping a VM above

8.      Once the kill is complete the files should be released.

 

 

 

 

 

 

 

Posted in Work | Tagged , | Leave a comment

Updating your WSUS for Windows 8 & Windows Server 2012

Posted on 17th September, 2012 by BigChaps

So Microsoft have released the Update for WSUS 3.0 SP2 (KB2734608) that allows it to be able to recognise Windows 8 and Windows Server 2012 Operating Systems and download the appropriate updates.

Here are the steps I completed to ensure I am ready for the new Windows 8 and Windows Server 2012 operating Systems in my environment. I have a simple Setup with one WSUS Server, my maintenance machine with the snap-in installed and (so far) just the one Windows 8 client. Your experience may be different depending on your own environment.

Downloads

Firstly, download the new update from Microsoft. I would also recommend reading the relevant KB Article: An Update for Windows Server Update Services 3.0 Service Pack 2.0 is available (KB2734608)

Installation

WSUS Server

  1. Install the Update on the server
  2. open Command Prompt and run the following commands
    1. IISReset
    2. net stop WSUSService
    3. net start WSUSService
  3. Open the Console, Update the Product List and begin a Synchronisation

Any Maintenance Consoles

If you use the WSUS MMC Snap-in on any maintenance machines, you will need to install the update on any of those machines. If you do not,all Windows 8 machines will be listed incorrectly. In my case, my Windows 8 x64 client was listed as Windows XP x64

Clients

If you have any clients that may have already been connected to the WSUS server prior to the update, you will need to update these clients as well. Follow these steps to do so:

  1. Open an elevated Command Prompt (Win + X menu)
  2. type the following commands and press enter after each one
    1. net stop wuauserv
    2. rd /s %windir%\softwaredistribution\
    3. y
    4. net start wuauserv
    5. wuauclt /update
  3. This will get the client to update and should register as Windows 8 or Windows Server 2012

image

Posted in Computers, Work | Tagged , , | 8 Comments

Windows 8 – The New Schrodinger’s Cat?

Posted on 29th August, 2012 by BigChaps

There have been many questions about whether or not to upgrade to Window 8 or to keep working on Windows 7. I had read so many articles about how Windows 8 would be great and others stating it would be catastrophic, the end of Microsoft. I was beginning to think that Windows 8 was going to be the next Schrodinger’s Cat. Realistically though, I was never going to not install it. I mean, this is Windows 8 after all!

Finally, the time was right, my Dell Latitude E6420 Laptop was in need of some attention and I had a freshly downloaded Windows 8 Enterprise and Office 2013 preview from our TechNet Subscription and I had a spare machine to use for work in the meantime. What better time to upgrade than a Friday Arvo?

Preparation

First of all, I did a complete backup of my laptop to a network location, you know, just in case. I haven’t actually gone back to it as yet and chances are that I might never need to but as we all know, backup, backups and backups. All my user data was already on a Data Partition so I didn’t have to worry about that. Because of this, I was able to do a fresh install rather than an upgrade. I had created a USB Boot disk with Windows 8 Enterprise ISO using the Windows 7 Boot Tool. The last step was to copy my Change Log from my original PC to my Virtual machine so I am ready for the rebuild. Game On!

Install

I’m not sure what I was expecting but what I got stunned me. I was logging into my laptop in 13 mins. The Office 2013 (and Visio 2013) preview took another few minutes and I was done. Wow. Not only that, it had picked up all but two devices in my laptop. The devices missed were my internal 3G Broadband card and the Fingerprint Sensor (which involved arcane driver rituals at the best of times). I created my Change log and set to work.

Configuration

First up, setting up my Start Screen. I still only spend about 10-20% of my time in the Start Screen but I wanted set up the way I liked. I created the separate groups based on Action groups. I did read this advice on a blog whose source I cannot remember (if this was your blog article, Thanks so much, it helped a lot). The next step was to pin a bunch of my most used Applications to the Taskbar. So far, this is just the RDP client and the Office Programs. I haven’t really been into the Windows Store to get a lot of apps. I did get Minesweeper, Solitaire and Mahjongg. The lure of Xbox Achievements was too much. I did have to rearrange my folders so they would look a lot less messy in the Pictures App. It has a much neater hierarchy now and ties in better with my Facebook Pics, SkyDrive Pics and Pics from my Home Machine (via SkyDrive App)


Usage

I’ve been using it for a few days now and I pretty much have my patterns of work down pat. I still have the occasional moment where I go around in circles a bit but these are happening less and less. I still find all sorts of great new features here and there when I am doing stuff or when aI am trying to do something else. I did come across Scott Hanselman’s post on Windows 8 Productivity. In that article he shares a number of tips and keyboard shortcuts as well (Win+X is the most important one). A great bunch of tips there delivered in a good article that cuts through the complaints.

Another interesting article that I came across that I recommend people to read is an article by Kyle Wagner over at Gizmodo. His article – You’re Being Lied To: Windows 8 Isn’t Bad for Gaming discusses some of the more recent statements by those in the gaming industry that are calling Windows 8 a catastrophe. His speculation is that these people tend to make a lot of revenue on the distribution of games and see competition in the Windows Store affecting that revenue.

I spend most of my time in the desktop (Win+D) and using Desktop Apps. I have snapped the Music App to the side to keep my tunes pumping while I try and get in some uninterrupted work. Having spent a lot of time using keyboard shortcuts before, I probably have found this transition to be a little less jarring than a lot of more mouse-centric people have. The swapping between the desktop and the Modern UI Interface has been easier than I thought it would be. The best way is with an Alt-Tab to swap between Apps. If you want to swap between Modern UI Apps then Win+Tab is your friend.

Conclusion

There are a lot of articles saying that Windows 8 will be the end of PC’s, Microsoft and life as we know it. So many, that it reminds me of all the articles that said much the same for Windows 95, Windows XP, and Windows Vista and so on. Granted, there is an adjustment period to adapt to a new operating system. Some people may find this harder than others. In the end, only you know how you use your computer and you can adapt Windows 8 AND your processes to be more effective than ever. There are enough gains in Power Management, speed and functionality to be worth your time to upgrade

Posted in Computers | Tagged | Leave a comment

Fixing Invalid VM’s in VMWare vCenter & ESX

Posted on 16th February, 2012 by BigChaps

 

We’ve had a Virtual Machine showing up in vCenter Server as Invalid (and greyed out) for a few days now. Ordinarily, I would have had a go at fixing it straight away but is is an archive server that is not actually in use so it has been down the priority list. Today is the day that I reached that item on the To-Do list so it was time to get onto it.

Why Invalid?

First I had to determine why it was showing as invalid. Alas, the history of what had happened left with the staff member who had recently left so I was going to be floundering the dark somewhat.

Firstly, I tried the initial step of removing the VM from the Inventory and re-adding it.

  1. Open vCenter Console
  2. Right-Click on VM, select remove from Inventory
  3. Browse to the DataStore and select the <VMDIR> folder
  4. Right-Click on the <servername>.vmx and select add to Inventory
  5. Complete the wizard toplace the VM in the correct group and Host.

Alas, from me that didn’t work. The Machine was still greyed out and marked as invalid.

To the Command-Line!

The next step was to check for file locks on any of the VM files. Theree are often in place on these files:

  • <servername>.vswp
  • <diskname>-flat.vmdk
  • <servername>.vmx
  • <servername>.vmxf
  • vmware.log

So to check the files for my VM I did the following steps:

  1. logged onto the ESX host via SSH
  2. ran command: vmware-cmd –l and noted the full path of the VM Files – /vmfs/volumes/<UUID>/<VMDIR>/<SERVERNAME>.vmx
  3. ran command: cd /vmfs/volumes/<UUID>/<VMDIR>/
  4. To check what files were locked, i ran the command: touch *
    1. any files that provided a result of  device or resource busy was a locked file.

Who Dares Lock my VM?

So now I know that the VM was invalid because of some locked files and now it was time to find out who had locked the files and how to unlock them. Back to the command-line! To look for the culprit, I did the following:

  1. ran command:   vmkfstools -D /vmfs/volumes/<UUID>/<VMDIR>/<LOCKEDFILE>.xxx
  2. Alas, as my Hosts are still ESX 4.0, I would then have to look at the System Logs instead of on-screen
  3. tail /var/log/vmkernel
  4. here is an example of the log file from a VMWare article

    5. Hostname vmkernel: 17:00:38:46.977 cpu1:1033)Lock [type 10c00001 offset 13058048 v 20, hb offset 3499520
    Hostname vmkernel: gen 532, mode 1, owner 45feb537-9c52009b-e812- 00137266e200 mtime 1174669462]
    Hostname vmkernel: 17:00:38:46.977 cpu1:1033)Addr <4, 136, 2>, gen 19, links 1, type reg, flags 0x0, uid 0, gid 0, mode 600
    Hostname vmkernel: 17:00:38:46.977 cpu1:1033)len 297795584, nb 142 tbz 0, zla 1, bs 2097152
    Hostname vmkernel: 17:00:38:46.977 cpu1:1033)FS3: 132: <END supp167-w2k3-VC-a3112729.vswp>

  5. the bolded number in there is the MAC address of the offending machine that has a lock on those files. In my case, it was another of my ESX hosts

Unlocking the File

There are a number of situations that can cause another ESX Host to lock the files of a VM. I suspected that a transfer between Hosts had not been successful. I chose the easiest option first – Check to see if the offending ESX host (identified by MAC Address) was still trying to manage that VM.

  1. log onto newly-identified host via SSH
  2. ran command: vmware-cmd –l
  3. VM is listed there. ( surprise surprise)
  4. To remove the VM from the old host, run command: vmware-cmd –s unregister /vmfs/volumes/<UUID>/VMDIR>/<SERVERNAME>.vmx

Once I had don that, I repeated the first step of the day by manually removing the VM from Inventory and re-adding it again via browsing to the DataStore.

Happy Days are Here Again!

I’m sure that this was some of the ugliest fault-finding any VMWare Administrator has ever seen and any VCP’s reading this will be curled up in the corner, screaming “find a happy place”. However, for a guy who hasn’t delved below the GUI too often and shudders at the thought of command-lines and case-sensitive syntax, it got the job done and the server working again,

and that, is what I needed to get done…

Posted in Computers, Work | Tagged | 3 Comments

WHS 2011 & Covecube StableBit Extender

Posted on 19th April, 2011 by BigChaps

In this next part of my WHS 2011 testing, i have used Stablebit Extender v1.0.0.718 Beta by Covecube. This software aims to recreate the functionalities of DE that was removed in WHS2011. There is an article written on their blog that gives a Technical Overview of what it does and how it works.

In this article, I will describe how I installed it and what I found

Installation

  1. Download and run the installation package
  2. The application is installed and listed in the Add-in section of the dashboard.

COnsole - Add-ins

Setup

Once installed, an extra icon is added in the Toolbar and all the options are available

  1. Add existing drives into a pool
  2. follow wizard steps and add the required Hard Drives into the new Pool
  3. Once complete, the pool will be displayed, listing the Hard Drives which make up that pool.

 

Console - Hard Drive Pooling

  1. to add a new folder, switch to the Shared Server Folders tab and click on Add Folder
  2. Enter folder name
  3. Choose whether it goes on a pooled or a non-pooled drive
  4. Specify if it is duplicated
  5. When completed, the Pooled Shares are shown separately. The Free Space will show the space available across the whole pool of disks

Console - Shared Server Folders

Testing

Once configured, I quickly created a Windows 7 Client to use as a test client. I browsed through the Network to the WHS2011 Server

browse to WHS2011

  1. Double-click on the WHS2011 icon under Other Devices
  2. This opens a browser with a big Download button
  3. Follow the wizard prompts

Add Client

 

The client is now connected to WHS2011

Testing

I seeded the Pooled Folders with some files and a movie file for testing

  1. Started the video from Pooled Storage
  2. Removed Drive from Server VM Settings
  3. The movie kept going without interruption
  4. When I added the Drive back into the pool, the movie did stop but as soon as I stopped and started it again, it was fine

Pooled Drive Testing

 

So it does work in its most basic function… protecting data from HDD Failure

Issues

There were a couple of points that I found that were a drawback. Be aware that this is still early beta software

  1. Cannot move default folders into the new pool – The Move Folder wizard doesn’t recognise the new Pool at all.
  2. The Pool not seemingly integrated into system at all. All actions are now done from the StableBit DrivePool Icon in the Ribbon
  3. Pools created are not indexed by default – can’t add to Windows 7 Client Libraries by default
  4. Separate process to change User permissions – Changing User permissions is done on Stablebit Drivepool Tab

 

clip_image008

Conclusion

While this goes a long way to added pooled storage across non-identical drives, I feel there is still someway to go. For me, there is not quite enough integration yet to make it worthy of production just yet.  I definitely look forward to future versions to see the progression of later versions.

Posted in Computers, Home | Tagged , , | 1 Comment

WHS 2011 – Upgrade or Not?

Posted on 12th April, 2011 by BigChaps

windows-home-server1

This is a tough question and one I have wrestled with ever since THAT day, that fateful day that Microsoft announced that DE was no longer going to be a part of Windows Home Server 2011 (or Vail, as it was known as at the time).

My existing WHS is an old Dell PowerEdge 715n with 4 Drives and is working wonderfully albeit with only a small drive pool. I need to update so I can add some larger capacity drives to expand to cover all our media from our Windows 7 Media Center. I do need to keep my WAF up high.

This brings me to the big decision – do I stay with WHS v1 or replace with WHS 2011?

I have read the articles from Sean Daniel, (Senior Program Manager for the Home and Small Business Server Team] at the Windows Home Server Blog and he did bring up some very valid points, enough to make me really want to use WHS 2011. However, it was at the last few paragraphs that I began to wonder… He listed what was lost with the removal of DE

  • In the event of a hard drive failure, all my data isn’t still available until I put in a new hard drive and restore that drive back to its original state, if I didn’t configure RAID.
  • I can’t easily extend my storage when I have a disk that’s running out of space.
  • I can’t grow my share sizes past the size of all the hard drives in the system
  • My server storage responds faster because I’m not using DEMigrate.exe all the time

The first three of those points are all but one of the reasons why I bought WHS in the first place ( the other being Client Backup). He also went on to talk about Data hoarding and whether or not we are right to keep such massive collections of data. Yes, I’ve got a lot of data, a LOT of data. Over a terabyte of movies alone plus loads of music, photos, videos, software etc.  However, I want to change my technology to support me, not change me to support my technology.

So I will be testing is the Third-Party Add-ins that are replicating the Drive Extender technology. I’ll be looking at how they work with inbuilt folders, PC Backup and how it deals with HDD crashes, replacement and adding non-identical drives. Thankfully I have some Virtual Hardware for testing this, our VMWare ESXi Test box here at work will be providing the platform for all this.

to be continued…

Posted in Computers, Entertainment, Home | Tagged , | Leave a comment

Lessons from the Cloud – How I Learned to Love the CSV

Posted on 15th November, 2010 by BigChaps

This last couple of weeks, I have been assigned what seems to be a relatively simple imageproject – get our Member’s Database into Salesforce.com.

Turns out we have had a Salesforce Professional Subscription for the better part of a year but my Manager had not yet had the time to use it as he would like. Now it was my opportunity to move our data into the cloud.

Here are a few Lesson’s I learned that I thought I’d share.

1. Training, Training and more Training

Salesforce.com actually has quite a lot of training modules and help available for every subscriber. Mane sure you use it. the commentary is dry (oh so very dry and monotonous) but well worth it. I suggest turning down the sound as it mostly just reads out what is displayed on screen.

2. NEVER assume the Source Data is complete

Turns out I took this for granted. After all, our Service Fees are based on this data and it is the central record on which all data in other systems is supposedly based. If I had any clue how wrong I was, I could have saved myself a week of issues. Here are the most important task I found:

  • Make sure your unique index fields are actually unique
  • There WILL be duplicates, find them and fix them.
  • Take the time to choose the field formats for numbers (i.e. phone numbers, dates etc.)
  • Be very careful if any of your fields have leading zeros (0851 is NOT the same as 851)

The more time you spend on the quality of your import data, the more time it will save you afterwards.

3. Check that all your existing Data is still in use.

Just because there are columns and columns of data, doesn’t mean you need to import it all. Also, the people that use the data every day don’t necessarily understand what the columns and/or the data means. They probably inherited it from the last person that worked on it so on through generations of admin staff, each with varying degrees of efficiency and capability.

Check every field and chase down what it is for and whether or not it is still required. I was able to cut around 40% of the data as it no longer applied to this company.

4. Save a Master-list of your Import Data

Once you have checked it, cleaned it up and got your column naming and number conventions right, make sure you have saved a copy elsewhere. this becomes the Golden Master from which all your import files are created.

5. Learn how to Mass-delete your Records

Whether you made a mistake or your manager wants to change the index field, you need to be able to start again when doing mass imports. I used a particular field that I could do a query to select it all and remove everything to start again.

6. Get Someone Else to Check after Each Import

If someone else checks your work after each import, they can often pick  up a number of errors that you may have missed. At the very least they will pick out that the ABN (Australian Business Number) only imported the first 2 of 10 digits. For bonus points, you should try and get the stakeholder to check on it so if he decides to change anything he can do it then and there. Trust me, at some stage, there will be a change part-way through.

In Conclusion…

These are just a few of the things I came across during this project. now that all the data is in, now I am moving onto adapting our business processes into Salesforce to make them even easier and, more importantly, less error-prone… but that is a story for another day..

Posted in Work | Tagged , , | Leave a comment

From Spaces to WordPress

Posted on 30th September, 2010 by BigChaps

Well, I decided to make the jump sooner rather than later and moved my blog from Windows Live Spaces over to WordPress.com.

The wizard process was quite painless and everything migrated just right. The post dates remained intact, all the comment (including the ton of Comment spam – they never did get a handle on that…) and cleaned and tidied it up.

the only trouble I had was switching my Windows Live Writer 2011 Beta to see the new Blog. It’d pick up all the details but error out when adding the XMLRPC point. Turns out it was as simple as manually adding the Proxy settings to WLW. I’m not sure if that is due to TMG, WLW or perhaps even the IE9 influence here… needless to say, if you are reading this post, it worked…

Now all that remains is some minor adjestments in design and most importantly, more content. I must say that I have fallen a bit by the wayside, having posted mostly through Twitter (@BigChaps) but try to post articles with problems/solutions that others might be looking for to provide some help.

Posted in Online | Tagged , , | 1 Comment

Updating VMWare ESXi Hosts with extra NICs

Posted on 6th September, 2010 by BigChaps

Adding a Intel Quad Server NIC to Dell R510 + ESXi

We wanted to implement our new ESXi Host to use for our lesser Servers and started by configuring a  brand new Dell PowerEdge R510 Server. Unfortunately, these don’t come with enough NICs to be an affective Host.  We added additional  NICs via an Intel Quad Server NIC card and began the installation of ESXi 4.1

However…. The error shows up as follows:

Intel Quad Server NIC Card drivers not in VMWare install.

Blast… back to the command-line to add the new drivers.

Update via vSphere client + vSphere CLI

I installed the vSphere CLI tool on my maintenance VM and entered the following command

“C:Program FilesVMwareVMware vSphere CLIbin>vihostupdate.pl –server=192.168.50.128 –install –bundle c:AdminINT-intel-lad-ddk-igb-1.3.19.12.1-offline_bundle-185976.zip”

i was then prompted for user credentials

Enter username: root
Enter password:

followed by

Please wait patch installation is in progress …
The update completed successfully, but the system needs to be rebooted for the changes to be effective.

After the reboot was done, we were fine. Seems that particular Intel Quad Server NIC doesn’t have inbuilt drivers in the ESXi image.

Posted in Work | Leave a comment

The Lazy Engineer Solution

Posted on 25th June, 2010 by BigChaps

Never underestimate the lazy engineers to find the simplest solution

I’ve been notified of some ECC RAM errors on some of my blades in the DataCenter…  here is the old way and the new way to deal with it

Old Way…

  1. Contact stakeholders to let them know of issues
  2. Schedule down time of Services
  3. Create maintenance tools Boot CD
  4. Wait until after Business Hours
  5. Drive into DC
  6. Reboot Server and boot Diagnostic CD, hoping that it copied OK.
  7. Chew nails nervously while tools run while fending off calls about when downtime ends
  8. Finish and hope (pray!) the server starts.
  9. Check that all Services have returned to Production Levels
  10. Drive back to Office, listening to 24 Voicemail messages about how Services are down
  11. Send out notices and apologies while boss berates you for downtime

New Way

  1. Let boss know
  2. Download tool and create iso
  3. Migrate Virtual Servers to new Host
  4. Connect to Remote Access Card and set Virtual CD to Diagnostic iso and Boot order
  5. Reboot Host and run tests while getting coffee (the only time I leave my seat…)
  6. Restart blade and migrate Virtual Server back to host
  7. Let boss know it’s all done with no interruption to services

Nothing like lazy Engineers to work out a way that I can do all that without leaving my seat… No wonder Virtualisation is the new way…

EDIT: except that if it was in the Cloud, I wouldn’t have to worry about it at all

Posted in Computers | Tagged , | Leave a comment