19 September 2010

I have upgraded a XenServer using the install disk, but all my NICs are missing

When upgrading a XenServer pool there is an important order to follow to perform what Citrix call a "rolling upgrade" where servers are upgraded one server at a time, leaving all the virtual machines in a running state.

The correct order is to shut down the pool master first, which puts the pool into an emergency mode where the other host servers continue to run even though the pool master is missing. The next step is to upgrade the pool master and restart it at which point it rejoins the pool. Then upgrade each of the other host servers in turn, with the end result of all your servers running and with all of your servers upgraded.

If the upgrade is started with one of the servers that isn't the pool master first all might seem to go well, but on restart of the upgraded server it may remain offline with no network interfaces defined. On attempting to redefine network interfaces through the console, an error will be reported that no NICs are found in the server.

The simplest fix assuming this is the first server in the pool with an attempted update is to restore the server to XenServer 5.5 and then make the upgrade in the correct order. As part of the updated process, a backup was automatically made on the server. It is important to reboot using a XenServer 5.5 CD, and select the restore option from the menu. If the restore is made using a XenServer 5.6 CD it will reach the end of the restore and report "Errno 16 Device or resource busy". The same error has also been seen when restoring with a XenServer 5.5 CD.

If a restore can't be performed with either a XenServer 5.5 or a XenServer 5.6 CD, it is possible to correct the restore process by relinking the files it is looking for:

• Boot the XenServer install CD as usual, and wait for the choice of keyboard type to appear.
• At the keyboard menu, press ALT-F2 to open a second command line:
  
  Typing
  
  ls /dev/disk/by-id/
  
  will show several files that end -part1, -part2, -part3
  
  
• We want to link an alternative name to these files, replacing -part1 with p1. We want to leave the original file in place, so we use a symbolic link, e.g.
  
  ln /dev/disk/by-id/cciss-3600508b1001030363535433339300d00-part1 /dev/disk/by-id/cciss-3600508b1001030363535433339300d00p1
  
  and repeat for the other parts.
  
  The filename after /dev/disk/by-id/ will depend on the system, and for example iscsi based systems will have a name like
  /dev/disk/by-id/iscsi-3600508b1001030363535433339300d00-part1
  
• Press ALT-F1 to switch back to the keyboard menu once the necessary links have been made.
• Continue with the recovery as normal.

The server should now reboot and have all of its configuration restored, at which point the upgrade process can be started again, this time in the correct order. Shutdown the master, apply the upgrade, reboot the master, wait for it to exit maintenance state, and then repeat with the other servers.

• Upgrading from XenServer 5.5 to 5.6 and missing NICs
• Catch a virus at Christmas
• Email on the move
• How to dispose of your PCs
• Sizing up file sending
• Vista Upgrade
• Spam Avalanche