Jump to the Q&A.
Choosing a virtual machine environment isn’t something a software developer usually does, but as I was familiar with virtual machines (ala Parallels and VMware) I was tasked with installing virtual machines on a new set of six Dell R210 II 1U rackmount server hardware. The purpose of the virtual machines is for development only, no production use, and developers would be the end users. I could also be using the VMs for developing automated deployment of a “pod,” which is a set of servers carefully installed and configured to run the AppFirst suite of software and all of the supporting open source packages.
At first I went with what I knew best: VMware Player. After loading an operating system (CentOS 5.7) on the metal and spending some days attempting to get VMware Player installed and running on an older version of CentOS with many applied updates that seemed to have made it impossible to install VMware Player, I was kindly asked if I knew what a hypervisor was (I sorta knew) and XenServer was suggested as a viable alternative to a full-blown OS install on the metal. It didn’t take much reading (on the Citrix web site) to find that XenServer (and XenCenter) could be very useful for the development servers and for automated deployment.
I was all in.
I installed the free version of XenServer on each machine using an external USB optical drive and XenServer 6.0.0 on DVD. The installation was easy and familiar, as it looked similar to a CentOS install. To make it easy for developers to manage their own VMs, I installed XenCenter, which unfortunately is a Windows application that I had to find a home for that everyone could access. For this I had to create a Windows 2003 Server VM under Parallels on a Mac mini. Once I had XenCenter installed I added all of the XenServer hosts to the managed resources in XenCenter. I started creating VMs using the same CentOS 5.7 DVD I had previously used to install on the metal, and again I used the external USB optical drive.
The first issue I encountered when creating a VM was that the list of built-in templates didn’t have everything I was looking into (e.g., Gentoo). Another problem I ran into was with using the CentOS 5 template to install CentOS 5.7, where the Anaconda installer was not able to find the external USB optical drive (or an NFS ISO SR, see below) from which Anaconda was executed. Instead, I had to use the “Other” template for all VM installs of CentOS 5.7 (though CentOS 5.5 worked with the CentOS 5 template), which went well enough, though I never did figure out how to get the XenServer tools installed when using the “Other” template (unable to create /dev/vxdd for some unknown reason). Later on I figured out how to enable the NFS install method within the Anaconda installer to point at an NFS location containing the ISO images and get the XenServer tools installed successfully. Once I had a few VMs installed and running I decided to use the CentOS 5.7 ISO image to install more VMs. To do this I needed to create a new ISO library storage repository (SR).
I attempted to create the SR using a CIFS share hosted on the Mac mini but failed due to some permissions problem (though I was able to map the share manually in Windows). Instead, I exported the share via NFS on the Mac mini and created an NFS ISO SR. This eliminated the need for the external USB optical drive and DVD, which had to be physically plugged into each XenServer that I was creating a VM on. The VM installations on that XenServer host went well with this SR, so I started to create the same SR for other XenServer hosts when I wondered whether a pool would help by sharing the NFS ISO SR between the XenServer hosts. (On a related topic, up to this point I was creating VMs using virtual disks on “Local storage” on each XenServer host.) I then began to add XenServer hosts to a pool and was able to create one NFS ISO SR for the pool used by all XenServer hosts. I attempted to create a virtual disk SR (NFS VHD) to use for virtual disk storage instead of local storage. Foiled again by a permissions error I did not fully understand. I knew I’d be able to backup VMs using XenCenter’s export function, so I didn’t worry about this failure.
After getting a number of VMs installed using the NFS ISO SR within the pool I thought I was stylin’, that is, until I tried to change the IP addresses of the XenServer hosts, especially the pool master. Oh, for such little gain did I endure so much pain. First, I did not fully understand the ramifications of the pool. Second, I did not grasp the subtleties of a virtual NIC. Third, I really should read more before I go “all in.” Changing the IP address of the pool master caused all other XenServer hosts to become unavailable. It would be simple enough, I thought, to login to each XenServer host on its console and change the IP address and recreate the pool. Unfortunately, the network interface on each XenServer host became “uninitialized” and I was unable to assign an IP address. Worse yet, after I started reading more on XenServer pools, I found that changing the IP address on a pool master before changing the IP addresses of the other pool members disables networking on the pool members. The pool master IP address changed because I changed the entire IP address space of the internal network. Since I was still in experiment mode, I didn’t really care that the XenServer host IP addresses were received via DHCP. Finding the pooled XenServer hosts incapacitated, I started to care considerably more. I backed out the new network IP address configuration and went back to square 1. Then I figured I’d better remove these XenServer hosts from the pool before applying the new network IP address configuration.
My anxiety peaked when I found that the VMs on a member’s local storage are obliterated when removing a XenServer host member from a pool (which made me think of a pool as a sort of Hotel California). This forced me to exercise the VM export feature, which thankfully worked as advertised. I was able to save all VMS by exporting each VM on each pool member, removing each member from the pool, and importing each VM back to the XenServer host now outside of the pool. Once all XenServer hosts were ‘de-pooled’ and all their respective VMs were imported and running again I shut everything down, re-applied the new network IP address configuration, and voila! All XenServer hosts came up with their network interfaces initialized, their VMs working fine, and my anxiety came back down to a normal level. I promptly locked in all IP addresses using the console on each XenServer host and did the same in turn to all VMs, but with a bit more difficulty. The IP address change messed with the virtual NIC on these CentOS VMs such that a duplicate NIC configuration for each NIC was added (e.g., eth0.bak). I wasn’t sure which one to assign the static IP address to until I went to the Networking tab for the VM in XenCenter and matched MAC addresses (I had to ignore/remove the .bak interfaces).
With the pool now closed I re-configured the NFS ISO SR for each XenServer host so I could get on with creating other VMs using ISO images I had downloaded into the NFS exported directory on the Mac mini. I am still a bit confused as to why I need to use a template to install XenServer tools, though it hasn’t been an issue yet not having them. And thankfully I haven’t had to use the external USB optical drive for any further VM installs.
And now for the question and answer section:
Q. What more difficult than it should have been?
A. Removing pool members from a pool.
Q. What was easier than you thought?
A. Adding XenServer hosts to XenCenter, and subsequently to a pool.
Q. What did you like?
A. Easy and familiar install of XenServer.
Q. What did you despise?
A. The ease of adding a XenServer host to a pool only to find out
the Local storage for all VMs would be wiped out upon de-pooling.
And having to use a Windows box for XenCenter.
Q. In the end does it work?
A. Yes. Using XenCenter to manage XenServers and VMs works
as advertised.
Q. What would you like to see different?
A. Do not blow away Local storage when de-pooling.
In conclusion, I found using XenServer itself to be fairly straightforward,
making it easy to manage VMs via XenCenter. I see possibly some day using the command line tools on the XenServer host in conjunction with automated deployment scripts to configure and create VMs.