Assign dynamic IP to VM without DHCP

The system we are developing is intensively using virtual machines. When the system core requires another machine it copies the template image, and starts it. Thus, many copies of essentially the same machine can work simultaneously.

Of course, at launch each virtual machine is identical to the template including the inherited set of the network settings. All virtual machines run on the same subnet, and therefore they should not use the same IP obtained from the template image to prevent conflicts. That is, each machine should get its own IP. It would seem that the solution is simple – use a DHCP server and dynamic IPs.

However, there are other options which I will discuss in this post.

1. Why do we need virtual machines?
2. The problem of duplicate IP addresses when running multiple VM clones
3. Encoding IP into MAC address
4. Allocating IP by MAC: DHCP
5. Allocating IP by MAC: mac2ip script
a) Linux
b) Windows
6. Final notes

1. Why do we need virtual machines?

Our project, Nerrvana, executes functional tests for web applications in different browsers. Tests are run with a special framework for functional testing – Selenium, which can simulate user actions inside a browser (clicks on elements, mouse movements, data input, text reading), make screenshots of web pages and some other stuff.

A web application test is a sequence of actions emulating user behavior, and comparison of their outcomes with expected ones. For example, a simple test – a login. You need to open the login page, enter your login and password, press enter, and make sure we’re logged in – say, seeing the standard welcome message which includes username. Everyone knows that browsers can quite differently display and work with the same page, and so it makes sense to perform the same tests in the most popular browsers. As I already mentioned, our system does just that – it launches Selenium grid in a virtual environment and executes tests.

Tests in the selected browsers and operation systems are run simultaneously and completely independently from each other. Execution of a test suite in one environment was given a funny name by us – speck. If I want, for example, to test login functionality inside IE8 and FF3.6, our system will create two independent specks and will launch tests using selected browsers. Not going into details too much, I should say that for every speck we create at least two virtual machines. One is a “hub”, will execute tests. It can launch tests in Java or PHP we support for now. The second machine is a “tester”, running Selenium RC, OS and the browser of your choice. Selenium RC is an interaction point between tests executed on the “hub” and a browser.

2. The problem of duplicate IP addresses when running multiple VM clones

We have a template image for each type of a virtual machine. When the core of the system requires another machine, this image is copied, and runs. The template image itself remains unchanged. Thus, many copies of essentially the same machine can be up and running at the same time.

And here a problem arises.
Naturally, at the moment of launch each virtual machine is identical to the template. All settings are inside an image, which can not be altered. This obviously includes a network setup. Since all the virtual machines run on the same subnet, it is clear that they must not use same IP, obtained from a template machine to prevent conflicts. So, each machine must dynamically obtain its own IP. It would seem that the solution is simple – to use a DHCP server, which will give each machine a unique address.

However, not all that easy. The fact is our core actively interacts with virtual machines. It plays an active role keeping virtual machines as slaves and doing a lot of work with them: makes sure all virtual machines (VMs) in a grid have successfully started, starts Selenium RC on all testers, loads and executes code on the hub, monitors tests execution, and then gets back results (screenshots, logs, etc.) before destroying VMs. System core, anyway, need to know VMs IP addresses which were just launched on its’ request.

We had two solutions in mind:
1) Once the virtual machine is booted, it gets a random address from DHCP, and then somehow registers itself with the core by saying: “I am – a machine of a this type. My IP, received from the DHCP, is this”.
2) Somehow, the core tells VM what address it should use. Here the core knows VMs’ IP before it boots.

The first approach seemed to us quite difficult to implement for several reasons. It required some shift of responsibilities from the core to VMs (we wanted to keep them as slaves), problems with matching requested and registered VMs as well as some inconveniences in changing core components.
The second option was more appealing to us, because the core has full control over the VM from the point of its creation without an uncertainty period while waiting for the VM to boot and register itself from option one.

3. Encoding IP into MAC address

How could we pass an IP to a starting VM? We have found the following way: when you start a virtual machine, you can specify a MAC address of the virtual network adapter. A virtual machine itself can read it any time. Our ‘invention’ was to use MAC address as a carrier of information about the IP (could be anything you want).

How we encode IP into MAC address? We use Xen for virtualization, and its virtual NICs must have MACs assigned which look like this: 00:16:3E:XX:XX:XX, where 00:16:3E – network adapter manufacturers’ code. We can use last three bytes at our discretion (remembering, of course, the fact that the MAC must be unique). Suppose that our system will be set on the subnet 10.0.0.0/8. A logical decision in this case will be to use for the last three bytes of MAC corresponding to the last three bytes of IP addresses that we want to encode. For example, to start VM with IP 10.1.1.3, we start VM with MAC 00:16:3E:01:01:03.

Now we only need to teach the VM to get an IP from its MAC. Again, this can be done in two ways.

4. Allocating IP by MAC: DHCP

The first way is pretty obvious. We will run a DHCP server, which will be configured to give IP by MAC-address in accordance with the described above method. Template image must be configured with DHCP enabled.

Work scenario will look like this:
1) The core needs to start a VM.
2) The core looks at IP-addresses pool, chooses the first unoccupied IP (for example, 10.4.0.15), and converts it to a MAC (00:16:3E:04:00:0F)
4) The core makes the template image copy and prepares a VM configuration file (which contains MAC address)
5) The core launches the prepared VM
6) OS inside launched VM requests IP address from DHCP server. DHCP server checks its MAC to IP table and responds with IP 10.4.0.15.
7) The core is pinging 10.4.0.15, and upon receiving a response, continues to work with a virtual machine (in our case connecting via ssh)

Pluses and minuses:
- requires a DHCP server
- if we move to a different subnet or get a different pool of IP-addresses, we will have to reconfigure not only the core but also a DHCP server
+ uses a standard approach to dynamic IP allocation

5. Allocating IP by MAC: mac2ip script

The second approach is less obvious and requires some extra work. It lies in a need to create an extra script which will be injected into OS start-up process. It will get a MAC address, convert it to an IP, and assign it to a network card. Work scenario will be the same with the exception of point 6 which will read:

6) OS’s start up script inside launched VM will read MAC, convert it to an IP address and assign it to VM’s network interface.

Pluses and minuses:
+ no need for DHCP server and MAC to IP conversion table maintenance inside it
- each type of OS will require own implementation of mac2ip script

We implemented the mac2ip approach. We are working with virtual machines running CentOS 5.6 and Windows XP PRO SP3, and so we had two scripts mac2ip – for each of the systems.
Let’s look at both scripts.

a) Linux
#!/bin/bash
 
# first byte will always be equal to 10
IP1=10
 
IFCFG=/etc/sysconfig/network-scripts/ifcfg-eth0
NETWORK=/etc/sysconfig/network
 
case "$1" in
*start)
;;
*)
exit
;;
esac
 
# get MAC and validate it
MAC=$(ifconfig eth0|grep HWaddr|awk '{print $NF}'|grep ^00:16:3E)
if [[ -z "$MAC" ]] ; then
echo "Can't determine MAC address" >&2
exit 1
fi
 
# convert MAC to IP
set -- $(echo $MAC|awk -F: '{print $4,$5,$6}')
IPADDR=${IP1}.$((0x$1)).$((0x$2)).$((0x$3))
 
# change network inreface
sed -i -e "s/^IPADDR.*/IPADDR=$IPADDR/" $IFCFG
sed -i -e "/^HWADDR/d" $IFCFG
sed -i -e "s/^HOSTNAME.*/HOSTNAME=localhost/" $NETWORK

Now force this script to run before the network starts.

b) Windows

As you expect the same work on Windows XP is done in more arcane ways. Perhaps, over time, we will find a better way to convert MAC to IP.

The first problem I encountered is that it is relatively hard to change the interface BEFORE it will be enabled. Thus, the template VM must have default networking turned off by default as otherwise two or more simultaneously launched VMs running Windows will conflict with their identical IPs (it is static, because we are not using DHCP).

Ok, this is not the problem – we can turn off network interface. However, the utility getmac, which we will be using to get MAC address can not return it for the turned off interface! To work around it we will assign a random interface IP first, turn interface on, get MAC, and finally assign a correct IP.

We used devcon utility to manipulate devices.
Here’s how it looks:

View Code WINBATCH
@echo off
SET MAC=
SET IP=
SET MASK=255.255.255.0
SET GATEWAY=
 
rem get random IP. This IP will be used for a few seconds,
rem and this is why it is good enough to random generate two last bytes.
rem 100 - shift, to assure that we will not use any used IP.
set /A TEMP_THIRD_BYTE=100+%RANDOM:~0,2%
set /A TEMP_FOURTH_BYTE=100+%RANDOM:~0,2%
set TEMP_IP="192.168.%TEMP_THIRD_BYTE%.%TEMP_FOURTH_BYTE%"
set TEMP_GATEWAY="192.168.%TEMP_THIRD_BYTE%.1"
 
rem set network interface params
netsh interface ip set address local static %TEMP_IP% %MASK% %TEMP_GATEWAY% 1
 
rem turn on interface. We use this to do it -
C:\devcon\i386\devcon enable *VEN_10E*
 
rem get three last bytes from MAC
FOR /F "Tokens=4-6 Delims=- " %%d in ('getmac^|find "Device\Tcpip_"') do (
set /a dec_d=0x%%d
set /a dec_e=0x%%e
set /a dec_f=0x%%f
)
 
rem set real IP and gateway
SET IP=10.%dec_d%.%dec_e%.%dec_f%
SET GATEWAY=10.%dec_d%.0.1
 
rem change interface params to real ones
netsh interface ip set address local static %IP% %MASK% %GATEWAY% 1
netsh interface ip set dns local static %GATEWAY%

Add this script into start-up sequence – for example, this way:

View Code WINBATCH
reg ADD "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run" /v "mac2ip" /t REG_SZ /d "c:\init\mac2ip.bat"

After these scripts completion a VM will receive an IP address specified by the core and will be ready to work. The core will know about it from ping replies from an assigned IP.
Perhaps, this diagram will add some clarity (click for a full-size picture in a new window):

6. Final notes

Our approach with mac2ip script is rather slow on Win XP. The current implementation does the job in 15-20 seconds. You understand our feelings if we will say that same 15-20 seconds required from starting OS to a moment mac2ip starts. Script doubles time to launch a VM.

However, we aren’t in a rush to start using DHCP method (DHCP with binding IP to MAC), because:

- firstly, we know for sure that Linux will boot via DHCP slower than working version with a static address. Will the Windows boot faster with DHCP comparing with mac2ip script? Hard to say. We need to check it out.

- secondly, VMs are not shut down properly in our system. We do ‘virsh destroy’ for the VM which is similar to disconnecting power for a physical host. This saves us tons of time as we loaded all the data we needed from VMs and do not care about their integrity. We do not reuse VMs and they will be immediately deleted after use. So, DHCP leased VM address will not be released immediately. It will happen after default-lease-time according to DHCP etiquette. This means that we can not immediately start the VM with the same MAC (and same IP). Setting default-lease-time to a small value (seconds) is not a good idea. A more realistic option – to look at and use OMAPI/omshell to manipulate DHCP and remove unwanted entries straight after ‘virsh destroy’ing a VM.

- in third – any new approach will bring its own hidden pitfalls.

So the slow work of the Windows script in the current version is not a good enough reason to switch to DHCP.

What’s in my name? It’s soulless,
It shall expire, like the dismal roar
Of waves that hit the distant shore, –
Like nighttime noises in the forest!

Alexander Pushkin

Print this post | Home

5 comments

  1. wwfalcon says:

    why not to use Xenstore? http://wiki.xen.org/xenwiki/XenStoreReference?highlight=%28Xenstore%29. Before the VM startup ,use it to write key-value,after use Xenstore to read key-value.

  2. bear says:

    wwfalcon, you are right – this is another option, thanks.
    We heard about it after described approach was already implemented.
    Besides, as far I know, there are no ready to use implementation for xenstore-read in Windows – so we would need to write it if we decided use xenstore.

  3. Anil says:

    how to configure different vm’s with different gateways which all are linux templates

  4. bear says:

    Anil,

    I am not sure I understand your question.

    As I remember (it was not my work) in Linux templates we configure default gateway when creating template on particular xen server, so there is no need to change this default in mac2ip script.
    You can set up gateway depending on IP by additional sed substitution in last several strings of the script.

  5. MeNok says:

    Nice post!
    I found this when i was googling if there`s way to change mac/ip address dynamically from outside the live VM (from host OS).
    Wonder if you can help~THX