Virtualization - Julio Merino (jmmv.dev)

Live from EuroBSDCon 2013, day 4

Live from Malta today attending the EuroBSDCon 2013 conference. The conference is over; today was the second and last day and it has just finished.

Hardware and virtualization

One of the three tracks today included a lot of talks on hardware, porting of BSDs to new hardware and virtualization techniques. Of all these, the few talks I attended covered the topics in great detail and proved to be very interesting.

September 29, 2013 · Tags: conference, eurobsdcon, kyua, virtualization
Continue reading (about 4 minutes)

VirtualBox and port forwarding (and Fusion)

I have been a VMware Fusion user for a long time and I think I am about to be a VirtualBox convert. Let me explain you my story.

With the release of the Windows 8 Developer Preview, I felt like giving the system a try. Not because I particularly like or care about Windows, but just because I am genuinely curious about operating systems and wanted to check out their new Metro interface. So I went ahead, downloaded the ISO image, created a new virtual machine under Fusion 3, started the virtual machine, and saw Fusion crash while the Windows installer was booting.

Oh well, an emulation problem. I tried to fiddle with the virtual machine settings a bit, selecting different CPU virtualization settings and different hardware devices. No luck. After that, I resorted to Google to look for a solution to the problem and found that other people were experiencing the same issue. But, at the same time, I also found that VMware had just released Fusion 4 and that this new release works with Windows 8 just fine. Ah, cool! Or not... $50 for the upgrade. For my use cases (which amount to booting a few BSD and Linux virtual machines in console mode), it is hard to justify the upgrade — specially when it would just be to fulfill my curiosity and then delete the virtual machine 10 minutes later.

But wait! There is that thing called VirtualBox. I had used this product a while ago and did not enjoy doing so because its interface was utterly (really) confusing. It was nowhere near the polish of Fusion or Parallels. However, to my surprise, most of my major itches have been fixed and the new interface of VirtualBox is bearable. And it boots the Windows 8 preview just fine!

Right. So after installing Windows 8, playing with it for 10 minutes, and later deleting it, I decided to look around and see what else VirtualBox has to offer. It has many settings. Yes, I know, Fusion has many settings too, but most are hidden from the UI and can only be accessed by modifying the vmx text file. In the case of VirtualBox they are easily accessible.

I kept looking around... and there it was, the "Port Forwarding" option staring at me under the Network settings.

One of the features I've been wishing for a long, long, long time is the ability to easily configure a NATed (not bridged) virtual machine in my laptop, assign a host name to it from the graphical interface and then be able to SSH into the virtual machine from the laptop. As it turns out, this is doable in Fusion but it requires some nasty tinkering: one has to edit a dhcpd configuration file by hand to add a new stanza for the virtual machine (/Library/Application Support/VMware Fusion/vmnet8/dhcpd.conf). This involves manually copy/pasting the MAC address of the virtual machine into that file (which you must first extract from the vmx file), assigning it an IP in the corresponding subnet, and sticking the IP plus host name in the /etc/hosts file. This gets boring pretty quickly.

VirtualBox may or may not support this kind of setting, but it has the aforementioned "Port Forwarding" option which is interesting on its own. As you can imagine, this sets up forwarding of a port from the host system to a port of the virtual machine. Creating an entry for SSH is as easy as clicking a couple of buttons, choosing a local port on the host system (e.g. 2022) and forwarding it to port 22 of the guest system. Voila. You can now access your virtual machine locally and remotely without having to go through any of the host name nor DHCP madness I experienced before:

$ ssh -p 2022 localhost

You can later create an alias in your ~/.ssh/config file to make accessing the virtual machine a breeze:

Host myvm
Hostname localhost
Port 2022

And later:

$ ssh myvm

It couldn't be easier. This trivial feature is a killer for the kind of environment I am interested in. And I also know that VirtualBox provides some other interesting features, like headless mode, that are not (easily) available with Fusion.

So yes, I think I am a convert.

September 19, 2011 · Tags: virtualization
Continue reading (about 4 minutes)

Home-made build farm

I'm about to publish the 0.4 release of ATF. It has been delayed more than I wanted due to the difficulty in getting time-limited test cases working and due to my laziness in testing the final tarball in multiple operating systems (because I knew I'd have to fight portability problems).

But finally, this weekend I have been setting up a rather-automated build farm at home, which is composed so far of 13 systems. Yes, 13! But do I use so much machines? Of course not! Ah, the joys of virtualization.

What I have done is set up a virtual machine for each system I want to test using VMware Fusion. If possible, I configure both 32-bit and 64-bit versions of the same system, because different problems can arise in them. Each virtual machine has a builder user, and that user is configured to allow passwordless SSH logins by using a private key. It also has full sudo access to the machine, so that it can issue root-only tests and can shutdown the virtual machine. And about the software it has, I only need a C++ compiler, the make tool and pkg-config.

Then I have a script that, for a given virtual machine:

Starts the virtual machine.
Copies the distfile inside the virtual machine.
Unpacks the distfile.
Configures the sources.
Installs the results.
Runs the build-time tests.
Runs the install-time tests as a regular user.
Runs the install-time tests as root.
Powers down the virtual machine.

Ideally I should also run some different combinations of compilers inside each system (for example, SUNpro and GCC in Solaris) and make tools (BSD make and GNU make). I'm also considering in replacing some of the steps above by a simple make distcheck.

I take a log of the whole process for later manual inspection. This way I can simply call this script for all the virtual machines I have and get the results of all the tests for all the platforms. I still need to do some manual testing in non-virtual machines such as in my PS3 or in Mac OS X, but these are minor (but yes, they should also be automated).

Starting and stopping the virtual machines is what was trickiest, but in the end I got it working. Now I would like to adapt the code to work with other virtual machines (Parallels and qemu), clean it up and publish it somehow. Parts of it do certainly belong inside ATF (such as the formatting of all logs into HTML for later publication on a web server), and I hope they will make it into the next release.

For the curious, I currently have virtual machines for: Debian 4.0r2, Fedora 8, FreeBSD 6.3, NetBSD-current, openSUSE 10.2, Solaris Express Developer Edition 2007/09 and Ubuntu Server 7.10. All of them have 32-bit and 64-bit variants except for Solaris, which is only 64-bit. Setting all of them up manually was quite a tedious and boring process. And the testing process is slow. Each system takes around 10 minutes to run through the whole "start, do stuff, stop" process, and SXDE almost doubles that. In total, more than 2 hours to do all the testing. Argh, an 8-way Mac Pro could be so sweet now :-)

February 4, 2008 · Tags: atf, virtualization
Continue reading (about 3 minutes)

A request to virtualization software developers

Here is a request for a feature I have not yet seen in any virtualization application — used Parallels Desktop 2, VMware Fusion 1.1 and another product I can't speak of yet — that I'd love to have. It'd make things so much easier for me... So here is an open request just in case one of the developers of free alternatives (e.g. VirtualBox) reads it and decides to get ahead of the competence by implementing it.

Before explaining my feature request, let's consider you have a server on your network on which you run multiple virtual machines (VMs) for whatever purpose. These machines are exported to the network using bridged networking so that other computers in the network can access them transparently as if they were physical computers. To make this setup trivial, you have a DHCP server on your network that hands out static IP addresses to these virtual servers, and you also have a DNS server that maps these addresses to static names. This way, users on your network can access the virtual machines by simply spelling out their host names.

Now let's move to the laptop world where you are connected to different networks all the time (e.g. at home or at work) or no network at all. Here I will assume that you will want to access the VMs exclusively from your laptop. In this case, you should not use bridged networking because you'd be exporting all your virtual machines to the possibly untrusted network. And you cannot rely on the external DHCP nor DNS servers to deal with static IP addresses nor host names for you because in many situations you have no control over them.

Your best bet is to use shared networking to configure your VMs (or host-only networking if they needn't access the outside world). But if you do so, your VMs will get random IP addresses because you have no control over the DHCP sever bundled into the virtualization application. And as a result of this, you cannot assign host names to them. As a workaround, you can manually configure each operating system running on a VM to have a static IP (bypassing DHCP), then add an entry in the host's /etc/hosts file to assign a host name to the guest OS and at last add an entry in the guest's /etc/hosts file to assign a host name to the host OS. Which is painful.

In my idea world, the virtualization applications could have the ability to fine-tune the bundled DHCP server to hand out specific addresses to the virtual machines (VMs) and a way to specify DNS host names for them, all from the configuration interface and without having to touch any configuration file in the host system (nor guest, for that matter). E.g. add a little configuration box for the IP address and host name of the guest OS alongside the box that already exists to configure the MAC address. Then have the bundled DHCP server hand out the appropriate entries to the guests, add an entry to the host's /etc/hosts and provide a virtual DNS server to the guests so that they can resolve each other's names.

An use case for this? I have two VMs that I carry around in my MacBook Pro that I use very frequently and that I do not want to expose to the outside network at all. One is a Fedora 8 installation and the other a NetBSD one. I start them up from the graphical interface and then access them through SSH exclusively. But in order to reliably use SSH, I need to do the above manual steps to set up a host name for them, or otherwise using SSH is a pain.

I am also trying to set up an automatic build farm for ATF (composed probably of 10-15 VMs) and the need to set all these details manually is extremely boring.

January 18, 2008 · Tags: networking, virtualization
Continue reading (about 4 minutes)

Parallels Desktop and VMware Fusion

Back in February, I bought a copy of Parallels Desktop 2 and have been a very happy user of it since then. However, when Parallels 3 appeared, I hesitated to pre-order it (even at a very low price) and I did well: after it was released, I tried it on my MacBook Pro and their 3D support is useless for me. I could not play neither Half-Life 2 nor Doom 3 at acceptable speeds, being the former much worse than the latter in this regard.

Now, I'm evaluating VMware Fusion RC1, and I'm almost convinced to pre-order it. This product is very similar to Parallels and in fact several of its features seem "inspired" on it, such as Unity (Coherence in Parallels speak). But it has some important features that Parallels cannot currently match. To know: support for 64-bit guests, support for 2 virtual CPUs in guests and support to network-boot the virtual machine. All of these are cool from a development point of view. The first two allow one to run some more versions of specific operating systems, such as NetBSD/amd64, as well as enabling SMP support in them. The last one makes it easy to boot development kernels without modifying any virtual disk (haven't tried this yet, though).

All is not good though. Fusion is also supposed to have experimental 3D support inside the guest machines (up to DirectX 8.1). However, when trying to launch Half-Life 2 inside a Windows XP SP2 virtual machine, Windows crashed with a BSOD. At least I could launch it in Parallels, albeit it was simply unplayable. But as none of the two products make for a good gaming experience, I personally don't care about this feature.

Let's conclude with some numbers about the speed of each product. I installed Debian GNU/Linux leeny under Parallels and Fusion and built monotone-0.35 from scratch under them. The virtual machines were configured to have 768MB of RAM of a total of 2GB and the machine was idle aside from the build jobs. Obviously, in the case of Parallels I could only run the test with the i386 port, but in Fusion I used both the i386 and amd64 ports with 1 and 2 virtual CPUs. I also ran the same tests on the native machine using Mac OS X 10.4.10. The timings only include the make command, not ./configure.

Parallels, 32-bit, 1 virtual CPU, 'make':
real 17m33.048s, user 14m24.342s, sys 3m4.080s

Fusion, 32-bit, 1 virtual CPU, 'make':
real 16m35.507s, user 14m57.016s, sys 1m29.134s

Fusion, 32-bit, 2 virtual CPUs, 'make -j2':
real 10m0.341s, user 17m23.541s, sys 2m12.604s

Fusion, 64-bit, 2 virtual CPUs, 'make -j2':
real 10m24.617s, user 18m26.985s, sys 1m26.133s

Native, Mac OS X, 'make':
real 12m50.640s, user 11m12.997s, sys 1m20.344s

Native, Mac OS X, 'make -j2':
real 7m3.536s, user 11m22.875s, sys 1m26.366s

See this thread for other opinions.

July 22, 2007 · Tags: fusion, parallels, virtualization
Continue reading (about 3 minutes)

Cross-platform development with Parallels

These days I'm seizing some of my free time to continue what I did as my SoC 2006 project: the Boost.Process library. There is still a lot of work to be done, but some items are annoying enough to require early attention (well, I can't speak of "early" because I hadn't touched the code for months).

Boost.Process aims to be a cross-platform library and currently works under POSIX-based systems (such as Linux, NetBSD or Mac OS X) as well as under Win32 systems. However, developing such a thing is not easy if you don't have concurrent access to both systems to test your code as you go. That is because, past summer, Win32 support was "second class": I first coded everything under NetBSD and, eventually, I fired up my Windows XP installation and fixed any problems that arised due to the new code. This was suboptimal and really slowed down the development of the library.

Now, with a MacBook Pro and Parallels Desktop for Mac, these issues have gone away. I can now code under whichever system I want and immediately test my changes on the other system without having to reboot! It's so convenient...

And, with Coherence mode, everything is so transparent... just check out the following screenshot:

To make things better I could share the project's code over the virtual network to avoid having to commit changes to the public repository before having tested them on the two systems. If you inspect the logs, you'll see many "Add feature X" commits followed by a "Fix previous under Win32". But it is a minor issue right now.

Kudos to the Parallels developers, who made this possible and painless. I now understand the "computer as a tool" paradigm rather than a "computer as a hobby".

April 2, 2007 · Tags: boost-process, parallels, virtualization
Continue reading (about 2 minutes)

Posts: Virtualization