Boost-Process - Julio Merino (jmmv.dev)

Child-process management in C for ATF

Let's face it: spawning child processes in Unix is a "mess". Yes, the interfaces involved (fork, wait, pipe) are really elegant and easy to understand, but every single time you need to spawn a new child process to, later on, execute a random command, you have to write quite a bunch of error-prone code to cope with it. If you have ever used any other programming language with higher-level abstraction layers — just check Python's subprocess.Popen — you surely understand what I mean.

The current code in ATF has many places were child processes have to be spawned. I recently had to add yet another case of this, and... enough was enough. Since then, I've been working on a C API to spawn child processes from within ATF's internals and just pushed it to the repository. It's still fairly incomplete, but with minor tweaks, it'll keep all the dirty details of process management contained in a single, one-day-to-be-portable module.

The interface tries to mimic the one that was designed on my Boost.Process Summer of Code project, but in C, which is quite painful. The main idea is to have a fork function to which you pass the subroutine you want to run on the child, the behavior you want for the stdout stream and the behavior you want for the stderr steam. These behaviors can be any of capture (aka create pipes for IPC communcations), silence (aka redirect to /dev/null), redirect to file descriptor and redirect to file. For simplicity, I've omitted stdin. With all this information, the fork function returns you an opaque structure representing the child, from which you can obtain the IPC channels if you requested them and on which you can wait for finalization.

Here is a little example, with tons of details such as error handling or resource finalization removed for simplicity. The code below would spawn "/bin/ls" and store its output in two files named ls.out and ls.err:

static
atf_error_t
run_ls(const void *v)
{
 system("/bin/ls");
 return atf_no_error();
}

static
void
some_function(...)
{
 atf_process_stream_t outsb, errsb;
 atf_process_child_t child;
 atf_process_status_t status;

 atf_process_status_init_redirect_path(&outsb, "ls.out");
 atf_process_status_init_redirect_path(&errsb, "ls.err");

 atf_process_fork(&child, run_ls, &outsb, &errsb, NULL);
 ... yeah, here comes the concurrency! ...
 atf_process_child_wait(&child, &status);

 if (atf_process_status_exited(&status))
     printf("Exit: %dn", atf_process_status_exitstatus(&status));
 else
     printf("Error!");
}

Yeah, quite verbose, huh? Well, it's the price to pay to simulate namespaces and similar other things in C. I'm not too happy with the interface yet, though, because I've already encountered a few gotchas when trying to convert some of the existing old fork calls to the new module. But, should you want to check the whole mess, check out the corresponding revision.

June 21, 2009 · Tags: atf, boost-process, c
Continue reading (about 3 minutes)

Boost.Process and SIGCHLD

For some unknown reason, I'm regaining interest in Boost.Process lately. I guess many of the people who have written me in the past asking for the status of the library will be happy to hear this, but I can't promise I will stick to coding it for long. I have to say that I have received compliments from quite a few people... thanks if you are reading and sorry if I did not reply you at all.

Anyway. So I downloaded my code and ran the unit tests under Mac OS X to make sure that everything still worked before attempting to do any further coding. Oops, lots of failures! All tests spawning a child process broke due to an EINTR received by waitpid(2). That doesn't look good; it certainly didn't happen before.

After these failures, I tried the same thing under Linux to make sure that the failures were not caused by some compatibility issue with Mac OS X. Oops, failures again! Worrisome. The curious thing is that the tests do work in Win32 — but that can be somewhat expected because all the internal code that does the real work is platform-specific.

Curiously, though, running the examples (not the tests, but the sample little programs distributed as part of the library documentation) did not raise any errors. Hence, I tried to run gdb on the actual tests to see if the debugger could shed any light on the failures. No way. Debugging the unit tests this way is not easy because Boost.Test does a lot of bookkeeping itself — yeah, newer versions of the library have cool features for debugging, but they don't work on OS X. Hmm, so what if I run gdb on the examples? Oh! The problem magically appears again.

It has taken me a long while to figure out the problem. Along the process, I have gone through thoughts of memory corruption issues and race conditions. In the end, the response was much simpler: it all turns out to SIGCHLD (as the error code returned by waitpid(2) well said).

SIGCHLD is received by a process whenever any of its children change status (e.g. terminates execution). The default behavior of the signal handler for SIGCHLD is to discard the signal. Therefore, when this signal is received, no system calls are aborted because it is effectively discarded. However, it turns out that newer versions of Boost.Test install signal handlers for a lot of signals (all?) to allow the test monitor to capture unmanaged signals and report them as errors. Similarly, gdb also installs a signal handler for SIGCHLD. As a result, Boost.Process does not work when run under gdb or Boost.Test because the blocking system calls in the library do not deal with EINTR, but it actually works for non-test programs run out of the debugger.

The first solution I tried was to simply retry the waitpid(2) whenever an EINTR error was received. This fixes the problem when running the tests under gdb. Unfortunately, the test cases are signaled as failed anyway because the test monitor still receives SIGCHLD and considers it a failure.

The second solution I have implemented consists on resetting the SIGCHLD handler to its default behavior when Boost.Process spawns a new child and restoring the old SIGCHLD handler when the last child managed by Boost.Process is awaited for. Eventually, the library could do something useful with the signal, but discarding it seems to be good enough for now.

This second solution is the one that is going to stay, probably, unless you have any other suggestion. I still feel it is a bit fragile, but can't think of anything better. For example: what if the user of Boost.Process had already programmed a handler for SIGCHLD? I just think that such a case shouldn't be considered because, after all, if you are using Boost.Process to manage children processes, you shouldn't have to deal with SIGCHLD on your own as long as the library provides a correct abstraction for it.

October 18, 2008 · Tags: boost-process
Continue reading (about 4 minutes)

Cross-platform development with Parallels

These days I'm seizing some of my free time to continue what I did as my SoC 2006 project: the Boost.Process library. There is still a lot of work to be done, but some items are annoying enough to require early attention (well, I can't speak of "early" because I hadn't touched the code for months).

Boost.Process aims to be a cross-platform library and currently works under POSIX-based systems (such as Linux, NetBSD or Mac OS X) as well as under Win32 systems. However, developing such a thing is not easy if you don't have concurrent access to both systems to test your code as you go. That is because, past summer, Win32 support was "second class": I first coded everything under NetBSD and, eventually, I fired up my Windows XP installation and fixed any problems that arised due to the new code. This was suboptimal and really slowed down the development of the library.

Now, with a MacBook Pro and Parallels Desktop for Mac, these issues have gone away. I can now code under whichever system I want and immediately test my changes on the other system without having to reboot! It's so convenient...

And, with Coherence mode, everything is so transparent... just check out the following screenshot:

To make things better I could share the project's code over the virtual network to avoid having to commit changes to the public repository before having tested them on the two systems. If you inspect the logs, you'll see many "Add feature X" commits followed by a "Fix previous under Win32". But it is a minor issue right now.

Kudos to the Parallels developers, who made this possible and painless. I now understand the "computer as a tool" paradigm rather than a "computer as a hobby".

April 2, 2007 · Tags: boost-process, parallels, virtualization
Continue reading (about 2 minutes)

Mac OS X vs. Ubuntu: Introduction

About a week ago, my desktop machine was driving me crazy because I couldn’t comfortably work on anything other than NetBSD and pkgsrc themselves. With “other work” I’m referring to Boost.Process and, most importantly, university assignments. Given my painless experience with the iBook G4 laptop I’ve had for around a year, I was decided to replace the desktop machine with a Mac—most likely a brand-new iMac 20"—to run Mac OS X on top of it exclusively—OK, OK, alongside Windows XP to satisfy the eventual willingness to play some games.

September 28, 2006 · Tags: boost-process, macos, ubuntu
Continue reading (about 2 minutes)

What have I learned during SoC?

One of SoC's most important goals is the introduction of students to the free software world; this way there are high chances that they will keep contributing even when SoC is over. Students already familiar with FOSS (as was my case both years) are also allowed to participate because they can seize the Summer to learn new stuff and improve their skills.

As I expected, the development of Boost.Process has taught me multiple new things. First of all, I wanted to get familiar with the Win32 API because I knew nothing about it. I have achieved this objective by learning the details about process and file management and making Boost.Process work under this platform. Sincerely, Win32 is overly complex but has some interesting features.

Secondly, I have got a lot more fluent with C++ templates and have learned some curious coding techniques that I never thought about in the past. The most impressive one in my opinion is that templates can be used to achieve build time specialization, avoiding expensive virtual tables at run time and inheritance when these are not really needed. (I only considered them for polimorphic containers before.)

At last, I have also got into several utilities used for Boost development. Among them are Quickbook for easy document writing, Boost.Build v2 for portable software building and the Boost Unit Test library for painlessly creating automated test suites.

All in all I'm happy with the outcome of the project and the new knowledge. If SoC happens again, you should really consider joining if you have the chance!

August 22, 2006 · Tags: boost-process, soc
Continue reading (about 2 minutes)

Boost.Process 0.1 published

SoC 2006 is officially over — at least for me in my timezone. Given that the Subversion repository has some problems with public access, I've tagged the current sources as the first public version and uploaded a couple of tarballs to the Boost Vault. Both the tag and the tarballs will also serve historical purposes, specially when newer ones come ;-)

You can download the archives from the Process directory in tar.gz and ZIP formats. Enjoy!

August 21, 2006 · Tags: boost-process, soc
Continue reading (about 1 minute)

Boost.Process tarballs posted

As everybody is not comfortable accessing Subversion repositories to download source code, I've posted two tarballs with Boost.Process' sources. They include an exported copy of the repository contents as well as prebuilt documentation in the libs/process/doc/html subdirectory.

You can download the compressed archive either in tar.gz format or in ZIP. Keep in mind that these will be updated very frequently so please do not use them to prepackage the library.

Changes from yesterday's announcement are minor at this point. For the curious ones: there is now a list of pending work and the Last revised item in the main page has been fixed. As a side effect of this last change, Boostbook will support SVN's $Date$ tags if my patch is integrated :-)

August 17, 2006 · Tags: boost-process, soc
Continue reading (about 1 minute)

SoC: Boost.Process published

In a rush to publish Boost.Process before the SoC deadline arrives, I've been working intensively during the past two days to polish some issues raised by my mentor. First of all I've added some Win32-specific classes so that the library does not seem Unix-only. These new classes provide functionality only available under Windows and, on the documentation side, they come with a couple of extra examples to demonstrate their functionality.

Speaking of documentation, it has been improved a lot. The usage chapter has been rewritten almost completely; it has gained a couple of tutorials and all the platform-specific details in it have been moved to two new chapters. One of them focuses on explaining those features available only under a specific operating system while the other summarizes multiple portability issues that may arise when using the generic classes. Additionally, a chapter about supported systems and compilers has been added.

There are still two big things missing that shall be worked on in the (very) short term: add a design decisions chapter to the documentation and incorporate asynchronous functionality to the library by using Boost.Asio. This last thing is needed to keep things simple from the user 's point of view (i.e. no threads on his code).

Check out the official announcement for more details.

I guess that this closes SoC for me this year. There are still going to be some changes before Monday but don't expect anything spectacular (I'll be away during the weekend... hopefully ;-). But don't be afraid! Work on this project will continue afterwards!

August 16, 2006 · Tags: boost-process
Continue reading (about 2 minutes)

SoC: Status report 3

Only 8 more days and SoC will be officially over... Time has passed very fast and my project required much more work than I initially thought. It certainly cannot be completed before the deadline but I assure you that it will not fall into oblivion afterwards; I have spent too much time on it to forget ;-)

There have been many changes in Boost.Process' code base since the previous status report; let's see a brief summary:

The library has been simplified removing all those bits that were aimed at "generic process management". Now it is focused on child process managing only, although extending it to support other process-related functionality is still possible (preserving compatibility with the current API). It'll be better to design and implement these features when really needed because they will require a lot of work and cannot be planned right now; doing so might result in an incomplete and clusmy design. Yup... my mentor (Jeff Garland) was right when he suggested to go this simplified route at the very beginning!
Due to the above simplifications, some classes are not templated any more (the stuff that depended on the template parameters is now gone). I bet some of them could still be, but this can be easily changed later on.
There is now a specialized launcher in the library to painlessly start command pipelines. This also comes with a helper process group class to treat the set of processes as a unique entity.
The user now has much more flexibility to specify how a child process' channels behave. While documenting the previous API it became clear that it was incomplete and hard to understand.
Code from all launchers has been unified in a base private class to avoid duplication and ensure consistency across those classes. Similar changes have ocurred in the test suite, which helped in catching some obscure problems.
Related to previous, many of the code used to do the actual task of spawning a process has been moved out of the individual launcher classes into some generic private functions. This was done to share more code, improve cohesion and readability.
The documentation is now much better, although it still lacks a chapter about design issues. See the online snapshot for more details.
And, of course, multiple bug fixes and cleanups.

Still, I haven't had a chance to ask for a public review in Boost's developers mailing list. The problem is that I continously find things to improve or to complete and prefer to do them before asking for the review. However, as time is running out I'll be forced to do this in the forthcoming week to get some more feedback in time.

August 12, 2006 · Tags: boost-process
Continue reading (about 3 minutes)

SoC: Playing with Doxygen

My Boost.Process prototype is almost feature complete; the major thing that is still not included is the ability to create pipelines. I should address that as soon as possible because I'm afraid it will have some impact on the existing classes, but for now I wanted to start documenting some code. There are already 21 header files to document and doing so is by no means an easy task.

In order to document the library's API I've decided to use Doxygen, a documentation system for multiple languages including, obviously, C++. Doxygen scans your source files looking for special comments that document classes, methods and any other part of the code. Then, the comments are extracted alongside with the code structure and are used to automatically generate reference documentation in whichever format you want (HTML, LaTeX, XML, etc.).

Doxygen is widely used and nicely integrated within Boost.Build. Boost's build system automatically generates the required configuration file for Doxygen (the Doxyfile) and lets you merge the resuting files with your other BoostBook (or QuickBook) documents painlessly.

So far I like this tool very much. Keeping the documentation alongside the source code helps in keeping it consistent and makes it immediately available to the curious developer reading the code. Furthermore, it provides tags to format anything you can imagine: preconditions, postconditions, thrown exceptions, results, etc.

The results? Take a look at the Reference section in the Boost.Process' manual ;-) At the moment of this writing only the classes in the detail subdirectory are documented, which correspond to sections 5.10 through 5.13.

July 26, 2006 · Tags: boost-process
Continue reading (about 2 minutes)

SoC: Status report 2

Another week has passed and I'm happy to announce that the Boost.Process prototype is now completely ported to the Win32 API. In other words, a program can use the current library to transparently manage child processes both from Windows and Unix systems.

There are still several rough edges and incomplete classes but the code passes the test suite on both systems :-) OK, you know that passing a test suite does not mean that the code is correct; it only means that it complies with the existing tests. So... more tests are needed to spot the existing failures.

I'm now going to clean up some parts of the code that make little sense after the huge rototill to port the code to Win32; basically, the internal pipe class and its usage. Then, I'll try to complete the missing Unix-specific bits.

Why did I say a "huge rototill"? After starting to port some code to Windows, I discovered the CRT library. For a moment, I thought that the porting could be easy, given that this supports the standard POSIX file descriptors and calls (open(2), read(2), etc.). Unfortunately, I quickly realized that using the CRT could not integrate well with the native Win32 API; and worse, I discovered that Windows only supports communicating with child processes through the three standard channels (stdin, stdout and stderr). This restriction has forced me to redo most of the existing design and code to offer a clean and common interface on both platforms; file descriptors are now hidden all around unless you explicitly want to see them.

Of course this means that the classes used to launch child processes now only accept these three channels, something that is not powerful enough in a Unix system. In these OSes, processes may need to set up extra communcation pipes with children to retrieve additional information (dbus and GPG come to my mind), so there shall be POSIX-specific classes that allow this interface.

I would like to finish the clean up and the addition of POSIX-specific code by the end of the month alongside some simple documentation (formal code examples). The idea is to be able to publish it for informal review soon afterwards (beginning of August).

July 21, 2006 · Tags: boost-process
Continue reading (about 2 minutes)

SoC: Status report

Mmm... SoC. Multiple things have been going on lately in my SoC project, yet I've kept you uninformed. As I already told you, my project aims to develop a C++ library for the Boost project to manage child processes; it's named Boost.Process.

During June I discussed with Jeff Garland — my mentor — the general design of the library. The design is surely not final but it is a lot better than it was at its first sketches. For example: it makes use of templates where appropriate to let the user replace any part of the library with his own code (more or less). I must say he has been very patient with all my questions and has provided extremely valuable information.

I also seized that month to investigate a bit the Win32 API because the library must work on Windows too. I couldn't do much more during that time because I was busy with semester's final exams. All passed by the way :-)

And now to the interesting thing. I've spent the past week (almost two) implementing a preliminary prototype. It is still incomplete but has already raised many issues; you know, it is hard to get into the details (that affect the design) without coding a bit. The prototype also includes multiple unit tests to ensure that everything works, as it shall be; Boost's Unit Test Framework is a really nice tool to implement them.

Browse the source code for more details.

July 13, 2006 · Tags: boost-process
Continue reading (about 2 minutes)

Win32: Mappings for Unicode support

I have spent some time during the past few days to play with the Win32 API again after a year since first looking at them. I must learn how to manage processes under Windows as part of my SoC project, Boost.Process, and this involves native Windows programming with the Win32 API.

After creating a fresh C++ Win32 console application project from Visual Studio 2005, I noticed that the template code had a _tmain function rather than a main one. I did not pay much attention to it until I looked at some code examples that deal with the CreateProcess call: they use weirdly named functions such as _tcsdup and types as _TCHAR instead of the traditional strdup and char * respectively. I could not resist to learn why they did this.

Spending some time searching and reading the MSDN documentation answered my question. These functions and types are wrappers around the standard objects: the functions and types they really point to depend on whether you define the _UNICODE macro during the build or not.

As you can easily guess, defining _UNICODE maps those routines and types to entities that can handle Unicode strings, effectively making your application Unicode-aware. Similarly, if you do not define the symbol, the application remains SBCS/MBCS compatible (the distinction between these two also depends on another macro, as far as I can tell). And because all these redirections are handled by the preprocessor, there is no run-time overhead.

For example: the _tmain function is mapped to the traditional main subroutine if and only if _UNICODE is undefined while it is mapped to wmain otherwise. The latter takes wide-character argv and envp pointers in contrast to the former.

I do not know to which extent this macro is supported by the standard libraries, although I bet almost everything supports it; I have seen many other functions taking advantage of this redirection. In the specific domain I am analyzing, there are two implementations for CreateProcess: CreateProcessW, the Unicode version; and CreateProcessA, the ANSI one.

OK, my knowledge about internationalization is very limited, and I do not know if this feature is very useful or not, but it seems quite interesting at the very least.

See Routine Mappings (CRT) and main: Program Startup (C++) for more details.

Edit (17:24): Changed MFC references to Win32. Thanks to Jason for pointing out the difference between the two in one of the comments. I am in fact investigating the latter.

June 14, 2006 · Tags: boost-process
Continue reading (about 2 minutes)

SoC: Accepted, again!

I am very proud to annouce that I have been accepted into Google's Summer of Code program — again! During Summer 2005 I developed an efficient, memory-based file system for the NetBSD operating system, baptized tmpfs. I must confess that I enjoyed very much hacking the NetBSD kernel and also learned a lot about file systems.

So this year I was eager to repeat the experience by taking part in SoC again. In order to ensure my participation, I thoroughly prepared three applications for three different projects. I had a hard time making the choices because there were tons of interesting projects (which could have taught me very different skills), but at last decided for the following:

Application 1: Add complete NetBSD support to GRUB 2. I chose this project because I knew I could do it, but mostly because I wanted to ensure that GRUB 2 had first-class support for BSD operating systems. GRUB currently lacks features to correctly boot these, which is a nuisance. After sending the application, I was quickly contacted by a GRUB developer telling me that there was another student willing to work on this project, and that he did not knew what to do. I offered to leave my spot to the other developer, hoping to capture another potential NetBSD hacker.
Application 2: Develop a process management library for Boost. This is something I have had in mind since February 2005, when I first discovered Boost. I was planning to do this as my final computer science degree next Spring, but applied for it now so that I could free myself from this idea. (I have other projects in mind that are currently blocked by the lack of Boost.Process.)
Application 3: Improve NetBSD's regression testing framework. After looking at the list of suggested projects for NetBSD and evaluating them, I felt that working on this area could have been very useful for the project, improving its overall quality; I like and enjoy writing software that is able to test itself as much as possible.

The decision between the Boost and NetBSD projects was quite hard to make, so I opted to send both in and let luck decide. Strictly speaking, I stated in my applications that I preferred to do the Boost project for several reasons and I guess Google made their choice based on that. But do not get me wrong: I enjoyed very much the time I spent hacking NetBSD, and I hope to continue doing so in the near future.

Summarizing: I will be developing the Boost.Process library this summer! See the links above for more some information: the wiki page holds some ideas about its design and the application lists my reasons to want to work on this.

I now feel sorry for not "being able" to work on NetBSD's regression testing framework. I do not know if anybody else has been picked to work on it, but if not, this project seems doomed... It was choosen past year but the student abandoned it half-way. This year it was also choosen by NetBSD but Google preferred me to work on Boost. However... while writing the application, my interest in this project raised, which means that I may retake it in the future if nobody else does; maybe as my CS final degree project? :-)

Now... stay tuned for further news on Boost.Process!

May 24, 2006 · Tags: boost-process
Continue reading (about 3 minutes)

MFC: Developing for Windows

During the past two days, I've been working (again) on my Boost Process library idea. While doing so, I realized that I don't know anything at all about coding for Windows using the MFC. I must learn how to handle processes under this platform to be able to design a correct abstraction layer for process management.

It's time to do so. I booted Windows XP, downloaded Borland's C++ Builder 5 command line tools (that is, the C++ free compiler) and installed it following the instructions (not a "trivial" task). Man, it's damn fast compared to GNU g++, as seen while building some Boost's code.

June 20, 2005 · Tags: boost-process
Continue reading (about 2 minutes)