GSoC 2014 idea: Port FreeBSD's old-style tests to ATF
Are you a student interested in contributing to a production-quality operating system by increasing its overall quality? If so, you have come to the right place! As you may already know, the Google Summer of Code 2014 program is on and FreeBSD has been accepted as a mentoring organization. As it so happens, I have a project idea that may sound interesting to you. During the last few months, we have been hard at work adding a standardized test suite to the FreeBSD upstream source tree as described in the TestSuite project page. However, a test suite is of no use if it lacks a comprehensive collection of tests!
March 12, 2014
·
Tags:
<a href="/tags/atf">atf</a>, <a href="/tags/freebsd">freebsd</a>, <a href="/tags/soc">soc</a>, <a href="/tags/testing">testing</a>
Continue reading (about
3 minutes)
NetBSD in Google Summer of Code 2010
For the 6th year in a row, NetBSD is a mentoring organization for Google Summer of Code 2010! If you are a bright student willing to develop full-time for an open source project during this coming summer, consider applying with us! You will have a chance to work with very smart people and, most likely, in the area that you are most passionate about. NetBSD, being an operating system project, has offers for project ideas at all levels: from the kernel to the packaging system, passing by drivers, networking tools, user-space utilities, the system installer, automation tools and more! I would like to point you at the 3 project proposals I'm willing to directly mentor: Optimize and speed-up ATF: Make the testing framework blazing fast so that running the NetBSD automated tests does not take ages on slow platforms. Reorganize ATF to improve modularity: Refactor pieces of the testing framework so that it is easier to redistribute, has cleaner interfaces and is easier to depend on from third-party projects. Rewrite pkg_comp with portability as a major goal: Use Python to create a tool to automatically build binary packages from within a sandbox.If you find any of the above projects interesting, or if you have any other project proposal that you think I could mentor, do not hesitate to contact me. Feel free to send me a draft of your application, together with a bit of information about you, so that we can discuss your proposal and make sure it gets selected! Or, if none of the projects above interests you, please do check out the full list of NetBSD project proposals. I'm sure you will find something that suits your interests :-)
March 19, 2010
·
Tags:
<a href="/tags/atf">atf</a>, <a href="/tags/netbsd">netbsd</a>, <a href="/tags/pkgsrc">pkgsrc</a>, <a href="/tags/soc">soc</a>
Continue reading (about
2 minutes)
NetBSD-SoC needs your application!
The Google Summer of Code 2009 application deadline for students is tomorrow and NetBSD has got very few applications so far. If you have the interest in working on a cool operating system project, where almost any project idea can fit, take the time to read our proposals and apply! New, original ideas not listed there will also be considered. It'd be a pity if the number of assigned slots to NetBSD was small due to the low number of applications! We did much better past year. Note that there are a couple of ATF-related proposals in there. Help will be certainly welcome (by me ;-) in those areas!
April 2, 2009
·
Tags:
<a href="/tags/netbsd">netbsd</a>, <a href="/tags/soc">soc</a>
Continue reading (about
1 minute)
ATF 0.6 released
I am very happy to announce the availability of the 0.6 release of ATF. I have to apologize for this taking so long because the code has been mostly-ready for a while. However, doing the actual release procedure is painful. Testing the code in many different configurations to make sure it works, preparing the release files, uploading them, announcing the new release on multiple sites... not something I like doing often. Doing some late reviews, I have to admit that the code has some rough edges, but these could not delay 0.6 any more. The reason is that this release will unblock the NetBSD-SoC atfify project, making it possible to finally integrate all the work done in it into the main NetBSD source tree. Explicit thanks go to Lukasz Strzygowski. He was not supposed to contribute to ATF during his Summer of Code 2008 project, but he did, and he actually provided very valuable code. The next step is to update the NetBSD source tree to ATF 0.6. I have extensive local changes for this in my working copy, but I'm very tired at the moment. I think I'll postpone their commit until tomorrow so that I don't screw up something badly. Enjoy it and I'm looking for your feedback on the new stuff!
January 18, 2009
·
Tags:
<a href="/tags/atf">atf</a>, <a href="/tags/netbsd">netbsd</a>, <a href="/tags/soc">soc</a>
Continue reading (about
2 minutes)
SoC 2008 Mentor Summit
The Google SoC 2008 Mentor Summit is now officially over. The summit has taken place during the whole weekend and has been pretty intensive. The organization of the whole event has been excellent thanks to the hard work of Leslie Hawthorn among others; sorry, can't remember your names... I'm very bad at this. We have had multiple sessions, ranging from technical ones such as distributed version control systems to more political ones such as how to deal with assholes in open source projects. There were lots of passionate people in these talks, and it was quite interesting to see it all. The cool thing, though, as opposed to other conferences, is that everyone here comes from a different project and background, so you get to see lots of different opinions and points of views for each topic. As regards the Google HQ campus, it is great. I thought the NYC offices were good, but these are spectacular to see. Unfortunately, there is not much to do out of them... so I'm not sure if it could be so good to work here for a long time. Now, I'm sitting in the San Jose airport (SJC) waiting for the flight back to NYC. Amazingly, there is free internet wireless connection and electrical plugs! Very, very nice detail. And there is few people around, which makes it very quiet and relaxed. Oh, and to those who will decipher this: some more mini golf training during the summit :P
October 26, 2008
·
Tags:
<a href="/tags/google">google</a>, <a href="/tags/soc">soc</a>
Continue reading (about
2 minutes)
Live@NYC: ... or not; now in MTV!
I've landed this morning in San Francisco at 9.00am (which means I left NYC at 6.00am!) and went straight down to the Google Headquarters in Mountain View. No sleep at all except for a little bit of pseudo-sleep in the plane. The Google campus is really nice. It puts the NYC offices in an inferior level than I thought :P But the only problem is that the area surrounding the campus is basically empty. Very small houses and lots of space between them, which is not bad per se... but means that there really is not much to do. Anyway. What am I doing here? I am attending the Google Summer of Code 2008 Mentors Summit this weekend, but came a bit earlier to be able to do a couple of meetings with coworkers in the Mountain View office. Pretty exhausting day, and it is not close to over yet! Just enjoy the few photos I've taken so far. PS: Been playing mini-golf on-board until I got an unasked segmentation fault.
October 24, 2008
·
Tags:
<a href="/tags/google">google</a>, <a href="/tags/nyc">nyc</a>, <a href="/tags/soc">soc</a>
Continue reading (about
1 minute)
Google Summer of Code 2008 and NetBSD
Google has launched the Summer of Code program once again this year, and NetBSD is a mentoring organization for the fourth time as announced in a netbsd-announce post. Unless things go very wrong in the following days, I will not take part this year as a student because I will be intering at Google SRE during the Summer! However, I will try to become a mentor for the "Convert all remaining regression tests to ATF" project. If you are looking for some interesting idea to apply for, this is a good one! Why? It will let you get into NetBSD internals in almost all areas of the system: you'll need to understand how the source tree is organized, how to add new components to it (because tests are almost in all aspects regular programs), how the current pieces of the system interact with each other... You will need to gain knowledge in some areas (such as the kernel or the libraries) to be able to port tests from the old framework (if it deserves that name ;-) to the new one and, if you are really up to it, even add new tests for functionality that is currently uncovered by the test suite. But adding new tests is something you will not be required to do, because the sole task of migrating the existing ones is a huge task already. Get involved in ATF's development because, as you study the existing test cases and their requirements, you will most likely find that it lacks some important functionality to make things really straightforward.And, of course, make a unvaluable contribution to the NetBSD operating system. Having a public test suite with high coverage means that the system will gain quality. Yes, you will most likely uncover bugs in many areas of the system and give them enough exposure so that someone else may fix them.Note that this project is really a Summer of Code project. It does not have a long design phase on its own so, once you have got used to the system and ATF, you'll just code and immediately make useful contributions. In the past, projects that had a heavy design phase involved were not good because, in the end, the student did not finish the code on time. So... don't hesitate to apply! I'm looking forward to see your applications for this project :-)
March 19, 2008
·
Tags:
<a href="/tags/atf">atf</a>, <a href="/tags/netbsd">netbsd</a>, <a href="/tags/soc">soc</a>
Continue reading (about
2 minutes)
SoC: Second preview of NetBSD with ATF
Reposting from the original ATF news entry: I have just updated the first preview of NetBSD-current release builds with ATF merged in to match the ATF 0.1 release published today. As already stated in the old news item: These will ease testing to the casual user who is interested in this project because he will not need to mess with patches to the NetBSD source tree nor rebuild a full release, which is a delicate and slow process. For the best experience, these releases are meant to be installed from scratch even though you can also do an upgrade of a current installation. They will give you a preview of how a NetBSD installation will look like once ATF is imported into it; we are not sure when that will happen, though.
August 20, 2007
·
Tags:
<a href="/tags/atf">atf</a>, <a href="/tags/netbsd">netbsd</a>, <a href="/tags/soc">soc</a>
Continue reading (about
1 minute)
SoC: Some statistics
Here go some statistics about what has been done during the SoC 2007 program as regards ATF: The repository weights at 293 revisions, 1,174 certificates (typically 4 per revision, but some revisions have more) and 221 files. This includes ATF, the patches to merge it into the NetBSD build tree and the website sources. (mtn db info will give you some more interesting details.) The clean sources of ATF 0.1 (not counting the files generated by the GNU autotools) take 948Kb and are 20,607 lines long (wow!). This includes the source code, the manual pages, the tests and all other files included in the distribution. The patches to merge ATF into NetBSD, according to diffstat, change 209 files, do 6,299 line insertions and 4,583 line deletions. Aside merging ATF into NetBSD, these changes also convert multiple existing regression tests to the new framework. As regards the time I have spent on it... I don't know, but it has been a lot. It should have been more as I had to postpone the start of coding some weeks due to university work, but I think the results are quite successful and according to the expectations. I have been able to cover all the requirements listed in the NetBSD-SoC project page and done some work on the would-be-nice ones. I am eager to see the results of the other NetBSD-SoC 2007 projects as there was very interesting stuff in them :-)
August 20, 2007
·
Tags:
<a href="/tags/atf">atf</a>, <a href="/tags/soc">soc</a>
Continue reading (about
2 minutes)
SoC: ATF 0.1 released
To conclude the development of ATF as part of SoC, I've released a 0.1 version coinciding with the coding deadline (later today). This clearly draws a line between what has been done during the SoC program and what will be done afterwards. See the official announcement for more details! I hope you enjoy it as much as I did working on it.
August 20, 2007
·
Tags:
<a href="/tags/atf">atf</a>, <a href="/tags/soc">soc</a>
Continue reading (about
1 minute)
SoC: Status report
SoC's deadline is just five days away! I'm quite happy with the status of my project, ATF, but it will require a lot more work to be in a decent shape — i.e. ready to be imported into NetBSD — and there really is no time to get it done in five days. Furthermore, it is still to unstable (in the sense that it changes a lot) so importing it right now could cause a lot of grief to end users. However, after a couple of important changes, it may be ready for a 0.1 release, and that's what I'm aiming for. I have to confess again that some parts of the code are horrible. That's basically because it has been gaining features in an iterative way, all which were not planned beforehand... so it has ended up being hack over hack. But, don't worry: as long as there is good test coverage for all the expected features, this can easily be fixed. With a decent test suite, I'll be able to later rewrite any piece of code and be pretty sure that I have not broken anything important. (In fact, I've already been doing that for the most inner code with nice results.) So what has changed since the preview? All files read by ATF as well as all data formats used for serialization of data have now a header that specifies their format (a type and a version). This is very important to have from the very beginning so that the data formats can easily be changed in the future (which will certainly happen).Rewrote how test programs and atf-run print their execution status. Now, the two print a format that is machine-parseable and which is "sequential": reading the output from top to bottom, you can immediately know what the program is doing at the moment without having to wait for future data.Added the atf-report tool, which gathers the output of atf-run and generates a user-friendly report. At the moment it outputs plain text only, but XML (and maybe HTML) are planned. The previous point was a pre-requisite for this one.Merged multiple implementation files into more generic modules.Merged the libatf and libatfprivate libraries into a single one. The simpler the better.Added build-time tests for all public headers, to ensure that they can be included without errors.Implemented run-time configuration variables for test programs and configuration files.Wow, that's a lot of stuff :-) And talking with my mentor five days ago, we got to the following list of pending work to get done before the deadline: Configuration files. Already done as of an hour ago!A plain text format that clearly describes the results of the test cases (similar to what src/regress/README explains). I haven't looked at that yet, but this will be trivial with the new atf-report tool.Would be nice: HTML output. Rather easy. But I'm unsure about this point: it may be better to define a XML format only and then use xsltproc to transform it.Manual pages: A must for 0.1 (even if they are not too detailed), but not really required for the evaluation.Code cleanups: Can be done after SoC, but I personally dislike showing ugly code. Unfortunately there is not enough time to spend on this. Cleaning up a module means: rewriting most of it, documenting each function/class and adding exhaustive unit tests for it. It is painful, really, but the results are rewarding. Keep the NetBSD patches in sync with development: I'm continuously doing that!Let's get back to work.
August 15, 2007
·
Tags:
<a href="/tags/atf">atf</a>, <a href="/tags/soc">soc</a>
Continue reading (about
3 minutes)
SoC: First preview of NetBSD with ATF
Reposting from the original ATF news entry: I have just uploaded some NetBSD-current release builds with ATF merged in. These will ease testing to the casual user who is interested in this project because he will not need to mess with patches to the NetBSD source tree nor rebuild a full release, which is a delicate and slow process. For the best experience, these releases are meant to be installed from scratch even though you can also do an upgrade of a current installation. They will give you a preview of how a NetBSD installation will look like once ATF 0.1 is made public, which should happen later this month. For more details see my post to the NetBSD's current-users mailing list.Waiting for your feedback :-) Edit (Aug 20th): Fixed a link.
August 8, 2007
·
Tags:
<a href="/tags/atf">atf</a>, <a href="/tags/netbsd">netbsd</a>, <a href="/tags/soc">soc</a>
Continue reading (about
1 minute)
SoC: Status report
It has already been a week since the last SoC-related post, so I owe you an status report. Development has continued at a constant rate and, despite I work a lot on the project, it may seem to advance slowly from an external point of view. The thing is that getting the ATF core components complete and right is a tough job! Just look at the current and incomplete TODO list to see what I mean. Some things worth to note: The NetBSD cross-build tool-chain no longer requires a C++ compiler to build the atf-compile host tool. I wrote a simplified version in POSIX shell to be used as the host tool alone (not to be installed). This is also used by the ATF's distfile to allow "cross-building" its own test programs. Improved the cleanup procedure of the test case's work directories by handling mount points in them. This is done through a new tool called atf-cleanup.Added a property to allow test cases specify if they require root privileges or not. Many bug fixes, cleanups and new test cases; these are driving development right now.On the NetBSD front, there have also been several cosmetic improvements and bug fixes, but most importantly I've converted the tmpfs' test suite to ATF. This conversion is what has spotted many bugs and missing features in ATF's code. The TODO file has grown basically due to this. So, at the moment, both the regress/bin and regress/sys/fs/tmpfs trees in NetBSD have been converted to ATF. I think that's enough for now and that I should focus on adding the necessary features to ATF to improve these tests. One of these is to add support for a configuration file to let the user specify how certain tests should behave; e.g. how to become root or which specific file system to use for certain tests. I also have a partial implementation to add a "fork" property to test cases to execute them in subprocesses. This way they will be able to mess all they want with the open file descriptors without disturbing the main test program. But to get here, I first need to clean up the reporting of test case's results. On the other hand, I also started preparing manual pages for the user tools as some of them should remain fairly stable at this point.
July 28, 2007
·
Tags:
<a href="/tags/atf">atf</a>, <a href="/tags/soc">soc</a>, <a href="/tags/tmpfs">tmpfs</a>
Continue reading (about
2 minutes)
SoC: ATF self-testing
ATF is a program, and as happens with any application, it must be (automatically) tested to ensure it works according to its specifications. But as you already know, ATF is a testing framework so... is it possible to automatically test it? Can it test itself? Should it do it? The thing is: it can and it should, but things are not so simple. ATF can test itself because it is possible to define test programs through ATF to check the ATF tools and libraries. ATF should test itself because the resulting test suite will be a great source of example code and because its execution will be on its own a good stress test for the framework. See the tests/atf directory to check what I mean; specially, the unit tests for the fs module, which I've just committed, are quite nice :-) (For the record: there currently are 14 test programs in that directory, which account for a total of 60 test cases.) However, ATF should not be tested exclusively by means of itself. If it did so, any failure (even the most trivial one) in the ATF's code could result in false positives or false negatives during the execution of the test suite, leading to wrong results hard to discover and diagnose. Imagine, for example, that a subtle bug made the reporting of test failures to appear as test passes. All tests could start to succeed immediately and nobody could easily notice, surely leading to errors in further modifications. This is why a bootstrapping test suite is required: one that ensures that the most basic functionality of ATF works as expected, but which does not use ATF to run itself. This additional test suite is already present in the source tree, and is written using GNU Autotest, given that I'm using the GNU Autotools as the build system. Check the tests/bootstrap directory to see what all this is about. ATF's self-testing is, probably, the hardest thing I've encountered in this project so far. It is quite tricky and complex to get right, but it's cool! Despite being hard, having a complete test suite for ATF is a must so it cannot be left aside. Would you trust a testing framework if you could not quickly check that it worked as advertised? I couldn't ;-)
July 20, 2007
·
Tags:
<a href="/tags/atf">atf</a>, <a href="/tags/soc">soc</a>
Continue reading (about
2 minutes)
SoC: Web site for ATF
While waiting for a NetBSD release build to finish, I've prepared the web site for ATF. It currently lacks information in a lot of areas, but the most important ones for now — the RSS feed for news and the Repository page — are quite complete. Hope you like it! Comments welcome, of course :-)
July 16, 2007
·
Tags:
<a href="/tags/atf">atf</a>, <a href="/tags/soc">soc</a>
Continue reading (about
1 minute)
SoC: Converting NetBSD 'regress' tests
I've finally got at a point where I can start converting some of the current regression tests in the NetBSD tree to use the new ATF system. To prove this point, I've migrated all the tests that currently live in regress/bin to the new framework. They all now live in /usr/tests/util/. This has not been a trivial task — and it is not completely done yet, as there still are some rough edges — but I'm quite happy with the results. They show me that I'm on the right track :-) and, more importantly, they show outsiders how things are supposed to work. If you want more information on this specific change you can look at the revision that implements it or simply inspect the corresponding patch file. By the way, some of the tests already fail! That's because they were not run often enough in the past, something that ATF is supposed to resolve. While waiting for a NetBSD release build to complete, I have started working on a real web site for ATF. Don't know if I'll keep working on it now because it's a tough job and there is still a lot of coding to do. Really! But, on the other hand, having a nice project page is very good marketing.
July 15, 2007
·
Tags:
<a href="/tags/atf">atf</a>, <a href="/tags/soc">soc</a>
Continue reading (about
2 minutes)
Book: Producing Open Source Software
This year, Google sent all the Summer of Code students the Producing Open Source Software: How to run a successful free software project book by Karl Fogel (ISBN 0-596-00759-0) as a welcome present. I've just finished reading it and I can say that it was a very nice read. The book is very easy to follow and is very complete: it covers areas such as the project's start-up, how to set things up for promoting it, how to behave in mailing lists, how to prepare releases, how to deal with volunteers or with paid developers, etc. Everything you need to drive your project correctly and without gaining much enemies. While many of the things stated in the book are obvious to anyone who has been in the open source world for a while (and already started a project on its own or contributed to an existing one), it is still a worthy read. I wish all the people involved in NetBSD (some more than others) read it and applied the suggestions given there. We'd certainly improve in many key areas and reduce pointless (or better said, unpleasant) discussions! Oh, and by the way: you can read the book online at its web page, as it is licensed under a Creative Commons Attribution-ShareAlike license. Kudos to Karl Fogel.
July 14, 2007
·
Tags:
<a href="/tags/books">books</a>, <a href="/tags/oss">oss</a>, <a href="/tags/soc">soc</a>
Continue reading (about
2 minutes)
SoC: Code is public now
Just in time for the mid-term evaluation (well, with one day of delay), I've made the atf's source code public. This is possible thanks to the public and free monotone server run by Timothy Brownawell. It's nice to stay away from CVS ;-) See the How to get it section at the atf's page for more details on how to download the code from that server and how to trust it (it may take an hour or two for the section to appear). You can also go straight to the source code browser. If, by any chance, you decide to download the code, be sure to read the README files as they contain very important information. And... don't start nitpicking yet! I haven't had a chance to clean up the code yet, and some parts of it really suck. Cleaning it up is the next thing I'll be doing, and I already started with the shell library :-)
July 10, 2007
·
Tags:
<a href="/tags/atf">atf</a>, <a href="/tags/soc">soc</a>
Continue reading (about
1 minute)
SoC: Short-term planning
SoC 2007's mid-term evaluation is around the corner. I have to present some code on June 9th. In fact, it'd be already public if we used a decent VCS instead of CVS, but for now I'll keep the sources in a local monotone database. We'll see how they'll be made public later on. Summarizing what has been done so far: I've already got a working prototype of the atf core functionality. I have to confess that the code is still very ugly (and with that I mean you really don't want to see it!) and that it is incomplete in many areas, but it is good enough as a "proof of concept" and a base for further incremental improvement. What it currently allows me to do is: Write test programs (a collection of test cases) in C. In reality it is C++ as we already saw, but I've added several macros to hide the internals of the library and simplify the definition of test cases. So basically the test writer will not need to know that he's using C++ under the hood. (This is also to mimic as much as possible the shell-based interface.)Write test programs in POSIX shell. Similar as above, but for tests written using shell scripts. (Which I think will be the majority.)Define a "tree of tests" in the file system and recursively run them all, collecting the results in a single log. This can be done without the source tree nor the build tools (in special make), and by the end user.Wrote many tests to test atf itself. More on this on tomorrow's post. What I'm planning to do now, before the mid-term evaluation deadline, is to integrate the current code into the NetBSD's build tree (not talking about adding it to the official CVS yet though) to show how all these ideas are applicable to NetBSD testing, and to ensure everything can work with build.sh and cross-compilation. Once this is done, which shouldn't take very long I hope, I will start polishing the current atf's core implementation. This means rewriting several parts of the code (in special to make it more error-safe), adding more tests, adding manual pages for the tools and the interfaces, etc. This is something I'm willing to do, even though it'll be a hard and long job.
July 2, 2007
·
Tags:
<a href="/tags/atf">atf</a>, <a href="/tags/soc">soc</a>
Continue reading (about
2 minutes)
SoC: The atf-run tool
One of the key goals of atf is to let the end user — not only the developer — to easily run the tests after their corresponding application is installed. (In our case, application = NetBSD, but remember that atf also aims to be independent from NetBSD.) This also means, among other things, that the user must not need to have any development tool installed (the comp.tgz set) to run the tests, which rules out using make(1)... how glad I'm of that! :-) Based on this idea, each application using atf will install its tests alongside its binaries, being the currently location /usr/tests/<application>. These tests will be accompanied by a control file — an Atffile — that lists which tests have to be run and in which order. (In the future this may also include configuration or some other settings.) Later on, the user will be able to launch the atf-run tool inside any of these directories to automatically run all the provided tests, and the tool will generate a pretty report while the tests are run. Given that atf is an application, it has to be tested. After some work today, it is finally possible for atf to test itself! :-) Of course, it also comes with several bootstrap tests, written using GNU Autotest, to ensure that atf's core functionality works before one can run the tests written using atf itself. Otherwise one could get unexpected passes due to bugs in the atf code. This is what atf installs:$ find /tmp/local/tests /tmp/local/tests /tmp/local/tests/atf /tmp/local/tests/atf/Atffile /tmp/local/tests/atf/units /tmp/local/tests/atf/units/Atffile /tmp/local/tests/atf/units/t_file_handle /tmp/local/tests/atf/units/t_filesystem /tmp/local/tests/atf/units/t_pipe /tmp/local/tests/atf/units/t_pistream /tmp/local/tests/atf/units/t_postream /tmp/local/tests/atf/units/t_systembuf $All the t_* files are test programs written using the features provided by libatf. As you can see, each directory provides an Atffile which lists the tests to run and the directories to descend into. The atf-run tool already works (*cough* it's code is ugly, really ugly) and returns an appropriate error code depending on the outcomes of the tests. However, the report it generates is completely un-understandable. This will be the next thing to attack: I want to be able to generate plain-text reports to be seen as the text progresses, but also to generate pretty HTML files. To do the latter, the plan is to use some intermediate format such as XML and have another tool to do the formatting.
June 30, 2007
·
Tags:
<a href="/tags/atf">atf</a>, <a href="/tags/soc">soc</a>
Continue reading (about
2 minutes)
SoC: Prototypes for basename and dirname
Today, I've attempted to build atf on a NetBSD 4.0_BETA2 system I've been setting up in a spare box I had around, as opposed to the Mac OS X system I'm using for daily development. The build failed due to some well-understood problems, but there was an annoying one with respect to some calls to the standard XPG basename(3) and dirname(3) functions. According to the Mac OS X manual pages for those functions, they are supposed to take a const char * argument. However, the NetBSD versions of these functions take a plain char * parameter instead — i.e., not a constant pointer. After Googling for some references and with advice from joerg@, I got the answer: it turns out that the XPG versions1 of basename and dirname can modify the input string by trimming trailing directory separators (even though the current implementation in NetBSD does not do that). This makes no sense to me, but it's what the XPG4.2 and POSIX.1 standards specify. I've resolved this issue by simply re-implementing basename and dirname (which I wanted to do anyway), making my own versions take and return constant strings. And to make things safer, I've added a check to the configure script that verifies if the basename and dirname implementations take a constant pointer and, in that (incorrect) case, use the standard functions to sanity-check the results of my own by means of an assertion. 1 How not, the GNU libc library provides its own variations of basename and dirname. However, including libgen.h forces the usage of the XPG versions.
June 28, 2007
·
Tags:
<a href="/tags/atf">atf</a>, <a href="/tags/portability">portability</a>, <a href="/tags/soc">soc</a>
Continue reading (about
2 minutes)
SoC: Start of the atf tools
Aside from the libraries I already mentioned in a past post, atf1 will also provide several tools to run the tests. An interesting part of the problem, though, is that many tests will be written in the POSIX shell language as that will be much easier than writing them in C or C++: the ability to rapidly prototype tests is a fundamental design goal; otherwise nobody could write them! However, providing two interfaces to the same framework (one in POSIX shell and one in C++) means that there could be a lot of code duplication in the two if not done properly. Not to mention that sanely and safely implementing some of these features in shell scripting could be painful. In order to resolve the above problem, the atf will also provide several binary tools that will be helpers for the shell scripts. Most of these tools will be installed in the libexec directory as they should not be exposed to the user, yet the shell scripts will need to be able to reach them. The key idea will be to later build the shell interface on top of the binary one, reusing as much code as possible. So far I have the following tools: atf-config: Used to dynamically query information about atf's installation. This is needed to let the shell scripts locate where the tools in libexec can be found (because they are not in the path!).atf-format: Pretty-prints a message (single- or multi-paragraph), wrapping it on terminal boundaries. atf-identify: Calculates a test program's identifier based on where it is placed in the file system. Tests programs will be organized in a directory hierarchy, and each of them has to have a unique identifier.The next one to write, hopefully, will be atf-run: the tool to effectively execute multiple test programs in a row and collect their results. Oh, and in case you are wondering. Yes, I have decided to provide each tool as an independent binary instead of a big one that wraps them all (such as cvs(1)). This is to keep them as small as possible — so that shell scripts can load them quickly — and because this seems to be more in the traditional Unix philosophy of having tools for doing very specific things :-) 1 Should I spell it atf, ATF or Atf?
June 27, 2007
·
Tags:
<a href="/tags/atf">atf</a>, <a href="/tags/soc">soc</a>
Continue reading (about
2 minutes)
SoC: A quote
I've already spent a bunch of time working on the packaging (as in what will end up in the .tar.gz distribution files) of atf even though it is still in very preliminary stages of development. This involved: Preparing a clean and nice build framework, which due to the tools I'm using meant writing the configure.ac and Makefile.am files. This involved adding some useful comments and, despite I'm familiar with these tools, re-reading part of the GNU Automake and GNU Autoconf manuals; this last step is something that many, many developers bypass and therefore end up with really messy scripts as if they weren't important. (Read: if the package used some other tools, there'd be no reason to not write pretty and clean build files.) Preparing the package's documentation, or at least placeholders for it: I'm referring to the typical AUTHORS, COPYING, NEWS, README and TODO documents that many developers seem to treat as garbage and fill them up at the last minute before flushing out a release, ending with badly formatted texts full of typos. Come on, that's the very first thing a user will see after unpacking a distribution file, so these ought to be pretty!Having spent a lot of time packaging software for pkgsrc and dealing with source code from other projects, I have to say that I've found dozens of packages that do not have the minimum quality one can expect in the above points. I don't like to pinpoint, but I have to: this includes several GNOME packages and the libspe. This last one is fairly "interesting" because the user documentation for it is high-quality, but all the credibility slips away when you look at the source code packaging... To all those authors: “Programs should be written and polished until they acquire publication quality.” — Niklaus Wirth
June 26, 2007
·
Tags:
<a href="/tags/atf">atf</a>, <a href="/tags/packaging">packaging</a>, <a href="/tags/soc">soc</a>
Continue reading (about
2 minutes)
SoC: Project name
The automated testing framework I'm working on is a project idea that has been around for a very long time. Back in SoC 2005, this project was selected but, unfortunately, it was not developed. A that time, the project was named regress, a name that was derived from the current name used in the NetBSD's source tree to group all available tests: the src/regress directory. In my opinion, the "regress" name was not very adequate because regression tests are just a kind of all possible tests: those that detect whether a feature that was supposed to be working has started to malfunction. There are other kinds of tests, such as unit tests, integration tests and stress tests, all of which seemed to be excluded from the project just because of its name. When I wrote my project proposal this year, I tried to avoid the "regression testing" name wherever possible and, instead, simply used the "testing" word to emphasize that the project was not focusing on any specific test type. Based on that, the NetBSD-SoC administrators chose the atf name for my project, which stands for Automated Testing Framework. This is a very simple name,and, even though it cannot be easily pronounced, I don't dislike it: it is short, feels serious and clearly represents what the project is about. And for the sake of completion, let me mention another idea I had for the project's name. Back when I proposed it, I thought it could be named "NetBSD Automated Testing Framework", which could then be shortened to nbatf or natf (very similar to the current name, eh?). Based on the latter name, I thought... the "f" makes it hard to pronounce, so it'd be reduced to "nat", and then, it could be translated to the obvious (to me) person name that derives from it: Natalie. That name stuck on my head for a short while, but it doesn't look too serious for a project name I guess ;-) But now, as the atf won't be tied to NetBSD, that doesn't make much sense anyway.
June 25, 2007
·
Tags:
<a href="/tags/atf">atf</a>, <a href="/tags/soc">soc</a>
Continue reading (about
2 minutes)
SoC: Getting started
This weekend I have finally been able to start coding for my SoC project: the Automated Testing Framework for NetBSD. To my disliking, this has been delayed too much... but I was so busy with my PFC that I couldn't find any other chance to get my hands on it. I've started by working on two core components: libatf: The C/C++ library that will provide the interface to easily write test cases and test suites. Among its features will be the ability to report the results of each test, a way to define meta-data for each test, etc. libatfmain: A library that provides a default entry point for test programs that takes care to run the test cases in a controlled environment — e.g. it captures all signals and deals with them gracefully — and provides a standard command-line interface for them.Soon after I started coding, I realized that I could need to write my own C code to deal with data collections and safe strings. I hate that, because that's a very boring task — it is not related to the problem at hand at all — and because it involves reinventing the wheel: virtually all other languages provide these two features for-free. But wait! NetBSD has a C++ compiler, and atf won't be a host tool1. So... I can take advantage of C++, and I'll try to. Calm down! I'll try to avoid some of the "complex" C++ features as much as possible to keep the executables' size small enough. You know how the binaries' size blows up when using templates... Oh, and by the way: keep in mind that test cases will be typically written in POSIX shell script, so in general you won't need to deal with the C++ interface. Furthermore, I see no reason for atf to be tied to NetBSD. The test cases will surely be, but the framework needn't. Thus I'm thinking of creating a standalone package for atf itself and distributing it as a completely independent project (under the TNF2 umbrella), which will later be imported into the NetBSD source tree as we currently do for other third-party projects such as Postfix. In fact, I've already started work on this direction by creating the typical infrastructure to use the GNU auto-tools. Of course this separation could always be done at a later step in the development, but doing it from the very beginning ensures the code is free of NetBSD-isms, emphasizes the portability desire and keeps the framework self-contained. I'd like to hear your comments about these "decisions" :-) 1 A host tool is a utility that is built with the operating system's native tools instead of with the NetBSD's tool-chain: i.e. host tools are what build.sh tools builds. Such tools need to be highly portable because they have to be accepted by old compilers and bizarre build environments. 2 TNF = The NetBSD Foundation.
June 24, 2007
·
Tags:
<a href="/tags/atf">atf</a>, <a href="/tags/netbsd">netbsd</a>, <a href="/tags/soc">soc</a>
Continue reading (about
3 minutes)
SoC: Selected again!
Yes, Google Summer of Code (SoC) 2007 is back and I'm in once again! This means I'll be able to spend another summer working on free software and deliver some useful contributions by the end of it. This time I sent just one proposal, choosing NetBSD as the mentoring organization. The project is entitled Automated testing framework and is mentored by Martin Husemann. This framework is something I've had in mind for a long time already; in fact, I also applied for this in SoC 2006 and attempted to develop this project as my undergraduate thesis. For more details on what the project is about, check out these notes. At last, take a look at the full list of accepted projects for NetBSD. It is rather short unfortunately, but they all look very promising. It is a pity ext3 support is not in them, but getting ZFS instead will be good too. Edit (April 19th): Fixed a link.
April 12, 2007
·
Tags:
<a href="/tags/netbsd">netbsd</a>, <a href="/tags/soc">soc</a>
Continue reading (about
1 minute)
NetBSD and SoC 2007
Yes, ladies and gentlemen: Google Summer of Code 2007 is here and NetBSD is going to become a mentoring organization again (unless Google rejected the application, that is)! We are preparing a list of projects suitable for SoC; spend some time looking for one that interests you (I'm sure there is something) and get ready to send your proposal between the 14th and 24th of this same month. I've already made my choice :-) See the official announcement for more details.
March 8, 2007
·
Tags:
<a href="/tags/netbsd">netbsd</a>, <a href="/tags/soc">soc</a>
Continue reading (about
1 minute)
What have I learned during SoC?
One of SoC's most important goals is the introduction of students to the free software world; this way there are high chances that they will keep contributing even when SoC is over. Students already familiar with FOSS (as was my case both years) are also allowed to participate because they can seize the Summer to learn new stuff and improve their skills. As I expected, the development of Boost.Process has taught me multiple new things. First of all, I wanted to get familiar with the Win32 API because I knew nothing about it. I have achieved this objective by learning the details about process and file management and making Boost.Process work under this platform. Sincerely, Win32 is overly complex but has some interesting features. Secondly, I have got a lot more fluent with C++ templates and have learned some curious coding techniques that I never thought about in the past. The most impressive one in my opinion is that templates can be used to achieve build time specialization, avoiding expensive virtual tables at run time and inheritance when these are not really needed. (I only considered them for polimorphic containers before.) At last, I have also got into several utilities used for Boost development. Among them are Quickbook for easy document writing, Boost.Build v2 for portable software building and the Boost Unit Test library for painlessly creating automated test suites. All in all I'm happy with the outcome of the project and the new knowledge. If SoC happens again, you should really consider joining if you have the chance!
August 22, 2006
·
Tags:
<a href="/tags/boost-process">boost-process</a>, <a href="/tags/soc">soc</a>
Continue reading (about
2 minutes)
Boost.Process 0.1 published
SoC 2006 is officially over — at least for me in my timezone. Given that the Subversion repository has some problems with public access, I've tagged the current sources as the first public version and uploaded a couple of tarballs to the Boost Vault. Both the tag and the tarballs will also serve historical purposes, specially when newer ones come ;-) You can download the archives from the Process directory in tar.gz and ZIP formats. Enjoy!
August 21, 2006
·
Tags:
<a href="/tags/boost-process">boost-process</a>, <a href="/tags/soc">soc</a>
Continue reading (about
1 minute)
Boost.Process tarballs posted
As everybody is not comfortable accessing Subversion repositories to download source code, I've posted two tarballs with Boost.Process' sources. They include an exported copy of the repository contents as well as prebuilt documentation in the libs/process/doc/html subdirectory. You can download the compressed archive either in tar.gz format or in ZIP. Keep in mind that these will be updated very frequently so please do not use them to prepackage the library. Changes from yesterday's announcement are minor at this point. For the curious ones: there is now a list of pending work and the Last revised item in the main page has been fixed. As a side effect of this last change, Boostbook will support SVN's $Date$ tags if my patch is integrated :-)
August 17, 2006
·
Tags:
<a href="/tags/boost-process">boost-process</a>, <a href="/tags/soc">soc</a>
Continue reading (about
1 minute)
Featured software
Featured posts
- Fast machines, slow machines
- EndBASIC 0.10: Core language, evolved
- Farewell, Microsoft; hello, Snowflake!
- Rust is hard, yes, but does it matter?
- Rust traits and dependency injection
- A year on Windows: Introduction
- Always be quitting
- How does Google keep build times low?
- How does Google avoid clean builds?
- Unit-testing a console app (a text editor)
- More...
