Build systems are one of my favorite topics in software engineering. If I recall correctly, my interest in this area started when I got into NetBSD in 2002—20 years ago—and became a pkgsrc contributor. Packaging software for NetBSD made me fight various build systems and, in particular, experience the pains of debugging the GNU Autotools.

Around that same time, I was also writing small tools here and there. Out of inertia, I used the GNU Autotools for these and, the more I used them, the more I saw an opportunity for improvement. The GNU Autotools were slow, hard to deal with, and they bloated every package. Why did you have to ship heavy configure, Makefile.in and libtool scripts in every single distribution file when you could instead rely on a few system-wide scripts? And thus Buildtool was born in the summer of 2002, just before I started college, and I worked on it for about two years.

The Buildtool project recently came to mind and I noticed that its website is still up and running (kudos to SourceForge for that), so I poked around a bit. Just by looking at the User’s Manual, I’m amazed by how comprehensive the tool is and makes me jealous of how much free time I had back then. Since noticing this, I had been meaning to try the tool again and write a post, and finally got to it just yesterday. So let’s take a tour of what Buildtool was and what it achieved.

A blog on operating systems, programming languages, testing, build systems, my own software projects and even personal productivity. Specifics include FreeBSD, Linux, Rust, Bazel and EndBASIC.

0 subscribers

History

Buildtool was originally inspired by the FreeBSD and NetBSD build systems. The idea was to leverage system-wide generic build logic files to build libraries and binaries—just like bsd.lib.mk and bsd.prog.mk provide—and have arbitrary packages rely on those installed files. As an end user, you would have to install Buildtool first before you could build any other package, but you would only pay the cost of the build infrastructure once. While this paradigm is accepted today, it was quite a departure from the traditions of Unix systems in the early 2000s.

Buildtool’s first version in 2002 was precisely what the NetBSD build system was. The 0.1 release shipped with a copy of NetBSD’s make tool, renamed as bt_make, along with a few mk files to build common targets. That release also included a rudimentary GNU Autoconf-like tool.

Later on, Buildtool grew a bt_wrap helper utility to deal with platform incompatibilities when invoking common tools such as the compiler. This idea was inspired by pkgsrc’s buildlink3 and wrappers infrastructure, which to this day still wrap the compiler and linker to paper over platform-specific oddities. Reading through the release notes, I can see how this helped make Buildtool work on Cygwin and Mac OS X systems of the day.

Towards the latest releases of the project by 2004, things took a significant turn: bt_make and bt_wrap were removed in favor of bt_logic, a custom-made build system purely based on shell scripts. If I recall correctly, the first version of bt_logic was written in Perl, but it never shipped because having Buildtool rely on the gigantic Perl dependency was a non-starter.

By the beginning of 2005, I cancelled the project for the reasons I’ll cover at the end of this post.

Installing Buildtool

The last release of Buildtool was 0.16, which was published on July 4th, 2004 (about 18 years ago) and is still available in the Downloads page if you want to try it out.

Buildtool itself was written in about 10,000 lines shell, but the distribution includes a slimmed down version of NetBSD’s /bin/sh named bt_sh. The raison d’être for bt_sh was pretty much the same as Debian’s dash: a portable, high-performance and standards-compliant shell interpreter to run scripts. bt_sh helped keep Buildtool’s implementation simpler as it did not have to deal with sh platform incompatibilities (Solaris’ version was a pain). Furthermore, bt_sh also ensured that the users of Buildtool writing their own build scripts wouldn’t have to care about shell portability, as is the case when writing configure.ac scripts.

The reason I’m mentioning bt_sh is because trying to build Buildtool 0.16 on a FreeBSD 13 system today fails due to a couple of bugs in the C code. Fixing those bugs is a matter of avoiding trivial conflicts among repeated symbols in different modules. After those simple fixes, Buildtool installs successfully:

/tmp/buildtool-0.16$ ./configure -p /tmp/local
configuring for buildtool-0.16
checking for machine type: amd64
checking for program gcc: not found
checking for program cc: /usr/bin/cc
checking whether /usr/bin/cc is GNU C: yes
[...]

===========================================================================
BUILDTOOL 0.16 CONFIGURATION SUMMARY

    Installation prefix: /tmp/local
    Configuration directory: /tmp/local/etc/buildtool

    Type `make' to start the build.
    Type `make install' to install, only _after_ the build has finished.

===========================================================================

/tmp/buildtool-0.16$ make && make install
[...]
===========================================================================
    Buildtool 0.16 has been successfully installed!

    Please take some time to read the `Testing' section in the README file
    to easily provide useful feedback.  This will not take more than ten
    minutes, but you will be contributing to the project.

    As part of the post installation stage, you should now create a system
    configuration file for the bt_config module, containing cached results
    for several common checks run by many configure scripts.
    For an automated setup, issue the following commands:
        mkdir -p /tmp/local/etc/buildtool
        cp /tmp/local/share/buildtool/templates/bt_config.conf.in \
            /tmp/local/etc/buildtool/bt_config.conf.in
        /tmp/local/bin/buildtool swcgen

    Thanks for choosing Buildtool.
===========================================================================

Post-installation configuration

The installation process quoted above tells us that we should create a system-wide configuration file and then run buildtool swcgen, which stands for “System-wide configuration generator”. Let’s poke at that file and try to follow the given steps.

Here is what /tmp/local/share/buildtool/templates/bt_config.conf.in has to say:

# $Id: bt_config.conf.in,v 1.8 2004/06/26 18:42:05 jmmv Exp $
#
# bt_config.conf - System wide configuration
#
# File automatically generated by @BT_PKG_NAME@, version @BT_PKG_VERSION@.
# Timestamp: @TIMESTAMP@
#
# bt_config will open this file from @BT_DIR_ETC@.
#

# C/C++ environments (leave both for now, all other macros rely on them).
bt_cache_env c cxx

# Basic programs (C/C++ utilities handled by environments).
bt_cache_prog info lex m4 sh yacc

# Header files
bt_cache_hdr sys/cdefs.h sys/utsname.h err.h poll.h stdarg.h stdbool.h ulimit.h
# C++ only header files
bt_cache_hdr bitset deque fstream iostream list map queue set stack string \
             vector

# Basic libraries
bt_cache_lib m

# System specific functions
bt_cache_func setenv strerror stricmp strlcat strlcpy strncat strncpy strndup \
              strftime poll readdir_r ulimit uname vfork vsnprintf

# Types
bt_cache_type gid_t int8_t int16_t int32_t int64_t size_t uid_t uint8_t \
              uint16_t uint32_t uint64_t u_int8_t u_int16_t u_int32_t u_int64_t

# Type sizes
bt_cache_sizeof char short int long "long long"

# Miscellaneous results
bt_cache_attribute

Interesting. This configuration file is all about “caching”. But caching what?

You see, one of the problems I had with the GNU Autotools is how every single package I built and installed on my machine had to go through a very costly configure invocation. It was mind-blowing to me (and still is today) how many CPU hours the world burns on a day-to-day basis checking if a system has standard headers and functions. Couldn’t we check just once and have all packages reuse the results? If the system has vfork, for example, it will continue to have that function for the foreseeable future; there is no need to test for it over and over again.

That’s precisely what the buildtool swcgen command addresses: it executes all of the checks specified in the configuration file and installs those cached results for later reuse. Let’s see what the invocation looks like:

$ cp /tmp/local/share/buildtool/templates/bt_config.conf.in \
    /tmp/local/etc/buildtool/bt_config.conf.in
$ /tmp/local/bin/buildtool swcgen
Input:  /tmp/local/etc/buildtool/bt_config.conf.in
Output: /tmp/local/etc/buildtool/bt_config.conf

bt_swcgen: running bt_wizard to create temporary project skeleton
Entering directory /tmp/bt_swcgen.1802... done.
Creating directories... src
Creating Generic.bt... done.
Creating README.bt... done.
Creating src/Logic.bt... done.
bt_swcgen: generating configuration script and cache
bt_config: starting configuration for bt_swcgen-0.16
checking for program gcc... not found.
checking for program cc... /usr/bin/cc
checking for C compiler name... gnu
checking for C compiler version... 13.0.0
checking for program cpp... /usr/bin/cpp
checking for program ld... /usr/bin/ld
[...]
checking for program sh... /bin/sh
checking for program yacc... /usr/bin/yacc
checking for c header sys/cdefs.h... yes
checking for cxx header sys/cdefs.h... yes
checking for c header sys/utsname.h... yes
checking for cxx header sys/utsname.h... yes
checking for c header err.h... yes
checking for cxx header err.h... yes
[...]
checking for c function vfork... yes
checking for cxx function vfork... no
checking for c function vsnprintf... yes
checking for cxx function vsnprintf... no
checking for c type gid_t... yes
checking for cxx type gid_t... yes
checking for c type int8_t... yes
checking for cxx type int8_t... yes
[...]
bt_config: creating bt_output
bt_config: generating configuration environment
bt_config: generating package dependent build logic

===========================================================================
Configuration summary for bt_swcgen-0.16:

    Prefix is: /usr/local
    Developer mode: yes
    Install documentation: yes
    Static libraries: no, Shared libraries: yes, rpath: yes

===========================================================================

bt_swcgen: creating system wide configuration file
bt_output: creating /tmp/bt_swcgen.1802/conf
bt_swcgen: /tmp/local/etc/buildtool/bt_config.conf created

===========================================================================
PLEASE NOTE THE FOLLOWING:

    Installed: /tmp/local/etc/buildtool/bt_config.conf

    By using a system wide configuration file for bt_config that stores
    check results, you assume that they may get obsoleted with respect to
    your system, specially after software updates.  Be careful to only
    store results that are unlikely to change with time.  Anyway, you are
    encouraged to re-run this program periodically to regenerate the file
    with new results.

    If a third party program fails to configure after a check that shows
    the `(cached)' string in it, try to pass the `--ignore-sw-config'
    flag to bt_config before thinking it is a bug in your system or in
    the package.  DO NOT DISTURB SOFTWARE AUTHORS BEFORE DOUBLE CHECKING
    THAT THERE IS A PROBLEM IN THEIR SOFTWARE.  YOU HAVE BEEN WARNED.
===========================================================================

Pretty impressive that this 18-year old code still works today, and running this shows precisely the problem that this was supposed to solve: computing these results took about 15 seconds on my modest server, and those are 15 seconds you would pay to build most software packages. When you are bulk-building all binary packages for a source-bootstrapped system (the common case in the BSD world), those quickly add up.

By the way: Autoconf supports this same feature although I did not discover it until later. autoswc is a follow-up tool I created for pkgsrc to implement this same idea. Using autoswc proved to be a nice speedup for daily operations when I was still an active pkgsrc maintainer. I suspect that most people are still unaware of this Autoconf feature, unfortunately.

Commands summary

Now that Buildtool is installed and configured, let’s poke around to see what it has to offer:

$ /tmp/local/bin/buildtool --help
buildtool version 0.16
usage: buildtool [options] target [target_options]

Copyright (c) 2002, 2003, 2004 Julio M. Merino Vidal.
This program is licensed under the terms of the BSD license.

Available options:
    {-h,--help}       Show detailed usage (this text).
    {-v,--version}    Show version number.
    {-w,--warnings}   Enable bt_sh warning messages.

Available targets:
    build       Build the package.
    clean       Soft clean the package (keeps configuration data).
    cleandir    Hard clean the package.
    config      Automatically configure a package.
    dist        Generate a distribution file.
    deinstall   Deinstalls the package.
    doc         Read build-time package documentation.
    install     Installs the package.
    lint        Validate the package according to standards.
    pkgflags    Show compilation flags for an installed package.
    siteinfo    Get site specific configuration details.
    swcgen      Aid with creation of system-wide configuration files.
    test        Run tests specific to the package after a successful build.
    wizard      Use an interactive wizard to create initial project files.

See buildtool(1) for more information.

Pretty standard output for a subcommand-based utility, but I can already see myself here. My style in developing command-line tools hasn’t changed much since then.

Anyhow. From the output above, we can distill the following command groups:

config: A configuration command to let each package discover system-wide features, obviously inspired by GNU Autoconf.
build, clean, cleandir, dist, deinstall, install, test: Build-related commands, some of which (like dist) are inspired by GNU Automake.
doc: An interactive documentation viewer for a package’s README, INSTALL, etc. distribution documents. Interesting.
pkgflags: A replacement for the heavy-weight pkg-config. These days we have pkgconf instead, which is compatible with pkg-config.
wizard: An interactive tool to create a new package, akin to what cargo init would offer today.

Demo

Alright. Let’s now use Buildtool’s package creation wizard to initialize a new demo project. The wizard is interactive so I’m just going to post the completed output of what I typed:

$ mkdir /tmp/demo
$ cd /tmp/demo
$ /tmp/local/bin/buildtool wizard
Welcome to Buildtool's Wizard
-----------------------------

This module will help you to set up a basic directory structure
for your project, based on your choices.

Each question has associated a default answer, shown inside the
brackets.  If you hit [RETURN] leaving a question in blank, it
will take the default value.

Project definitions:
- Unix name [foobar]? demo
- Initial version [0.1]?
- License [bsd]?
- Maintainer's email [foo@bar.net]? julio@meroh.net
- Homepage (if any) []?
- Short comment [Sample package]? Remembering Buildtool

Code details:
- Will this package provide one or more programs [y]?
- Will this package provide one or more libraries [n]? y
- Will you use the C language [y]?
- Will you use the C++ language [n]?
- Will you use CVS [y]? n

Dependancies:
- Do you need pkgconfig (not bt_pkgflags) [n]?
- Do you need threads [n]?
- Do you need an X Window System [n]?
- Do you need awk [n]?
- Do you need a lexical analyzer [n]?
- Do you need a LARL parser generator [n]?

Begin process:
- Where should files be created [.]?

Creating directories... src data lib
Creating Generic.bt... done.
Creating README.bt... done.
Creating src/Logic.bt... done.
Creating lib/Logic.bt... done.
Creating data/demo.bpf.in... done.
Creating data/Logic.bt... done.

This asked many more questions than I expected. Looks like I implemented support for many different things.

After completing the wizard, we are left with these files:

/tmp/demo$ find . -type f | sort
./Generic.bt
./README.bt
./data/Logic.bt
./data/demo.bpf.in
./lib/Logic.bt
./src/Logic.bt

From the looks of it, we have a top-level configuration file called Generic.bt, a mysterious data directory with a bpf file, a lib directory to contain the library we’ll write, and a src directory to contain the code of the program we’ll write.

The top-level Generic.bt file

First, let’s look at the top-level Generic.bt file:

#
# Buildtool Generic Script
#

defs() {
    BT_REQUIRE="0.16"

    BT_PKG_NAME="demo"
    BT_PKG_VERSION="0.1"
    BT_PKG_LICENSE="bsd"

    BT_PKG_MAINTAINER="julio@meroh.net"
    BT_PKG_COMMENT="Remembering Buildtool"
}

config_init() {
}

config() {
    bt_check_env_c

    bt_generate_output data/demo.bpf
    bt_generate_configh
}

docs() {
    bt_doc CHANGES "Major changes between package versions"
    bt_doc PEOPLE "Authors and contributors"
    bt_doc README "General documentation"
    bt_doc TODO "Missing things"
}

logic() {
    bt_target  lib data src
}

Generic.bt is the package-wide file where it all begins. This file is composed of a collection of user-defined functions that tell Buildtool what to do:

defs: Describes the package’s metadata, much like what Cargo.toml, go.mod or package.json do today in popular ecosystems.
config_init and config: Allow the package to request system feature checks. In other words, these contain what you would typically put in a configure.ac file with GNU Autoconf.
docs: Configures documentation files, consumed both by the build process and by the buildtool doc interactive viewer.
logic: The entry point to the build process, aka the equivalent of a top-level Makefile.am file.

Creating source files

Before we can successfully build our demo, we have to perform a few edits to the source tree. The wizard didn’t tell us we had to do any of these, so I encountered various failures while I played with the tool. Fear not, it wasn’t too hard to get past these failures.

First, let’s create the top-level documents referenced by the docs function:

/tmp/demo$ touch CHANGES PEOPLE README TODO

Next, let’s look at what building the lib subdirectory will require by looking at the lib/Logic.bt build script:

logic() {
    bt_target demo

    target_demo() {
        BT_LIB_MAJOR=0
        BT_LIB_MINOR=1
        BT_MANPAGES="demo.1"
        BT_SOURCES=func1.c
        BT_TYPE=library

        BT_INCLUDESDIR="${BT_DIR_INC}/demo"
        BT_INCLUDESDIR_demo_h="${BT_DIR_INC}"
        BT_INCLUDES="demo.h"
    }
}

Based on this, let’s create the files lib expects:

/tmp/demo$ cat >lib/demo.h
int func1(void);
^D
/tmp/demo$ cat >lib/func1.c
int func1(void) { return 42; }
^D
/tmp/demo$ touch lib/demo.1

After that, let’s do the same with the src subdirectory. Here is what the src/Logic.bt build script will require:

logic() {
    bt_target demo

    target_demo() {
        BT_MANPAGES="demo.1"
        BT_SOURCES=main.c
        BT_TYPE=program
    }
}

And, as before, let’s create the files src expects:

/tmp/demo$ cat >src/main.c
#include <stdio.h>
#include "lib/demo.h"

int main(void) {
    printf("Calling library: %d\n", func1());
    return 0;
}
^D
/tmp/demo$ touch src/demo.1

Finally, because we have made the binary in src depend on the library in lib, we have to express this dependency in the src/Logic.bt script. We can add these two lines to the target_demo function in src/Logic.bt:

BT_FLAGS_LD="${BT_FLAGS_LD} -L../lib"
BT_LIBS="${BT_LIBS} -ldemo"

Configuring the demo

We are ready to go. Let’s configure the demo project against our current system:

/tmp/demo$ /tmp/local/bin/buildtool config -p /tmp/root
bt_config: loading system-wide configuration
bt_config: starting configuration for demo-0.1
checking for program gcc cc bcc... /usr/bin/cc (cached)
checking for C compiler name... gnu (cached)
checking for C compiler version... 13.0.0 (cached)
checking for program cpp... /usr/bin/cpp (cached)
checking for program ld... /usr/bin/ld (cached)
checking for c header stdio.h... yes (cached)
checking for c header sys/types.h... yes (cached)
checking for c header sys/stat.h... yes (cached)
checking for c header stdlib.h... yes (cached)
checking for c header string.h... yes (cached)
checking for c header unistd.h... yes (cached)
checking for host system name... FreeBSD
checking for host system type... FreeBSD-13.1-STABLE-amd64
checking for program ar... /usr/bin/ar (cached)
checking for program ranlib... /usr/bin/ranlib (cached)
checking for program fastdep... not found.
bt_config: creating bt_output
bt_output: creating data/demo.bpf
bt_config: generating C include header
bt_config: generating configuration environment
bt_config: generating package dependent build logic

===========================================================================
Configuration summary for demo-0.1:

    Prefix is: /tmp/root
    Developer mode: no
    Install documentation: yes
    Static libraries: no, Shared libraries: yes, rpath: yes

===========================================================================

Pretty standard configure-like output.

What I want you to notice is how pretty much all results were cached. These results come from the system-wide cache we populated with swcgen after installing Buildtool, so running this step was almost instantaneous. Good time savings.

Building the demo

With a successful configuration step, let’s try to build:

/tmp/demo$ /tmp/local/bin/buildtool build
bt_logic: entering directory `lib' for `build'
[depend] func1.c -> func1.dep
c++: warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated [-Wdeprecated]
[compile] /usr/bin/cc -I/tmp/demo/lib -I/tmp/demo -I/tmp/demo -fPIC -DPIC -shared -o func1.po -c /tmp/demo/lib/func1.c
cc: warning: argument unused during compilation: '-shared' [-Wunused-command-line-argument]
[link] /usr/bin/cc -Wl,-R/usr/local/lib -shared -Wl,-soname=libdemo.so.0 -o libdemo.so.0.1 func1.po
[exec] ln -fs libdemo.so.0.1 libdemo.so.0
[exec] ln -fs libdemo.so.0.1 libdemo.so
bt_logic: leaving directory `lib' for `build'
bt_logic: entering directory `data' for `build'
bt_logic: leaving directory `data' for `build'
bt_logic: entering directory `src' for `build'
[depend] main.c -> main.dep
c++: warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated [-Wdeprecated]
[compile] /usr/bin/cc -I/tmp/demo/src -I/tmp/demo -I/tmp/demo -fPIC -DPIC -shared -o main.po -c /tmp/demo/src/main.c
cc: warning: argument unused during compilation: '-shared' [-Wunused-command-line-argument]
[runscript] demo
[link] /usr/bin/cc -L/tmp/demo/lib -Wl,-R/usr/local/lib -o real.bt/demo main.po -ldemo
bt_logic: leaving directory `src' for `build'

We get a few warnings around shared libraries, as you could imagine from build scripts written almost 20 years ago, but no failures. Neat.

But does the built product work?

Running the demo

Let’s try to run the final binary we got:

/tmp/demo$ ./src/demo
Calling library: 42

Yay! It does work! But wait a moment:

/tmp/demo$ file src/demo
src/demo: a /tmp/local/libexec/buildtool/bt_sh script, ASCII text executable

What is this about? Why is our demo a script and not a proper binary given that we used C?

/tmp/demo$ cat src/demo
#!/tmp/local/libexec/buildtool/bt_sh
# File generated by bt_logic.
# Fri May 13 06:35:23 PDT 2022

LD_LIBRARY_PATH=:/tmp/demo/lib; export LD_LIBRARY_PATH
/tmp/demo//src/real.bt/demo

Aha. Much like when using GNU Libtool, a binary cannot be run from the source tree if it links against a library, because the library has not been installed yet. The script is just a wrapper to configure the dynamic loader and make things work. After installation, the rpath functionality will kick in and make the dynamic loader find the library in the right place.

Installing the demo

Let’s install the demo:

/tmp/demo$ /tmp/local/bin/buildtool install
bt_logic: entering directory `lib' for `install'
[install] installing data file /tmp/root/lib/libdemo.so.0.1
[exec] ln -fs libdemo.so.0.1 libdemo.so.0
[exec] ln -fs libdemo.so.0.1 libdemo.so
[install] creating missing directory /demo
mkdir: /demo: Permission denied
bt_logic: process stopped; command exited with error status `1'
bt_logic: leaving directory `lib' for `install'
bt_logic: entering directory `data' for `install'
[install] installing data file /tmp/local/share/buildtool/pkgflags/demo.bpf
bt_logic: leaving directory `data' for `install'
bt_logic: entering directory `src' for `install'
[install] installing binary file /tmp/root/bin/demo
[install] installing data file /tmp/root/man/man1/demo.1
bt_logic: leaving directory `src' for `install'
[install] installing data file /tmp/root/share/doc/demo/CHANGES
[install] installing data file /tmp/root/share/doc/demo/PEOPLE
[install] installing data file /tmp/root/share/doc/demo/README
[install] installing data file /tmp/root/share/doc/demo/TODO

Oops, a little mkdir error, but it seems to have worked. If we inspect the installation prefix we provided during the configuration step (the -p /tmp/root flag we specified):

/tmp/demo$ find /tmp/root
/tmp/root
/tmp/root/bin
/tmp/root/bin/demo
/tmp/root/lib
/tmp/root/lib/libdemo.so
/tmp/root/lib/libdemo.so.0.1
/tmp/root/lib/libdemo.so.0
/tmp/root/share
/tmp/root/share/doc
/tmp/root/share/doc/demo
/tmp/root/share/doc/demo/PEOPLE
/tmp/root/share/doc/demo/CHANGES
/tmp/root/share/doc/demo/TODO
/tmp/root/share/doc/demo/README
/tmp/root/man
/tmp/root/man/man1
/tmp/root/man/man1/demo.1

And, from these installed files, we can run the demo:

$ /tmp/root/bin/demo
Calling library: 42

Finally, note how the installed demo is really a proper executable that uses the shared library that was installed along it:

/tmp/demo$ file /tmp/root/bin/demo
/tmp/root/bin/demo: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 13.0 (1300525), FreeBSD-style, with debug_info, not stripped
/tmp/demo$ ldd /tmp/root/bin/demo
/tmp/root/bin/demo:
        libdemo.so.0 => /tmp/root/lib/libdemo.so.0 (0x200000)
        libc.so.7 => /lib/libc.so.7 (0x200000)
[preloaded]
        [vdso] (0x7ffffffff5d0)

Distributing the demo

As a software package author, installing the program in one’s machine is nice, but what about giving the program to other people? This is where the dist command comes into play, much like GNU Automake’s dist target:

/tmp/demo$ /tmp/local/bin/buildtool dist
bt_dist: cleaning tree (cleandir)
bt_logic: entering directory `lib' for `clean'
[remove]  func1.po
[remove]  libdemo.so.0.1 libdemo.so.0 libdemo.so
bt_logic: leaving directory `lib' for `clean'
bt_logic: entering directory `lib' for `cleandir'
[remove]  func1.dep
bt_logic: leaving directory `lib' for `cleandir'
bt_logic: entering directory `data' for `clean'
[remove]  demo.bpf
bt_logic: leaving directory `data' for `clean'
bt_logic: entering directory `data' for `cleandir'
bt_logic: leaving directory `data' for `cleandir'
bt_logic: entering directory `src' for `clean'
[remove]  main.po
[remove]  demo
[remove]  real.bt/demo
bt_logic: leaving directory `src' for `clean'
bt_logic: entering directory `src' for `cleandir'
[remove]  main.dep
bt_logic: leaving directory `src' for `cleandir'
[remove]  /tmp/demo/bt_config.h /tmp/demo/bt_config.env /tmp/demo/bt_logic.env /tmp/demo/bt_config.log /tmp/demo/bt_config.sed /tmp/demo/bt_output
bt_dist: validating package
=> Checking root files
WARN: COPYING not found; it is highly recommended
=> Checking definitions (/tmp/demo/Generic.bt)
WARN: no package homepage
=> Checking configuration script (/tmp/demo/Generic.bt)
=> Summary
bt_lint: package should be corrected; 2 warnings
bt_dist: building compressed dist, type tar.gz

The above invocation above generates a demo-0.1.tar.gz distributable source package, which we can inspect:

/tmp/demo$ tar tzf ../demo-0.1.tar.gz
demo-0.1/
demo-0.1/PEOPLE
demo-0.1/lib/
demo-0.1/CHANGES
demo-0.1/TODO
demo-0.1/README.bt
demo-0.1/data/
demo-0.1/README
demo-0.1/src/
demo-0.1/Generic.bt
demo-0.1/src/Logic.bt
demo-0.1/src/main.c
demo-0.1/src/demo.1
demo-0.1/data/Logic.bt
demo-0.1/data/demo.bpf.in
demo-0.1/lib/func1.c
demo-0.1/lib/Logic.bt
demo-0.1/lib/demo.1
demo-0.1/lib/demo.h

Peeking into the data directory

I glanced over the data directory earlier on, skipping that mysterious bpf file. If we look at the files that were installed under Buildtool’s own prefix (a mistake) when we installed the demo, we can find that our package installed a share/buildtool/pkgflags/demo.bpf file. Looking at its contents, we see:

# $Id: pkgflags,v 1.1 2003/04/26 21:48:17 jmmv Exp $
# pkgflags file
#
# This file is mostly useful for packages providing libraries.
# If not needed (i.e., if the package is a program), remove it.
#

BT_PREFIX="/tmp/root"
BT_DIR_LIB="/tmp/root/lib"
BT_DIR_INCLUDE="/tmp/root/include"

bpf_name="demo"
bpf_descr="Remembering Buildtool"
bpf_version="0.1"
bpf_libs="-L${BT_DIR_LIB} -ldemo"
bpf_cflags="-I${BT_DIR_INCLUDE}"

This is the equivalent of what a pkg-config file would provide. And it works. We can query the file using Buildtool’s pkgflags command and obtain the right compiler and linker flags for our demo library:

$ /tmp/local/bin/buildtool pkgflags --cflags --libs demo
setting undefined variable `BT_PREFIX'
setting undefined variable `BT_DIR_LIB'
setting undefined variable `BT_DIR_INCLUDE'
-I/tmp/root/include -L/tmp/root/lib -ldemo

And this concludes the demo.

What happened to Buildtool?

Buildtool as it was in its 0.16 release in 2004 seems fairly impressive. The user manual is comprehensive, the tool provides many more features than I remembered, and it still works to this day.

So what happened to Buildtool? Why did I officially cancel it in 2005? There are a few reasons.

The first is that Buildtool collapsed under its own complexity. Shell is not the right language to write a build system in, and bt_logic became an unmanageable mess to deal with. I’m surprised it still works on a modern system to be honest.
The second is that, even though Buildtool seems “complete”, it still lacks lots of functionality—and the missing functionality wasn’t easy to implement. When I was writing the tool, I found myself leaning on the GNU Autotools manuals to understand how things worked across systems, and I relied on those details to implement my own versions. The more I did this, the more I learned about the GNU Autotools, and the more I realized how knowledgeable the GNU Autotools authors were and how far I was from providing something comparable. This was humbling. Furthermore, during this process, I had become so “fluent” in the GNU Autotools that it made no sense to continue developing my own thing.
The third is that having to install a supporting build tool to compile fundamental system packages wasn’t well-seen at the time. Every time I had to deal with a similar system in pkgsrc (Boost.Jam or whatever Mozilla had, for example), it was a pain. Like it or not, the GNU Autotools are the de-facto standard, and despite all of their flaws, they are the ones that integrate well with packaging systems and system tools.

My interest in build systems remains and what you have read here partially explains my original critique of Bazel. I ended up working in Bazel for a few years though because I like the topic. No matter my original comments, Bazel is a great build system from which many others should learn. But it’s still missing the kind of system-level integration that Buildtool intended to provide via the install and pkgflags commands.

One thing I’ll note is how Buildtool already had a concept of high-level targets from a semantical perspective. You could make a single build file define various targets (libraries, programs, etc.) and each of those would know how to build, install, and clean itself. You can see that the same ideas have evolved in most of today’s build systems in some form or another, and that the file-level dependency tracking that make provides is “a thing of the past”.

To conclude, let me add that I still believe there is room for something like Buildtool in this day and age to support the foundations of our free Unix-like systems… but it would need to be much better designed and implemented than Buildtool. I have ideas; I just wish I had the same amount of free time that I had when I was a student.