Lua - Julio Merino (jmmv.dev)

Three productive days on the Kyua front

This being Thanksgiving week in the U.S. and Google giving us Thursday and Friday off, I decided to take Monday to Wednesday off as well to spend some time hacking on Kyua — yes, finally, after months of being inactive. And what a three productive days! Here comes a little briefing on the three fronts in which I made progress. (This puts on hold the header files series until next Monday... but most of you are probably away anyway. Enjoy the holidays if they apply to you!)

November 28, 2013 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/fedora">fedora</a>, <a href="/tags/freebsd">freebsd</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/lua">lua</a>
Continue reading (about 6 minutes)

Exposing a configuration tree through Lua

In the previous post, I discussed the type-safe tree data structure that is now in the Kyua codebase, aimed at representing the configuration of the program. In this post, we'll see how this data structure ties to the parsing of the configuration file. One goal in the design of the configuration file was to make its contents a simple key/value association (i.e. assigning values to predetermined configuration variables). Of course, the fact that the configuration file is just a Lua script means that additional constructions (conditionals, functions, etc.) can be used to compute these values before assignment, but in the end all we want to have is a collection of values for known keys. The tree data structure does exactly the latter: maintain the mapping of keys to values, and ensuring that only a set of "valid" keys can be set. But, as a data structure, it does not contain any of the "logic" involved in computing those values: that is the job of the script. Now, consider that we have the possible following syntaxes in the configuration file: simple_variable = "the value" complex.nested.variable = "some other value" These assignments map, exactly, to a tree::set() function call: the name of the key is passed as the first argument to tree::set() and the value is passed as the second argument. (Let's omit types for simplicity.) What we want to do is modify the Lua environment so that these assignments are possible, and that when such assignments happen, the internal tree object gets updated with the new values. In order to achieve this, the configuration library modifies the Lua environment as follows: The newindex metatable method of _G is overridden so that an assignment causes a direct call to the set method of the referenced key. The key name is readily available in the newindex arguments, so no further magic is needed. This handles the case of "a = b" (top-level variables).The index metatable method of _G is overridden so that, if the indexed element is not found, a new table is generated and injected into _G. This new table has a metatable of its own that performs the same operations as the newindex and index herein described. This handles the case of "a.b = c", as this trick causes the intermediate tables (in this case "a") to be transparently created.Each of the tables created by index has a "key" metatable field that contains the fully qualified key of the node the table corresponds to. This is necessary to be able to construct the full key to pass to the set method.There is further magic to ensure that values pre-populated in the tree (aka default values) can be queried from within Lua, and that variables can be set more than once. These details are uninteresting though.At the moment, we deny setting variables that have not been pre-defined in the tree structure, which means that if the user wants to define auxiliary variables or functions, these must be declared local to prevent calling into the _G hooks. This is quite nice, but we may need to change this later on if we want to export the standard Lua modules to the configuration files.

June 2, 2012 · Tags: <a href="/tags/cxx">cxx</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/lua">lua</a>
Continue reading (about 3 minutes)

Rethinking Kyua's configuration system

In the previous blog post, I described the problems that the implementation of the Kyua configuration file parsing and in-memory representation posed. I also hinted that some new code was coming and, after weeks of work, I'm happy to say that it has just landed in the tree! I really want to get to explaining the nitty-gritty details of the implementation, but I'll keep these for later. Let's focus first on what the goals for the new configuration module were, as these drove a lot of the implementation details:Key/value pairs representation: The previous configuration system did this already, and it is a pretty good form for a configuration file because it is a simple, understandable and widespread format. Note that I have not said anything yet about the types of the values.Tree-like representation: The previous configuration schema grouped test-suite specific properties under a "test_suites" map while it left internal run-time properties in the global namespace. The former is perfect and the latter was done just for simplicity. I want to move towards a tree of properties to give context to each of them so that they can be grouped semantically (e.g. kyua.report.*, kyua.runtime.*, etc.). The new code has not changed the structure of the properties yet (to remain compatible with previous files), but it adds very simple support to change this in the shortcoming future.Single-place parsing and validation: A configuration file is an external representation of a set of properties. This data is read (parsed) once and converted into an in-memory representation. All validation of the values of the properties must happen at this stage, and not when the properties are queried. The reason is that validation of external values must be consistent and has to happen in a controlled location (so that errors can all be reported at the same time). I have seen code in other projects where the configuration file is stored in memory as a set of key/value string pairs and parsing to other types (such as integers, etc.) is delayed until the values are used. The result is that, if a property is queried more than once, the validation will be implemented in different forms, each with its own bugs, which will result in dangerous inconsistencies.Type safety: This is probably the trickiest bit. Every configuration node must be stored in the type that makes most sense for its value. For example: a timeout in seconds is an integer, so the in-memory representation must be an integer. Or another example: the type describing the "unprivileged user" is a data structure that maps to a system user, yet the configuration file just specifies either a username or a UID. Keeping strict type validation in the code is interesting because it helps to ensure that parsing and validation happen in just a single place: whenever the configuration file is read, every property will have to be converted to its in-memory type, and this means that the validation can only happen at that particular time. Once the data is in memory, we can and have to assume that it is valid. Additionally, strict types ensure that the code querying such properties uses the values as intended, without having to do additional magic to map them to other types.Extensibility: Parsing a configuration file is a very generic concept, yet the previous code made the mistake of tying this logic with the specific details of Kyua configuration files. A goal of the new code has been to write a library that parses configuration files, and allows the Kyua-specific code to define the schema of the configuration file separately. (No, the library is not shipped separately at this point; it's placed in its own utils::config module.)With all this code in place, there are a bunch of things that can now be easily implemented. Consider the following:Properties to define the timeout of test cases depending on their size (long-standing issue 5).Properties to tune the UI behavior: width of the screen, whether to use color or not (no, there is no color support yet), etc.Properties to configure how reports look like "by default": if you generate reports of any form frequently, it is very likely that you will want them to look the same every time and hence you will want to define the report settings once in the configuration file.Hooks: one of the reasons for using Lua-based configuration files was to allow providing extra customization abilities to the user. Kyua could theoretically call back into Lua code to perform particular actions, and such actions could be explicitly stated by the user in the form of Lua functions. Neither the current configuration code nor Kyua has support for hooks, but the new implementation makes it rather easy to add them.And that's all for today. Now that you know what the current code is trying to achieve and why, we will be able to look at how the implementation does all this in the next posts.

May 28, 2012 · Tags: <a href="/tags/cxx">cxx</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/lua">lua</a>
Continue reading (about 4 minutes)

Name your C++ auto-cleaners

As you may already know, RAII is a very powerful and popular pattern in the C++ language. With RAII, you can wrap non-stack-managed resources into a stack-managed object such that, when the stack-managed object goes out of scope, it releases the corresponding non-stack-managed object. Smart pointers are just one example of this technique, but so are IO streams too. Before getting into the point of the article, bear with me for a second while I explain what the stack_cleaner object of Lutok is. The "stack cleaner" takes a reference to a Lua state and records the height of the Lua stack on creation. When the object is destroyed (which happens when the declaring function exits), the stack is returned to its previous height thus ensuring it is clean. It is always a good idea for a function to prevent side-effects by leaving its outside world as it was — and, like it or not, the Lua state is part of the outside world because it is an input/output parameter to many functions. Let's consider a piece of code without using the stack cleaner: void my_function(lutok::state& state, const int foo) { state.push_integer(foo); ... do something else in the state ... const int bar = state.to_integer(); if (bar != 3) { state.pop(1); throw std::runtime_error("Invalid data!"); } state.pop(1); } Note that we have had to call state.pop(1) from "all" exit points of the function to ensure that the stack is left unmodified upon return of my_function. Also note that "all exit points" may not be accurate: in a language that supports exceptions, any statement may potentially raise an exception so to be really safe we should do: void my_function(lutok::state& state, const int foo) { state.push_integer(foo); try { ... do something else in the state ... const int bar = state.to_integer(); if (bar != 3 throw std::runtime_error("Invalid data!"); } catch (...) { state.pop(1); throw; } state.pop(1); } ... which gets old very quickly. Writing this kind of code is error-prone and boring. With an "auto-cleaner" object such as the stack_cleaner, we can simplify our code like this: void my_function(lutok::state& state, const int foo) { lutok::stack_cleaner cleaner(state); state.push_integer(foo); ... do something else in the state ... const int bar = state.to_integer(); if (bar != 3) throw std::runtime_error("Invalid data!"); } And we leave the boring task of determining when to actually call state.pop(1) to the compiler and the runtime environment. In this particular case, no matter how the my_function terminates, we ensure that the Lua stack will be left as the same size as it was before. But, as I said earlier, all this was just an introduction to the idea that made me write this post. When you declare an auto-cleaner object of any kind, be sure to give it a name. It has happened to me a few times already that I have written the following construct: lutok::stack_cleaner(state); ... which is syntactically correct, harmless and "looks good" if you don't look closely. The compiler will chew along just fine because, even though we are declaring an anonymous object, its constructor and destructor may be doing who-knows-what, so their code must be called and thus the "unused variable" warning cannot really be raised. However this does not give us the desired behavior. The cleaner object will be constructed and destructed in the same statement without having a chance to wrap any of the following code, because its scope is just the statement in which it was defined. In other words, the cleaner will have absolutely no effect on the rest of the function and thus will be useless. So, moral of the story: always give a name to your auto-cleaner objects so that their scope is correctly defined and their destructor is run when you actually expect: lutok::stack_cleaner ANY_NAME_HERE(state);

September 17, 2011 · Tags: <a href="/tags/cxx">cxx</a>, <a href="/tags/lua">lua</a>, <a href="/tags/lutok">lutok</a>
Continue reading (about 4 minutes)

Introducing Lutok: A lightweight C++ API for Lua

It has finally happened. Lutok is the result of what was promised in the "Splitting utils::lua from Kyua" web post. Quoting the project web page: Lutok provides thin C++ wrappers around the Lua C API to ease the interaction between C++ and Lua. These wrappers make intensive use of RAII to prevent resource leakage, expose C++-friendly data types, report errors by means of exceptions and ensure that the Lua stack is always left untouched in the face of errors. The library also provides a small subset of miscellaneous utility functions built on top of the wrappers. Lutok focuses on providing a clean and safe C++ interface; the drawback is that it is not suitable for performance-critical environments. In order to implement error-safe C++ wrappers on top of a Lua C binary library, Lutok adds several layers or abstraction and error checking that go against the original spirit of the Lua C API and thus degrade performance. Lutok was originally developed within Kyua but was later split into its own project to make it available to general developers.Coming up with a name for this project was quite an odyssey, and is what has delayed is release more than I wanted. My original candidate was "luawrap" which, although not very original, was to-the-point and easy to understand. Unfortunately, that name did not clear with the legal department and I had to propose several other names, some of which were not acceptable either. Eventually, I settled with "Lutok", which comes from "LUa TOolKit". At this point, the source tree of Lutok provides pretty much the same code as the utils::lua module of Kyua. While it may be enough to get you started, I'm pretty sure you will lack some functions in the state class. If that is the case, don't hesitate to file a bug report to let me know what is missing. In case you missed the link above, the project page is here: Lutok in Google Code.

September 15, 2011 · Tags: <a href="/tags/announce">announce</a>, <a href="/tags/cxx">cxx</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/lua">lua</a>, <a href="/tags/lutok">lutok</a>
Continue reading (about 2 minutes)

Splitting utils::lua from Kyua

If you remember a post from January titled C++ interface to Lua for Kyua (wow, time flies), the Kyua codebase includes a small library to wrap the native Lua C library into a more natural C++ interface. You can take a look at the current code as of r129. Quoting the previous post: The utils::lua library provides thin C++ wrappers around the Lua C API to ease the interaction between C++ and Lua. These wrappers make intensive use of RAII to prevent resource leakage, expose C++-friendly data types, report errors by means of exceptions and ensure that the Lua stack is always left untouched in the face of errors. The library also provides a place (the operations module) to add miscellaneous utility functions built on top of the wrappers.While the RAII wrappers and other C++-specific constructions are a very nice thing to have, this library has to jump through a lot of hoops to interact with binary Lua versions built for C. This makes utils::lua not usable for performance-critical environments. Things would be way easier if utils::lua linked to a Lua binary built for C++, but unfortunately that is not viable in most, if not all, systems with binary packaging systems (read: most Linux distributions, BSD systems, etc.). That said, I've had requests from a bunch of people to provide utils::lua separately from Kyua regardless of any performance shortcomings it may have, and this is what I have started doing this weekend. So far, I already have a pretty clumsy standalone package (I'll keep the name to myself for now ;-) that provides this library on its own with the traditional Automake, Autoconf and Libtool support. Once this is a bit better quality, and once I modify Kyua to link against this external library and assess that things work fine, I'll make the decision on how to publish this (but most likely it should be a separate project in Google Code). Splitting the code doesn't come with its own issues though: maintaining a separate package will involve more work and hopefully/supposedly, dealing with quite a few feature requests to add missing functionality! Also, it means that utils::lua cannot use any of the other Kyua libraries (utils::sanity for example), so I lose a bit of consistency across the Kyua codebase. I am also not sure about how to share non-library code (in particular, the m4 macros for Autoconf) across the two packages. So, my question is: are you interested in utils::lua being shipped separately? :-) Do you have any cool use cases for it that you can share here? Thanks!

September 3, 2011 · Tags: <a href="/tags/cxx">cxx</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/lua">lua</a>
Continue reading (about 3 minutes)

Injecting C++ functions into Lua

The C++ interface to Lua implemented in Kyua exposes a lua::state class that wraps the lower-level lua_State* type. This class completely hides the internal C type of Lua to ensure that all calls that affect the state go through the lua::state class. Things get a bit messy when we want to inject native functions into the Lua environment. These functions follow the prototype represented by the lua_CFunction type:typedef int (*lua_CFunction)(lua_State*);Now, let's consider this code:int awesome_native_function(lua_State* state) { // Uh, we have access to s, so we bypass the lua::state! ... do something nasty ... // Oh, and we can throw an exception here... //with bad consequences. } void setup(...) { lua::state state; state.push_c_function(awesome_native_function); state.set_global("myfunc"); ... run some script ... }The fact that we must pass a lua_CFunction prototype to the lua_pushcfunction object means that such function must have access to the raw lua_State* pointer... which we want to avoid. What we really want is the caller code to define a function such as:typedef int (*cxx_function)(lua::state&)In an ideal world, the lua::state class would implement a push_cxx_function that took a cxx_function, generated a thin C wrapper and injected such generated wrapper into Lua. Unfortunately, we are not in an ideal world: C++ does not have high-order functions and thus the "generate a wrapper function" part of the previous proposal does not really work. What we can do instead, though, is to make the creation of C wrappers for these C++ functions trivial. And this is what r42 did. The approach I took is similar to this overly-simplified (and broken) example:template< cxx_function Function > int wrap_cxx_function(lua_State* state) { try { lua::state state_wrapper(state); return Function(state_wrapper); } catch (...) { luaL_error(state, "Geez, don't go into C's land!"); } }This template wrapper takes a cxx_function object and generates a corresponding C function at compile time. This wrapper function ensures that C++ state does not propagate into the C world, as that often has catastrophical consequences. (Due to language limitations, the input function must have external linkage. So no, it cannot be static.) As a result, we can rewrite our original snippet as:int awesome_native_function(lua::state& state) { // See, we cannot access lua_State* now. ... do something ... throw std::runtime_error("And we can even do this!"); } void setup(...) { lua::state state; state.push_c_function( wrap_cxx_function< awesome_native_function >); state.set_global("myfunc"); ... run some script ... }Neat? I think so, but maybe not so much. I'm pretty sure there are cooler ways of achieving the above purpose in a cleaner way, but this one works nicely and has few overhead.

January 17, 2011 · Tags: <a href="/tags/cxx">cxx</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/lua">lua</a>
Continue reading (about 2 minutes)

Error handling in Lua: the Kyua approach

About a week ago, I detailed the different approaches I encountered to deal with errors raised by the Lua C API. Later, I announced the new C++ interface for Lua implemented within Kyua. And today, I would like to talk about the specific mechanism I implemented in this library to deal with the Lua errors. The first thing to keep in mind is that the whole purpose of Lua in the context of Kyua is to parse configuration files. This is an infrequent operation, so high performance does not matter: it is more valuable to me to be able to write robust algorithms fast than to have them run at optimal speed. The other key point to consider is that I want Kyua to be able to use prebuilt Lua libraries, which are built as C binaries. The approach I took is to wrap every single unsafe Lua C API call in a "thin" (FSVO thin depending on the case) wrapper that gets called by lua_pcall. Anything that runs inside the wrapper is safe to Lua errors, as they are caught and safely reported to the caller. Lets examine how this works by taking a look at an example: the wrapping of lua_getglobal. We have the following code (copy pasted from the utils/lua/wrap.cpp file but hand-edited for publishing here):static int protected_getglobal(lua_State* state) { lua_getglobal(state, lua_tostring(state, -1)); return 1; } void lua::state::get_global(const std::string& name) { lua_pushcfunction(_pimpl->lua_state, protected_getglobal); lua_pushstring(_pimpl->lua_state, name.c_str()); if (lua_pcall(_pimpl->lua_state, 1, 1, 0) != 0) throw lua::api_error::from_stack(_pimpl->lua_state, "lua_getglobal"); }The state::get_global method is my public wrapper for the lua_getglobal Lua C API call. This wrapper first prepares the Lua stack by pushing the address of the C function to call and its parameters and then issues a lua_pcall call that executes the C function in a Lua protected environment. In this case, the argument preparation for protected_getglobal is trivial because the lua_getglobal call does not require access to any preexisting values on the Lua stack. Things get much trickier when that happens as in the case of the lua_getglobal wrapper. I'll leave understanding how to do this as an exercise to the reader (but you can cheat by looking at line 154). Anyway. The above looks all very nice and safe and the tests for the state::get_global function, even the ones that intentionally cause a failure, all work fine. So we are good, right? Nope! Unfortunately, the code above is not fully safe to Lua errors. In order to prepare the lua_pcall execution, the code must push values on the stack. As it turns out, both lua_pushcfunction and lua_pushstring can fail if they run out of memory (OOM). Such failure would of course be captured inside a protected environment... but we have a little chicken'n'egg problem here. That said, OOM failures are rare so I'm going to leverage this fact and not worry about it. (Note to self: install a lua_atpanic handler to complain loudly if that ever happens.) Addendum: Bundling Lua within my program and building it as a C++ binary with exception reporting enabled in luaconf.h would magically solve all my issues. I know. But I don't fancy the idea of bundling the library into my source tree for a variety of reasons.

January 14, 2011 · Tags: <a href="/tags/cxx">cxx</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/lua">lua</a>
Continue reading (about 3 minutes)

C++ interface to Lua for Kyua

Finally! After two weeks of holidays work, I have finally been able to submit Kyua's r39: a generic library that implements a C++ interface to Lua. The code is hosted in the utils/lua/ subdirectory. From the revision description:The utils::lua library provides thin C++ wrappers around the Lua C API to ease the interaction between C++ and Lua. These wrappers make intensive use of RAII to prevent resource leakage, expose C++-friendly data types, report errors by means of exceptions and ensure that the Lua stack is always left untouched in the face of errors. The library also provides a place (the operations module) to add miscellaneous utility functions built on top of the wrappers.In other words: this code aims to decouple all details of the interaction with the Lua C API from the main code of Kyua so that the high level algorithms do not have to worry about Lua C API idiosyncrasies. Further changes to Kyua to implement the new configuration system will follow soon as all the basic code to talk to Lua has been ironed out. Also expect some extra posts regarding the design decisions that went on this helper code and, in particular, about error reporting as mentioned in the previous post. (Yep, Lua and Kyua sound similar. But that was never intended; promise!)

January 8, 2011 · Tags: <a href="/tags/cxx">cxx</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/lua">lua</a>
Continue reading (about 2 minutes)

Error handling in Lua

Some of the methods of the Lua C API can raise errors. To get an initial idea on what these are, take a look at the Functions and Types section and pay attention to the third field of a function description (the one denoted by 'x' in the introduction). Dealing with the errors raised by these functions is tricky, not to say a nightmare. Also, the ridiculously-short documentation on this topic does not help. This post is dedicated to explain how these errors may be handled along with the advantages and disadvantages of each case. The Lua C API provides two modes of execution: protected and unprotected. When in protected mode, all errors caused by Lua are caught and reported to the caller in a controlled manner. When in unprotected mode, the errors just abort the execution of the calling process by default. So, one would think: just run the code in protected mode, right? Yeah, well... entering protected mode is nontrivial and it has its own particularities that make interaction with C++ problematic. Let's analyze error reporting by considering a simple example: the lua_gettable function. The following Lua code would error out when executed:my_array = nil return my_array["test"]... which is obvious because indexing a non-table object is a mistake. Now let's consider how this code would look like in C (modulo the my_array assignment):lua_getglobal(state, "my_array"); lua_pushstring(state, "test"); lua_gettable(state, -2);Simple, huh? Sure, but as it turns out, any of the API calls (not just lua_gettable) in this code can raise errors (I'll call them unsafe functions). What this means is that, unless you run the code with a lua_pcall wrapper, your program will simply exit in the face of a Lua error. Uh, your scripting language can "crash" your host program out of your control? Not nice. What would be nice is if each of the Lua C API unsafe functions reported an error (as a return value or whatever) and allowed the caller to decide what to do. Ideally, no state would change in the face of an error. Unfortunately, that is not the case but it is exactly what I would like to do. I am writing a C++ wrapper for Lua in the context of Kyua and fine granularity in error reporting means that automatic cleanup of resources managed by RAII is trivial. Let's analyze the options that we have to control errors caused within the Lua C API. I will explain in a later post the one I have chosen for the wrapper in Kyua (it has to be later because I'm not settled yet!). Install a panic handler Whenever Lua code runs in an unprotected environment, one can use lua_atpanic to install a handler for errors. The function provided by the user is executed when the error occurs and, if the panic function returns, the program exits. To prevent exiting prematurely, one could opt for two mechanisms: Make the panic handler raise a C++ exception. Sounds nice, right? Well, it does not work. The Lua library is generally built as a C binary which means that our panic handler will be called from within a C environment. As a result, we cannot throw an exception from our C++ handler and expect things to work: the exception won't propagate correctly from a C++ context to a C context and then back to C++. Most likely, the program will abort as soon as we leave the C++ world and enter C to unwind the stack.Use setjmp before the call to the unsafe Lua function and recover with longjmp from within the panic handler. It turns out that this does work but with one important caveat: the stack is completely cleared before the call to the panic handler. As a result, this prevents the requirement of "leave the stack unmodified on failure" as is desired of any function (report errors early before changing state).Run every single call in a protected environment This is doable but complex and not completely right: to do this, we need to write a C wrapper function for every unsafe API function and run it with lua_pcall. The overhead of this approach is significant: something as simple as a call to lua_gettable turns into several stack manipulation operations, a call to lua_pcall and then further stack modifications to adjust the results. Additionally, in order to prepare the call to lua_pcall, one has to use the multiple lua_push* functions to prepare the stack for the call. And, guess what, most of these functions that push values onto the stack can themselves fail. So... in order to prepare the environment for a safe call, we are already executing unsafe calls. (Granted, the errors in these case are only due to memory exhaustion... but still, the solution is not fully robust.) Lastly, note that we cannot use lua_cpcall because it does discard all return values of the executed function. Which means that we can't really wrap single Lua operations. (We could wrap a whole algorithm though.) Run the whole algorithm in a protected environment This defeats the whole purpose of the per-function wrapping. We would need to provide a separate C/C++ function that runs all unsafe code and then call it by means of lua_pcall (or lua_cpcall) so that errors are captured and reported in a controlled manner. This seems very efficient... albeit not transparent and will surely cause issues. Why is this problematic? Errors that happen inside the protected environment are managed by means of a longjmp. If the code wrapped by lua_pcall is a C++ function, it can instantiate objects. These objects have destructors. A longjmp outside of the function means that no destructors will run... so objects will leak memory, file descriptors, and anything you can imagine. Doom's day. Yes, I know Lua can be rebuilt to report internal errors by means of exceptions which would make this particular problem a non-issue... but this rules out any pre-packaged Lua binaries (the default is to use longjmp and henceforth what packaged binaries use). I do not want to embed Lua into my source tree. I want to use Lua binary packages shipped with pretty much any OS (hey, including NetBSD!), which means that my code needs to be able to cope with Lua binaries that use setjmp/longjmp internally. Closing remarks I hope the above description makes any sense because I had to omit many, many details in order to make the post reasonably short. It could also be that there are other alternatives I have not considered, in which case I'd love to know them. Trying to find a solution to the above problem has already sucked several days of my free time, which translates in Kyua not seeing any further development until a solution is found!

January 7, 2011 · Tags: <a href="/tags/c">c</a>, <a href="/tags/cxx">cxx</a>, <a href="/tags/lua">lua</a>
Continue reading (about 6 minutes)

Using RAII to clean up temporary values from a stack

For the last couple of days, I have been playing around with the Lua C API and have been writing a thin wrapper library for C++. The main purpose of this auxiliary library is to ensure that global interpreter resources such as the global state or the execution stack are kept consistent in the presence of exceptions — and, in particular, that none of these are leaked due to programming mistakes when handling error codes. To illustrate this point, let's forget about Lua and consider a simpler case. Suppose we lost the ability to pass arguments and return values from functions in C++ and all we have is a stack that we pass around. With this in mind, we could implement a multiply function as follows:void multiply(std::stack< int >& context) { const int arg1 = context.top(); context.pop(); const int arg2 = context.top(); context.pop(); context.push(arg1 * arg2); }And we could call our function as this:std::stack< int > context; context.push(5); context.push(6); multiply(context); const int result = s.top(); s.pop();In fact, my friends, this is more-or-less what your C/C++ compiler is internally doing when converting code to assembly language. The way the stack is organized to perform calls is known as the calling conventions of an ABI (language/platform combination). Anyway, back to our point. One important property of such a stack-based system is that any function that deals with the stack must leave it in a consistent state: if the function pushes temporary values (read: local variables) into the stack, such temporary values must be gone upon return no matter how the function terminates. Otherwise, the caller will not find the stack as it expects, which will surely cause trouble at a later stage. The above example works just fine because our function is extremely simple and does not put anything on the stack. But things get messier when our functions can fail halfway through, and, in particular, if such failures are signaled by exceptions. In these cases, the function will abort abruptly and the function must take care to clean up any values that may still be left on the stack. Let's consider another example:void magic(std::stack< int >& context) { const int arg1 = context.top(); context.pop(); const int arg2 = context.top(); context.pop(); context.push(arg1 * arg2); context.push(arg1 / arg2); try { ... do something with the two values on top ... context.push(arg1 - arg2); try { ... do something with the three values on top ... } catch (...) { context.pop(); // arg1 - arg2 throw; } context.pop(); } catch (...) { context.pop(); // arg1 / arg2 context.pop(); // arg1 * arg2 throw; } context.pop(); context.pop(); }The above is a completely fictitious and useless function, but serves to illustrate the point. magic() starts by pushing two values on the stack and then performs some computation that reads these two values. It later pushes an additional value and does some more computations on the three temporary values that are on the top of the stack. The "problem" is that the computation code can throw an exception. If it does, we must sanitize the stack to remove the two or three values we have already pushed. Otherwise, the caller will receive the exception, it will assume nothing has happened, and will leak values on the stack (bad thing). To prevent this, we have added a couple of try/catch clauses to capture these possible exceptions and to clean up the already-pushed values before exiting the function. Unfortunately, this gets old very quickly: having to add try/catch statements surrounding every call is boring, ugly, and hard to read (remember that, potentially, any statement can throw an exception). You can see this in the example above with the two nested try/catch blocks. To mitigate this situation, we can apply a RAII-like technique to make popping elements on errors completely transparent and automated. If we can make it transparent, writing the code is easier and reading it is trivial; if we can make it automated, we can be certain that our error paths (rarely tested!) correctly clean up any global state. In C++, destructors are deterministically executed whenever a variable goes out of scope, so we can use this to our advantage to clean up temporary values. Let's consider this class:class temp_stack { std::stack< int >& _stack; int _pop_count; public: temp_stack(std::stack< int >& stack_) : _stack(stack_), _pop_count(0) {} ~temp_stack(void) { while (_pop_count-- > 0) _stack.pop(); } void push(int i) { _stack.push(i); _pop_count++; } };With this, we can rewrite our function as:void magic(std::stack< int >& context) { const int arg1 = context.top(); context.pop(); const int arg2 = context.top(); context.pop(); temp_stack temp(context); temp_stack.push(arg1 * arg2); temp_stack.push(arg1 / arg2); ... do something with the two values on top ... temp_stack.push(arg1 - arg2); ... do something with the three values on top ... // Yes, we can return now. No need to do manual pop()s! }Simple, huh? Our temp_stack function keeps track of how many elements have been pushed on the stack. Whenever the function terminates, be it due to reaching the end of the body or due to an exception thrown anywhere, the temp_stack destructor will remove all elements previously registered from the stack. This ensures that the function leaves the global state (the stack) as it was on entry — modulo the function parameters consumed as part of the calling conventions. So how does all this play together with Lua? Well, Lua maintains a stack to communicate parameters and return values between C and Lua. Such stack can be managed in a similar way with a RAII class, which makes it very easy to write native functions that deal with the stack and clean it up correctly in all cases. I would like to show you some non-fictitious code right now, but it's not ready yet ;-) But when it is, it will be part of Kyua. Stay tuned! And, to conclude: to make C++ code robust, wrap objects that need manual clean up (pointers, file descriptors, etc.) with small wrapper classes that perform such clean up on destruction. These classes are typically fully inlined and contain a single member field, so they do not impose any performance penalty. But, on the contrary, your code can avoid the need of many try/catch blocks, which are tricky to get right and hard to validate. (Unfortunately, this technique cannot be applied in, e.g. Java or Python, because the execution of the class destructors is completely non-deterministic and not guaranteed to happen whatsoever!)

December 27, 2010 · Tags: <a href="/tags/c">c</a>, <a href="/tags/cxx">cxx</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/lua">lua</a>
Continue reading (about 6 minutes)

Kyua: Design of the configuration system

Over a week ago, I mostly finished the implementation of the runtime engine for test cases of Kyua and, along the way, realized that it is imperative to write a configuration system right now before the configuration code becomes messier than it already is. To that end, I spent the last week working on a design document for the configuration system. Summarizing, the document describes what the requirements for the configuration files of Kyua are, what the possible alternatives to implement them are, and advocates the use of Lua — a tiny embedded programming language — to bring these configuration files to life. It is your chance to get involved in the early stages of the development of Kyua! :-) Take a look at the email to kyua-discuss asking for comments and feel free to join the mailing list ;-)

December 22, 2010 · Tags: <a href="/tags/kyua">kyua</a>, <a href="/tags/lua">lua</a>
Continue reading (about 1 minute)