It's Alive!

I've been quietly hacking away on cmonster (née csnake). I like this name even more: I think it describes my creation better. If you thought preprocessors were horrible before, well...


What is cmonster?

cmonster is a C preprocessor with a few novel features on top of the standard fare:

• Allows users to define function macros in Python, inline.
• Allows users to define a callback for locating #include files, when the file can not be found in the specified include directories.
• Provides a Python API for iterating over tokens in the output stream.

cmonster is built on top of Clang, a modern C language family compiler, which contains a reusable, programmable preprocessor. At present, cmonster requires Clang 3.0 APIs, which has not yet been released. I have been working off Clang's subversion trunk.

I have just uploaded a binary distribution of the first alpha version (0.1) of cmonster to pypi. I have only built/tested it on Linux 32-bit, Python 3.2, and I don't expect it will work on anything else yet. If you want to play around with it, you can install cmonster using "easy_install cmonster" or by grabbing it off pypi and installing it manually.


Demonstration

Seeing is believing - how does this thing work? We'll ignore everying except for inline Python macros for now, because that's the most stable part of the API.

It is possible to define macros inline in cmonster, using the builtin "py_def" macro. For example:

py_def(factorial(n))
    import math
    return str(math.factorial(int(str(n))))
py_end

When cmonster sees this, it will grab everything up to "py_end", and define a Python function. It will also create a preprocessor macro with the function's name (as given in the py_def heading), and this macro will be directed to call the Python function. The Python function will be passed the argument tokens that the macro was invoked with. It can return either a sequence of tokens, or a string that will subsequently be tokenised.

Addressing Shortcomings

In my previous post about csnake I mentioned a few things that needed to be done. I have addressed some of these things in cmonster:

A way of detecting (or at least configuring) pre-defined macros and include paths for a target compiler/preprocessor. A standalone C preprocessor isn't worth much. It needs to act like or delegate to a real preprocessor, such as GCC.

 I have added support for emulating GCC. This is done by consulting GCC for its predefined macros (using "gcc -dM"), and using the new "include locator" callback feature. By setting an include locator callback on a cmonster preprocessor, you provide cmonster with a second-chance attempt at locating an include file when the file can not be found in the specified include directories. This method can be used to determine GCC's system include directories: whenever we can't find an include file, we consult GCC and add parse the output to determine the location of the file on disk. I intend to add support for more (common) compilers/preprocessors in the future. Namely, MSVC.

I had another crazy idea for (ab)using include locators: handling specially formatted #include directives to feed off-disk files into the preprocessor. Buh? I mean, say, the ability to pull down headers from a hosted repository such as github (e.g. #include <github.yourproject/header.h>), and feeding them into the preprocessor as in-memory files. Or generating headers on the fly (e.g. #include <myapi.proto>, to automatically generate and include Protobuf headers).

A #pragma to define Python macros in source, or perhaps if I'm feeling adventurous, something like #pydefine.

Support for inline Python macros has been implemented: see the "Demonstration" section above. It's unlikely I'll attempt to create a #pydefine, as it would be more work than it's worth.

A simple command line interface with the look and feel of a standard C preprocessor

The distribution now contains a "cmonster" script, which invokes the preprocessor on a file specified on the command-line. This will need a lot of work: presently you can't add user include directories or (un)define macros. Neither of these things are difficult to add, they just haven't been top priorities.


Future Work

Still remaining to do (and sticking out like sore thumbs) are unit-tests and documentation. Now that I've got something vaguely usable, I will be working on those next.

Once I've tested, documented and stabilised the API, I'll look at (in no definite order):

• Improvement the command line interface. Add the standard "-I", "-D" and "-U" parameters.
• Portability. Probably Windows first, since it's common and I have access to it.
• Emulation of additional preprocessors.
• Passing in-memory files ("file like" Python objects) to #include.

Controlling Remote Agents from Google App Engine

A month or so ago I was brainstorming ideas related to Google App Engine (GAE), as I had been wanting a reason to play with it for a while. One idea that stuck was connecting a remote Python process to GAE via Pushy, so we could either control GAE or GAE could control the remote process. I'm still working on the C/Python Preprocessor thingy, but I took a break from that this weekend to look into the possibility of a GAE Pushy transport.

So yesterday morning I signed up for an account, and started tinkering. I had been reading the docs already, and I figured there were a few possible approaches:

• The obvious: use HTTP. This has one major drawback in that it is inherently synchronous and wholly driven by the client. Moreover, GAE only allows request handlers around 30s to complete, so no kind of fancy long-polling is possible here.
• Channel API. This sounds like the right kind of thing to use, but it's aimed at interacting with Javascript in a webpage.
• XMPP. Huh? Isn't that for instant messaging? Exactly. The client (remote Python process) and server (GAE) are peers in XMPP, and either one can initiate sending messages to the other. Let's look into this a bit more...

So I did a quick search for Python XMPP libraries, and a few came up. I settled on xmpppy, but to be honest I didn't find any of them particularly compelling. The APIs are a bit clunky. Anyway, the approach I took was to have an XMPP handler in my GAE application create a persistent Pushy connection object associated with a pair of read/write files that wrapped the XMPP API. When an XMPP message came in, the application would extract the Pushy request from base-64 encoded body of the message, and return the result in a similar manner.

And it worked, but only just. I had to make a few hacks to Pushy to get all of this work. There were some oddities I had to work around, such as the "eval" built-in in GAE's Python not taking any keyword arguments. Unfortunately, I don't think this particular transport is very useful in the flaky state it's in at the moment. Also, it's not terribly valuable to build a transport to control a GAE application, since other APIs exist for that purpose (ProtoRPC, remote_api). More useful would be to have the GAE application control the remote process without the need for polling. I'll be looking into this further.

Le sigh

I've been coming across more problems with Boost Wave. The current ones blocking me are:

• A lack of an efficient way to conditionally disable a macro. The "context policy" provides hooks for handling macro expansions, and its return value is meant to control whether the expansion takes place. It doesn't work. I'll write up a bug report when I get some time.
• Wave isn't very forgiving about integer under/overflow. For example, the GNU C library's header "/usr/include/bits/wchar.h" has the following tidbit to determine the sign of wide characters, which Boost Wave barfs on:

#elif L'\0' - 1 > 0

I think the latter "problem" might actually be reasonable - I believe the standards say that handling of overflow is undefined, and preprocessor/compiler-specific. That doesn't help me much though. I could fix this by writing code to parse the expressions, which seems silly, or by passing off to the target preprocessor (e.g. GCC), which seems like overkill.

I'm going to have a look at how hard it would be to use LLVM/Clang's preprocessor instead. If that's a better bet, I may go that way. Otherwise, it might be time to get approval to send patches to the Wave project.

Inline Python macros in C

I had a bit of time today to do some more work on that which is soon to be called something other than csnake. I've added a couple of features:

• You can now define custom pragmas, providing a Python handler function. Unfortunately Boost Wave, which csnake uses for preprocessing, only provides callbacks for pragmas that start with "#pragma wave".
• Built-in pragmas and macros to support defining Python macros inline in C/C++ code.
• A __main__ program in the csnake package. So you can now do "python3.2 -m csnake ", which will print out the preprocessed source.

So for example, you can do something like as follows, entirely in one C++ file:

// factorial macro.
py_def(factorial(n))
    import math
    f = math.factorial(int(str(n)))
    return [Token(T_INTLIT, f)]
py_end

int main()
{
    std::cout << factorial(3) << std::endl;
    return 0;
}

This works as follows: py_def and py_end are macros, which in turn use the _Pragma operator with built-in pragmas. Those pragmas are handled by csnake, and signal to collect the tokens in between. When the py_end macro is reached, the tokens are concatenated and a Python function macro is born.

I'm intending to do additonal "Python blocks", including at least a py_for block, which will replicate the tokens within the block for each iteration of a loop.

There's one big problem with the py_def support at the moment, which is that the tokens go through the normal macro replacement procedure. I think I'll have a fix for that soon.

Name clash

Rats, looks like I didn't do enough homework on the name "csnake". Turns out there's another Python-based project called CSnake: https://github.com/csnake-org/CSnake/. Incidentally - and not that it matters much - I had the name picked out before that project was released. Now I need to think up another puntastic name.

C-Preprocessor Macros in Python

TL;DR: I've started a new project, csnake, which allows you to write your C preprocessor macros in Python.

Long version ahead...

You want to do what now?

I had this silly idea a couple of years ago, to create a C preprocessor in which macros can be defined in Python. This was borne out of me getting sick of hacking build scripts to generate code from data, but pursued more for fun.

I started playing around with Boost Wave, which is a "Standards conformant, and highly configurable implementation of the mandated C99/C++ preprocessor functionality packed behind an easy to use iterator interface". With a little skulduggery coding, I managed to define macros as C++ callable objects taking and returning tokens. Then it was a simple matter of adding a Python API.


The Result

What we end up with is a Python API that looks something like this:

import sys
from _preprocessor import *
def factorial(n):
    import math
    return [Token(T_INTLIT, math.factorial(int(str(n))))]

p = Preprocessor("test.cpp")
p.define(factorial)
for t in p.preprocess():
    sys.stdout.write(str(t))
sys.stdout.write("\n")

Which will take...

int main(){return factorial(3);}

And give you...

int main(){return 6;}

If it's not immediately clear, it will translate "factorial()" into an integer literal of the factorial of the input token. This isn't a very interesting example, so if you can imagine a useful application, let me know ;)

The above script will work with the current code, using Python 3.2, compiling with GCC. If you'd like to play with it, grab the code from the csnake github repository. Once you've got it, run "python3.2 setup.py build". Currently there is just an extension module ("csnake._preprocessor"), so set your PYTHONPATH to the build directory and play with that directly.

I have chosen to make csnake Python 3.2+ only, for a couple of major reasons:

• All the cool kids are doing it: it's the way of the future. But seriously, Python 3.x needs more projects to become more mainstream.
• Python 3.2 implements PEP 384, which allows extension modules to be used across Python versions. Finally. I always hated that I had to recompile for each version.

... and one very selfish (but minor) reason: I wanted to modernise my Python knowledge. I've been ignoring Python 3.x for far too long.

The Road Ahead

What I've done so far is very far from complete, and not immediately useful. It may never be very useful. But if it is to be, it would require at least:

• A way of detecting (or at least configuring) pre-defined macros and include paths for a target compiler/preprocessor. A standalone C preprocessor isn't worth much. It needs to act like or delegate to a real preprocessor, such as GCC.
• A #pragma to define Python macros in source, or perhaps if I'm feeling adventurous, something like #pydefine.
• A simple, documented Python API.
• A simple command line interface with the look and feel of a standard C preprocessor.
• Some unit tests.

I hope to add these in the near future. I've had code working for the first two points, and the remaining points are relatively simple. I will post again when I have made some significant progress.

Pushy 0.5 Released

After just a few short months since the 0.4 release, I am pleased to announce Pushy 0.5. As usual, this release is mostly bug fixes, with a couple of features to make life easier. Instructions on downloading and installing can be found here:
    http://awilkins.id.au/pushy/install.php

On top of standard bug fixes and features, I have made an effort to beautify some of the code, and speed everything up in general, based on code profiling. This will be evident if your Pushy program performs a lot of proxying: 0.5 performs up to ~2x  faster than 0.4 in my tests.


New Features

There are two new features in Pushy 0.5: with-statement support, and a simple "execute" interface.

With-statement support
Pushy connections now support the "with" statement. This is functionally equivalent to wrapping a Pushy connection with "contextlib.closing": it will close the connection when the with-statement exits. For example:

with pushy.connect("local:") as con:
    ...

Compile/execute interface
Previously if you wanted to execute a statement in the remote interpreter, you would have to first obtain the remote "compile" built-in function, invoke it to compile a remote code object, and then evaluate that with the "connection.eval" method. For example, to execute "print 'Hello, World!'":

code_obj = con.eval("compile")("print 'Hello, World!'", "<filename>", "exec")
con.eval(code_obj, locals=..., globals=...)

This is a bit unfriendly, so I thought it would be a good idea to add a simpler interface. There are two new methods: "connection.compile" and "connection.execute". The former will compile a remote code object, and the latter executes either a string (statement) or a function. Continuing our very simple example, we get the much simpler:

con.execute("print 'Hello, World!'")

Under the hood, this will do exactly what we would previously have had to do manually: remotely compile the statement and then evaluate the resultant code object. Now that suffices for a very simple use case like we have discussed above, but what if we want to execute a series of statements? Wouldn't it be nice if we could remotely execute a locally defined function? Well, now you can, using "connection.compile".

def local_function(*args, **kwargs):
    return (args, kwargs)
remote_function = con.compile(local_function)
remote_function(...)

The "connection.compile" method will take a locally defined function and define it in the remote interpreter. It will then return a reference to the remote function, so that we can invoke it as if it were a local function. This allows the user to define complex and/or expensive logic that will be executed entirely remotely, only communicating back to the peer when proxy objects are involved, or to return the result.


Bug Fixes

The most noteworthy bugs fixed in 0.5 are:

#738216 Proxied old-style classes can't be instantiated
Old-style/classic classes (i.e. those that don't inherit from "object") previously could not be instantiated via a Pushy connection. This has been rectified by defining an additional proxy type for old-style classes.


#733026 auto-importer doesn't support modules not imported by their parent
Previously, importing a remote submodule via "connection.modules" would only work if the parent of the submodule imported it. For example, "connection.modules.os.path" would work since os imports os.path. If the parent does not import the submodule, Pushy would fail to import the module. This has been fixed in 0.5; remotely imported modules will provide the same sort of automagical importing interface as "connection.modules".


#734311 Proxies are strongly referenced and never deleted; __del__ isn't transmitted to delete proxied object
This is the most significant change in Pushy 0.5. Since inception, Pushy has never released proxy objects, meaning eventual memory bloating for objects that would otherwise by garbage collected. As of 0.5, Pushy will now garbage collect proxies by default. Every so often (by default, 5 seconds), Pushy will send a message to the peer to let it know which proxies have been garbage collected. This allows the peer to release the proxied objects, and reclaim resources.


#773811 Daemon/socket transport is slow
Due to the many small messages sent back and forth by Pushy, the "daemon" transport was found to be quite slow in stress testing. This is due to Nagle's Algorithm, which delays sending network packets so that they can be combined. This has a considerable impact on the latency of Pushy's operations, and so it is now disabled by Pushy.

Enjoy!