Recently I found myself looking for a particular type of build automation tool, only to discover that such a tool does not exist. That got me thinking about how such a tool might be implemented. After tossing ideas around in my mind, some basics started coming together, and I soon found myself building the tool I thought was missing.
What I ended up with was Cobble: a fast, flexible, monorepo-aware build automation tool. I’ll explain a little about my journey towards deciding to build something new and Cobble’s philosophical approach to build automation.
The Gap in Build Automation Tools
In a personal project of mine, I have some Rust code and some Typescript code, which both have their own build processes, and the build artifacts from those source files get combined together into an Electron application. Specifying the build tasks as npm scripts was a natural way to go given that it was an Electron application, but npm scripts are fairly limited in what they can do. They have no way to specify dependencies between scripts, (besides directly invoking one script in the command for another script,) and no support for determining whether a script’s inputs have changed.
This got me looking at other options for defining the build commands for my project. GNU Make is of course an option, but I’ve tended to avoid using Make since it has a somewhat archaic-feeling language for defining rules. I discovered Doit a little while back, which is a brilliant little tool, though I wasn’t keen on introducing a Python environment dependency to my project.
My sense of a need for a new kind of build automation tool was magnified as I participated in discussions at work related to what build system to use for our Python projects. We were migrating our projects towards "monorepos", and we had already moved all of our C++ code into a monorepo based on Bazel. The next question was what to do with our Python code. I didn’t like the idea of moving our Python code into the Bazel-based monorepo because IDE integration would suffer. There’s a Python-focused monorepo tool, Pants, though this would require a significant migration effort, and IDE integration with Pants was still worse than what we would otherwise have. As we were using Poetry and Doit for our Python projects, I decided to use some extensibility features in doit to support a multi-project repository and allow dependencies across projects. While hacking that together, I thought, “Wouldn’t it be great if there was an easy-to-use, general-purpose build tool like Make or Doit, but that supports multi-project repos as a first-class feature?”
An Effective Foundation for a Build Automation Tool
At first, the idea of creating a tool like I had imagined was just a nice thought. Eventually, though, it occurred to me how convenient it would be to have a build automation tool that did not depend on any kind of script environment, and how languages like Rust and Go are great for creating cross-platform, self-contained, native applications.
Another one of the challenges in making a build automation tool is providing a language for defining the build tasks and their dependencies. Doit is Python-based, and its configration language is also Python. Make provides its own language, as does Bazel in the form of Starlark. Creating a new language is always a tall task. One problem with building a build automation tool with Rust or Go is that neither of those languages have a built-in scripting language that can be used as the project definition language for the tool. Lua, however, is great as an embedded language, and seemed like a good candidate for being the project definition language for a Rust-based build-automation tool.
There were also some concepts I was tossing around in my mind, one of which was the virtual environment. Many language platforms support some kind of environment isolation, like the node_modules directory for Npm, Python virtual environments, or Docker for general-purpose isolation. Ideally, the tools for building and analyzing the project are made available in the isolated environment, which allows the project to be as self-contained as possible. I was interested in seeing if I could add concepts into a build automation tool that would treat isolated environments as first-class concepts.
After tossing these ideas around in my head for a while, I inevitably started building a new build automation tool.
Philosophy: Simple, Flexible, and Permissive
One of the potential pitfalls of using a tool like Doit or Make is that if you don’t define the dependencies and artifacts of rules correctly, you can end up with an incorrectly behaving build. Monorepo tools like Bazel and Pants address this issue with the concept of hermeticity, which enforces that any build rule defined only has access to the files declared as dependencies. The advantage to hermeticity, especially for large monorepos, is that you have an extra guarantee on the correctness of your build rules. The disadvantage is that achieving that hermeticity comes at the cost of not being able to run those rules within the project directory tree itself.
Cobble’s approach is to not worry about hermeticity,-- If you have a monorepo that’s complex enough to where you would benefit from hermeticity, then Bazel or Pants may be the best option for you-- but, by not worrying about hermeticity, it is able to run build rules directly in the project’s directory tree. You can create projects in your monorepo that behave just as well as independent projects as they do as members of a monorepo. IDE integration is easier to achieve using the out-of-the-box language integrations that most IDEs provide.
Cobble also aims to be simple. When creating a Bazel monorepo, people generally rely on existing build rules defined and maintained by the community. Diving into the Starlark language and defining a custom build rule tends to be an “expert mode” activity. Cobble aims to not have an "expert mode". Defining a custom rule is easy and straightforward. Given knowledge of how to write Lua and a few pages of documentation, someone should be able to write whatever rules they need to define their build rules, no matter what they are.
Cobble is also flexible. It is not aligned with a specific language: similar to Make and Doit, as long as you can write the command to execute for a build rule, you can create the build rule. This has the nice feature of allowing you to combine projects based on any language platform without having to worry about how well your build automation tool supports it.
Broad Use Cases
I think these philosophical approaches make Cobble ideal for small monorepos that combine a handful of projects, but can also scale to a large number of projects. At some point, a monorepo may become complex enough that Bazel or Pants makes sense, but until then, Cobble can give you the monorepo power you need without sacrificing broad IDE support.
Currently, Cobble adoption mostly includes my own personal projects, but I imagine it would be useful for a broad set of use cases. If you think Cobble could be useful for something you’re working on, I’d like to hear about it!
This is a companion discussion topic for the original entry at https://saltytron.com/posts/2024-09-16-cobble/