The necessarily components-based architecture of free technology products

March 2016

The nice thing about free technology products is that they don't have an exclusive producer who calls the shots. It is often published under a brand name which has an owner, but the technology itself which this brand stands for, cannot be controlled by anyone, otherwise it wouldn't be free.

This is at the same time a big problem for free technology, because it hinders consistency. It leads to small underfunded projects which more often than not, don't cooperate with each other.

To build any high quality product, you usually need at least 10 full-time people for several years, and one of these people needs to be the boss (the Benevolent Dictator) who ultimately takes all the decisions. This is also how proprietary technology products are produced. But free technology is usually built by part-time contributors (sometimes even drive-by contributors, who just write one pull request and then disappear again), and volunteer or crowd-funded projects rarely have any full-time people for more than one or two years in a row, and many disappear again before they deliver a useful product.

To build a free technology product that can compete with proprietary products is therefore very hard. Ubuntu, Firefox, and ownCloud are exceptional success stories. If a lot of technology startups fail, the percentage of free technology projects that lead to nothing seems to be even greater.

I think this is because we approach the problem of building free technology in the wrong way. Most people who try to contribute (and I myself am no exception) start by blueskying their own big master plan for the ideal product, and then just start building prototypes, in the hope that the necessary engineers will voluntarily join the project along the way, once they realize its potential to disrupt. In practice of course, where even startups with seed funding to attract good engineers usually don't manage to build something which survives the competition, for free technology projects with more balls than bucks, it is almost impossible for their big master plan to succeed before the contributors shift their curiosity to some other project.

The solution? We should work on small components which other projects can reuse. A component is much easier to build successfully than an end-product. If it is still too much work, split it up into smaller components, and build those one-by-one.

It's of course not really that simple.

First, components can usually only be reused within one programming language or production technique.

Second, components are always interconnected, and trying to disconnect them often leads to complicated configuration files, so it takes a lot of work to integrate them into a consistent product.

Third, some projects require all their components to adhere to some format or standard, and it is hard to establish such a standard without someone else establishing a competing, slightly different standard. Maybe in a different programming language, or with different design decisions as its starting point.

Fourth, nobody can know about all the components that exist, so you often find out about an existing component after having already decided that you need to build it yourself.

And fifth, in most free technology projects, contributors want to have fun, and building something from scratch is more fun than investigating existing components that sort of do what you need, but never exactly, or not in the way you would have done it yourself.

But sometimes, standards like internet and web standards, and reusable components like debian packages, npm modules, Ruby gems, and Dockerfiles do work. And when this happens, many small projects produce compatible components, and usable free technology products can be built from them. For the IndieHosters project, we are now trying to break things up into smaller pieces, to develop the tools we need one-by-one instead of building one monolithic server image, and to package the applications we want to host for people into Docker images which adhere to certain restrictions, so that they all fit into the hosting tools in the same way.

Maybe these tool building and application packaging efforts are more important than the actual service we offer as an end-product. After all, other independent hosters will be able to reuse them without asking us for permission.

A monolithic unfinished end-product is unlikely to be reusable at all once it is no longer maintained. But if you develop your end-product out of reusable components, you are much more likely to contribute to the progress of free technology in the long run.

Projects can run out of fuel, but once reusable components have been created and published as free technology, this cannot be undone.