Arch Linux¶
When we set out to build uberspace 8, we analyzed a number of different linux distributions for their suitability. We ended up with a seemingly unlikely choice for our servers:
✨ Arch Linux. ✨
Due to its agile, rolling release reputation Arch Linux has a reputation of being "unstable". However, it works just fine for our use case and we'd like to explain the details in this article.
Stability and Arch Linux¶
Many of us use Arch Linux personally, mostly on their laptops, but we also know of people running it on their servers. Anecdotally, we have not encountered any stability issues, beyond laptop hardware compatability problems to be expected with any or most Linux distributions. But since uberspace 8 is hosted within VMs exlusively, hardware support is not a factor.
Package updates are usually swift and of high quality. However, breaking changes
sometimes need to happen because of changins in the packaging structure or
upstream changes. In that case, breaking changes are announced ahead of time via
arch-announce@lists.archlinux.org and the Arch Linux news site.
In such changes the maintainers always provide a way forward. This may include
configuration updates as well as alternative packages, for example dovecot23
to help people struggling with the 2.4 update published in late 2025.
Security issues within packaged software are published on security.archlinux.org, which may not be on par with the Red Hat CVE Database, which we used to use. However, it is defenitly useful and constantly improving. We also build a lot of software ourselves, so we need to keep an eye out for security issues ourselves anyway.
Testing¶
Compared to a regular Linux server, uberspace 8 is a very involved setup. There are multiple configuration management systems at work, many concurrent users, APIs, a lot of inconsistent use, etc. A breaking change or bug also affects quite a few people all at once, so we do our best to ensure everything works before we press the big, red DEPLOY button.
Our setup is tested on two levels: software tests and integration tests, both of which are automated.
Software Testing¶
Software in use on uberspace 8 is mostly written in python and tested using
classic tools like py.test. We aim for a test coverage of >95% and
run tests both manually and automatically in CI systems. Deployments only happen
once the tests pass. After deployment, many tools like our API have basic E2E
tests ensuring the deployed instance works as intended. During operation,
systems are monitored in detail using various metrics. If something goes wrong,
our on-call team is notified.
Inegration Testing¶
The actual uberspace 8 setup is also tested automatically. During development we
deploy testing VMs with the newest development version and make sure everything
works as intended. This is mostly done using a tool we wrote ourselves:
shellinspector. Shellinspector opens up an actual SSH
session to the server, executes the commands given and checks for the proper
output as well as command return codes. It has various features, which are not
documented here, but feel free to read the docs, if you are
intersted.
To see it in action, take a look at one of our tests making sure basic httpd features like serving static content work:
---
fixture: user-create
---
$_ echo "Hello uberspace!" > /var/www/virtual/${SI_USER}/html/index.html
% http --pretty=none --header :81 Host:${SI_USER_DOMAIN}
Server: Apache
% http --pretty=none --body :81 Host:${SI_USER_DOMAIN}
Hello uberspace!
$ echo 'RewriteRule \.html$ - [R=410,L]' > ~/www/html/.htaccess
% http --pretty=none --body :81/index.html Host:${SI_USER_DOMAIN}
Error 410
$ echo 'TriggerError' > ~/www/html/.htaccess
% http --pretty=none :81/triggererror Host:${SI_USER_DOMAIN}
500 Internal Server Error
% rg -F "/var/www/virtual/${SI_USER}/html/.htaccess: Invalid command" /readonly/${SI_USER}/logs/error_log_apache
TriggerError
The test first adds a simple index.html to the asteroid and then proceeds to
make HTTP requests, checking for headers and the actual content. It then
proceeds to test that configuration using .htaccess works and that the error
log functions as intended.
All features within uberspace are tested in this fashion and all of those tests run before new features, bugfixes or OS updates are actually deployed.
Updates¶
While we trust that Arch Linux packages software properly, configuration and features will change more frequently than on a more "static" distribution like RHEL or Debian. We would like to include those changes within uberspace 8, however, we might need to apply some changes to our setup before we can.
To resolve this, we don't apply Arch Linux updates right away. Instead, we take a few precautions: Once a week we fetch the versions available in the repos and freeze them. This set of packages is then tested using the setup outlined above. If the tests succeed, the state is deployed to a few internal and external early bird servers. If and only if all those prerequisites are met, the versions are deployed to all production servers. Sometimes it turns out that the new packages are incompatible with our setup. In that case, we first fix underlying issue by patching our setup accordingly. Afterwards updates are deployed.
This works well for regular updates, however there is one class of updates which needs rapid deployment: Security issues. Once we become aware of a security issue within software installed on our servers, we identify the version which contains the fix and run the entire process on overdrive. The affected package update is added to the production repository state right away, tested, deployed to an early bird and finally to all servers. We are aware that this is a form of partial updates, which are not supported by arch linux. However, for software which is not used to build other packages (e.g. nginx vs. glibc) this is fine in isolated cases when considering dependencies between the various packages.