A Parastatal Problem

To reproduce something close to this figure, we would first need to know what constitutes "just [getting] started". Is this with one user just after the server has started, or with many users after the server has run for some time? Does this include the memory used by the SQL server and the rest of the operating system? Providing such a shocking figure, without an explanation of how it was observed, is a very dirty way to obtain a following. This figure should be explained further, or removed from the announcement.

Claiming that the minimum resource usage is the average is obviously incorrect. However, we now should ask ourselves if Mastodon was designed for high performance with enough resources⁵, or efficiency. An intuitive relation between resource consumption and performance is that providing more resources increases performance (and efficiency is the ratio of consumption and performance). Some strategies can improve performance if additional resources are present, but do not necessarily decrease the minimum efficiency with fewer resources.

One such way to utilise otherwise free memory is to cache values in memory, allowing them to be used without recomputing them or reading them off a relatively slow disk or network. The Linux kernel used to be too up-front about caching, and would report memory used for caching as "used", even though it could be released to programs very quickly. This caused people to think that the kernel was wasting their memory, and someone else to write a website describing the situation.

The ZFS file system also caches other data to improve performance, with similar effects at first sight; but still works (with reduced performance) with less memory available for caching.

Linux is borrowing unused memory for disk caching. This makes it looks like you are low on memory, but you are not! Everything is fine!

https://linuxatemyram.com

ZFS uses different layers of disk cache to speed up read and write operations. Ideally, all data should be stored in RAM, but that is usually too expensive. Therefore, data is automatically cached in a hierarchy to optimize performance versus cost.

The ZFS Wikipedia page

This telling of the story should convince you that the kernel was not wasting memory, and that it was a simple misunderstanding caused by confusing terminology⁶.

With this in mind, we may now ask if the Mastodon instance was caching some media with the 2GB figure, and thus running "faster" by having to read from the disk less often. Thus, we cannot decide that Mastodon is inefficient solely from this memory usage figure.

In contrast to this 2GB memory figure, the Parastat developers believe they can support 1,000 users (which is also poorly defined; are they all active at the same time, or are none online, etc) with 64 megabytes of memory; about a third of the memory used by Ben's one-user server. This is roughly a 3000-fold increase in user/memory density⁷; is it possible to do so without disposing of performance techniques like caching?

This 2GB memory figure is unlikely to be reproduced, and it misrepresents the minimum a Mastodon server could work with, which is an inappropriate way to gain support for a project. It also may show the developers are not entirely familiar with optimisation techniques such as caching; nonetheless, we want the developers to explain this figure better.

Much of the content we have seen about Parastat has been about the user-facing design, but we believe programmer- and operator-facing design may be neglected from using the C programming language.

It is evident that writing a stable C program is quite difficult without much testing and deliberation. Stability, of course, relates to the hosting costs that the developers wish to minimise, as a more stable server does not require its operators to take time restarting and checking that it works. Take John Rose's experience with Sun's server software, as an example of C servers being difficult to keep running:

There are some network things with truly stupendous-sized data segments. Moreover, they grow over time, eventually taking over the entire swap volume, I suppose. So you can't leave a Sun up for very long. That's why I'm glad Suns are easy to boot!

John Rose to sun-users, from The Unix Hater's Handbook

We would be pleasantly surprised if the Parastat developers were able to achieve better uptime than the small professional programmer group at Sun⁸. Of course, many projects written in C are fairly safe and stable, such as the Apache web server⁹, but without many person-hours spent reviewing the code and analysing problems like John's memory leaks, it is unlikely that Parastat will achieve excellent stability. To give some sense of scale of how much review should be done, Ravenbrook reviewed the Memory Pool System, a well-deployed project that is safe and is written in C, at a rate less than 10 lines/minute by four to six people¹⁰, on top of any other cursory analysis done while developing the product.

John has his own opinion on how the software could have been greatly improved:

But why should a network server grow over time? You've got to realize that the Sun software dynamically allocates very complex data structures. You are supposed to call "free" on every structure you have allocated, but it’s understandable that a little garbage escapes now and then because of programmer oversight. Or programmer apathy. So eventually the swap volume fills up! This leads me to daydream about a workstation architecture optimized for the creation and manipulation of large, complex, interconnected data structures, and some magic means of freeing storage without programmer intervention. Such a workstation could stay up for days, reclaiming its own garbage, without need for costly booting operations.

That doesn't sound very much like C, but it does sound like what we would want, in order to keep the system running.

It will also be tricky to ensure all errors are handled appropriately (or at least that the server shuts down cleanly) in C, due to the non-presence of a exception or condition system. Recovering in a concurrent situation is still tricky, and how to approach it best is not immediately obvious, but C does not do any favours there.

There is also a desire to produce very readable code by the community:

i take a literate stance; i believe source files should be readable by people who aren't programmers (which DOES mean repeating yourself); but that's also a LOT of work

A Parastat Discord participant

I take your stance, but usually from the perspective of "I'm gonna forget the mental state I wrote this in and I wanna preserve that and I don't want to have to read the code to understand what it does when I inevitably forget"

The frontend developer

This is quite achievable using Elm¹¹, but less so using C, which by nature of being a low level language, "requires attention to the irrelevant", severely limiting readability. Having fewer means of abstraction¹² also complicates understanding the behaviour of some code at a glance, further complicating matters. It seems very unlikely that casual programmers would be able to begin analysing and modifying Parastat server code quickly, let alone have non-programmers be able to read the code.

Of course, crashing is a fairly acceptable thing to do when things go wrong. Being able to construct a remote code execution exploit from a memory bug is much more dangerous, and the effects could be much less obvious without further investigation. The Chromium project reports 70% of security bugs are due to memory safety problems, and suggests that the most thorough way of avoiding those bugs is to avoid using unsafe languages like C++ or C. The Parastat developers are doing exactly the opposite of this.

While not a security vulnerability per se, the claim that the developers want to produce a new, more private, protocol and maintain backwards compatibility for the less private ActivityPub, will be difficult to fulfil. Federating with an ActivityPub server would then imply that the other server is not going to "leak" user information in any way, and transitively, servers federating with those may also have to be as private. It is thus unlikely that a server can properly benefit from the new, more private, protocol and use the old protocol simultaneously.

Most of my concerns could be addressed with a source code release, as well as some more; we would be able to see for ourselves that we can indeed host 1,000 users in 64MB of memory, and that the server is stable and performant. However, we have been told the code isn't ready to be read, and that it is in our interest to wait until it is.

It would be preferable to release messy code, as then we can check if these claims are true, and experienced programmers can even possibly help clean up the code, leading to a readable product sooner. At the very least, we want to know if any of what we've been told about the project is true.

The other issue is that we have to go through administrators to get anything done. In the worst case scenario, the administrators don't act upon any reports they receive, possibly due to intoxication by "free speech" ideology, or by just being busy with other things. Typically, there is some latency between someone notifying a administrator and an adminstrator acting upon it, in the order of hours. This, again, also "costs" people, as they have to take some time to process all the reports; moreso when they have to wade through some unpleasant content.

With many more normal users than administrators, it should be possible to leverage the flags produced by users with software that tries to filter out content which it believes a given user would probably flag. This technique already exists, and is called collaborative filtering. While it is usually used in unfortunate places such as targeted advertising, the designs produced should be more than enough to effectively automate most of the role of an administrator.