Before the weekend I announced a downtime/maintenance windows to upgrade PostgreSQL from v15 to v17 – because of the Debian upgrade from Bookworm to Trixie. After some tests with a cloned VM I decided use the quick path of pg_ugradecluster 15 main -v 17 -m upgrade –clone. As this would be my first time to upgrade PostgreSQL that way, I made several backups. In the end everything went smooth and the database is now on v17.

However, there was also a new Proxmox kernel and packages, so I also upgrade one Proxmox node and rebootet it. And then the issues began:

But before that I also encountered an issue with Redis for Mastodon. It complained about this:

Unable to obtain the AOF file appendonly.aof.4398.base.rdb

Solution to this was to change redis configuration to autoappend no.

And then CephFS was unavailable again, complaining about laggy MDS or no MDS at all, which – of course – was totally wrong. I search for solutions and read many forum posts in the Proxmox forum, but nothing helped. I also read the official Ceph documentation. After a whole day offline for all of the services to my thousands of users, I somehow managed to get systemctl reset-failed mnt-pve-cephfs && systemctl start mnt-pve-cephfs again. Shortly before that I followed the advice in the Ceph docs for RADOS Health and there especially section about Troubleshooting Monitors.

In the end, I can’t say which step exactly did the trick that CephFS was working again. But as it seems, I will have one or two more chances to find out, because only one server out of three is currently updated.

Another issue during the downtime also was that one server crashed/rebooted and didn’t came back. It hang in the midst of an upgrade at the point of upgrade-grub. Usually it wouldn’t be a big deal: just go the IPMI website and reboot the server.

Nah! That’s too simple!

For some unknow reason the IPMI interfaces lost their DHCP leases: the DHCP server at the colocation was not serving IPs. So I opened a ticket, got some acknowledgement from the support but also a statement “maybe tomorrow or on Monday…”. Hmpf!

On Sunday evening I managed to bring back CephFS. As said: no idea what specific step did the trick. But the story continues: On Monday before lunch time the IPMI DHCP was working again and I could access the web interfaces again, logged in…. and was forcefully locked out again:

Your session has timed out. You will need to open a new session

I hit the problem described here. But cold resetting the BMC didn’t work. So still no working web interface to deal with the issue. But on my phone I got “IPMIView” as app and that still worked and showed the KVM console. But what I saw there didn’t make me happy as well:

The reason for this is apparently the crash while running update-grub. Anyway, using the Grub bootloader and selecting an older kernel works fine. The server boots, Proxmox is showing the node as up and…. the working CephFS is stalled again! Fsck!

Rebooting the node or stopping Ceph on that node results immediatedly in a working CephFS again.

Currently I’m moving everything off of Ceph to the local disks of the two nodes. If everything is on local disks I can work on debugging CephFS without interrupting the service for the users (hopefully). But this also means that there will be no redundancy for Mastodon and mail.

When I have more detailled information about possible reasons and such, I may post to the Proxmox forum.

Overview

Shortly after the update from Debian 12 “Bookworm” to Debian 13 “Trixie” I also updated the Debian-based Proxmox installations. And then the issues began and I had sleepless nights, many downtimes and frustrated users, because the usually rock-stable Ceph storage became unstable. The OSDs went off the net, the Ceph Filesystem got degraded and everything became slow. The Ceph Filesystems (CephFS) also holds the mail storage as well as the shared storage (code & data) for my Nerdculture.de Mastodon instance.

Just to outline what I’m about to discuss, here’s the cabling plan for my 3-node hyperconverged Proxmox server setup:

Basically you see 3 types of connections:
1) Internet connection to the colocation switch
2) Internal Proxmox connections between the 3 nodes
3) Internal Ceph connections between the 3 nodes

The internal, directly wired connections are necessary because the colocation provider have had no additional Copper 10 Gbit/s ports (10GbaseT) available. So I had to wire up all those by directly attached patch cables.

Ceph has a backend and frontend network. You can run Ceph with just one network, but well, then Proxmox and Ceph would need to share the same network and access to Ceph would slow down when Virtual Machines (VMs) were migrated between the nodes.

What happened the last weeks?

The problem started, as said, after updating the Proxmox nodes. On Sept. 24th the first outage happened. You can read my summary here. Somehow the network Ceph connection between didn’t work anymore. The setup that was running for years now didn’t work anymore. The Ceph backend network couldn’t see the disks (OSD) any longer, so added manual routes between the nodes instead of relying on FRR with OSPF (a dynamic routing protocol). This solved the problem back then.

The next issue happened a week later on Oct. 2nd: since the last issue a week before I discovered that CephFS was awkfully slow. Loading the mails took like 10 seconds instead of being instantly there. So I tried to find the reason. My best assumption was: the WD Red SA500 2 TB that are holding the WAL/DB for the Ceph cluster are reaching the wear level end. These SSDs are not made for that kind of workload.

Another reason might be that the Ceph frontend network, which uses the Proxmox network, because the VMs need to access the Ceph frontend, is a OpenVSwitch bridge and traffic from Baldur is using the link via Pepper to Gate, for example, instead of using the direct connection, which adds some latency and reducing bandwidth.

And with that being said, this was the reason why there was an outage yesterday as well:

For the Ceph backend network, I use an internal Linux bridge in Proxmox to hold the IP for the Ceph backend on each node. Then there are two network cards, as described in the drawing. On the link between the nodes I configured Point to Point connections and added a route for the direct neighbor with a lower metric and a route for the other node with a higher metric. The other link vice versa. This works pretty well for the Ceph backend.

Yesterday I wanted to deploy those changes as well to the Proxmox network and get rid off of that Layer 2 network via OpenVSwitch. Settings this up in the operating system was no big deal, but unfortunately Proxmox complained later about the nodes having more than one IP. And there the issue started again.

But there was another problem, because even when reverting that network change, the Ceph cluster had issues again and couldn’t find its peers. I restarted services, rebooted nodes, etc… whatever to make it work again. But still OSDs were failing, coming online again, and failing again. The service mnt-pve-cephfs.mount was not able to mount the CephFS and thus CephFS was not available for the VMs and therefore the services like mail and Mastodon failed to load as well as nearly all services that need SSL certificates which – you guess it! – lies on CephFS as well. No CephFS available, no SSL certs and no service.

But why was it not possible to mount the CephFS on the Proxmox host nodes? I had a look onto the syslog and other logs while restarting services, but the output was that much and fast, that I couldn’t find the root cause for it.

At one time I was lucky and spotted this line:
2025-10-12T01:05:28.209050+02:00 baldur ceph-mgr[40880]: ERROR:root:Module 'xmltodict' is not installed.
And the solution was as simple as searching the web for that error message and stumble across this post in the Proxmox forum:

I was able to correct this with python3-xmltodict, that resolved one issue
So, after installing that package the Ceph cluster was happy again and Proxmox could finally mount CephFS with restarting mnt-pve-cephfs.mount.

Then it was just a matter of restarting VMs and services and finally Mastodon on https://nerdculture.de/ was available again as well as mail started to come in.

Lessons learned

For one I’m going to buy new SSDs for WAL/DB in Ceph, most likely Micron 5400 MAX. This should bringt he latency down with Ceph and increase the overall speed, because data is only written for the client, when all 3 nodes have written their data to disk. The slowest node or disk is the resulting speed of Ceph. WD Red SSDs might be good enough for NAS systems, but for constant disk writes like in the case of WAL/DB in Ceph, they seem to hit their limit rather soon.

Another thing I could improve is the network. It is a complex setup and prone to errors. I need to talk to the colocation if I can get 6x 10 Gbps ports on their switch or if I can bring in my own switch and what that would cost?

Speaking of: what switch would you recommend?

Für dich und für mich.

Wer rechtsradikale Parteien wählt, macht das in voller Absicht. Da kann man nichts beschönigen oder verharmlosen. Da sehe ich auch nichts, was unter “Protest” laufen könnte. Aus Protest kann man die “PARTEI” wählen, solange sie unter, sagen wir mal, 5% bleibt. Oder die Tierschutzpartei. Aber nicht die angebliche Alternative.

Natürlich kann man die derzeitige Politik kritisieren. Gerade beim Kanzler ist das berechtigt. Oder bei den Ministern der FDP, die lieber Klientelpolitik betreiben als konstruktiv in der Regierung mitzuarbeiten. Mag sein, dass der Klimawandel in den letzten Jahren durch Corona, Ukraine-Krieg und im Gaza-Streifen in den Hintergrund gedrängt wurde.

Aber wenn etwas alternativlos ist, dann der Kampf gegen den Klimawandel. Dazu gehören die Energiewende und auch Wärmepumpen. Wie drängend diese Probleme sind, konnte man dieser Tage in Süddeutschland bei den Überschwemmungen sehen. Die Regenfälle, die zu den Überschwemmungen führten, sind quasi das neue Normal und nicht die Ausnahme.

Wir haben keine Zeit, den notwendigen Wandel auf die lange Bank zu schieben. Das macht das Problem nur größer, nicht kleiner. Da helfen auch keine E-Fuels für den Porsche von Hr. Lindner. Wenn das FDP-geführte Verkehrsministerium nicht den CO₂-Ausstoß reduziert, dann sind Wärmepumpen das kleinste Übel.

Der Wandel ist für einige Gegenden oder Branchen natürlich schwieriger als für andere. Es wird Gewinner und Verlierer geben. Zu den Verlierern gehören zum Beispiel (zwangsweise) die Branchen und Regionen, die bisher auf fossile Brennstoffe setzen. Hierzu gehören auch der Bergbau von Stein- und Braunkohle. Während vor einigen Jahren z.B. schon die letzte Steinkohle im Ruhrgebiet gefördert wurde, wird in der Lausitz die Braunkohleförderung spätestens 2038 beendet. Das ist für das Klima viel zu spät.

Ein Strukturwandel ist dringend notwendig. Die Leute wollen ja auch Arbeit haben. Und gerade die Bergleute sind ja bekanntlich kreuzbrave Leute, die teilweise Jahrhunderte lang Grundlage für den wirtschaftlichen Erfolg waren. Das ist ein identitätsstiftendes Selbstbewusstsein für Generationen. Daher kann ich verstehen, dass einige Menschen deshalb Zukunftsängste haben.

Aber es ist Aufgabe der Politik, die Notwendigkeit des Wandels zu vermitteln und für den notwendigen Strukturwandel zu sorgen. Das sind komplexe Probleme. Komplexe Probleme lassen sich aber nicht durch simplen Populismus lösen. Deshalb ist die sogenannte Alternative auch keine Alternative, sondern im Gegenteil ein Garant dafür, dass es zukünftig weitaus schlimmer werden wird, als es notwendig wäre, wenn wir uns auf den Wandel einstellen und in aktiv gestalten.

Die Augen zu verschließen und zu hoffen, dass alles nicht so schlimm werden wird, hat noch nie funktioniert.

Also beschwert/bedankt euch bei ihm, wenn ich hier wieder meine Gedanken und Dinge raushaue… 😉

Ich hatte mich nach meinem vorherigen Artikel nochmal beim Bausenator erkundigt und seine – durchaus nachvollziehbare – Erklärung war:

Hallo Herr Jürgensmann,
Krankheit, Urlaub etc.
Es ist mir auch peinlich. Aber…
Nächste Woche.

Und in der Tat: nachdem die Schulferien nun in Mecklenburg-Vorpommern beendet sind, wurde die neue Regelung auch prompt umgesetzt.

Jetzt müssen sich nur noch die Autofahrer an das neue Tempo gewöhnen und langsamer fahren. Ob das allerdings eine deutliche Lärmminderung zur Folge hat, bezweifle ich ein bißchen, da der Asphalt halt sehr laut ist und das vermutlich erst in ein paar Jahren mit der Sanierung der Parkstraße geändert werden kann.

Wir werden sehen.

Im Ortsbereirat wurde am 14. Juni 2022 das Thema besprochen und auch in den Medien wurde eine Umsetzung von Tempo 30 in der Parkstraße und anderen Straßen in Warnemünde in Aussicht gestellt:

Die Hansestadt Rostock reagiert auf das wachsende Verkehrsaufkommen im Ostseebad Warnemünde: Noch bis Ende Juni soll in der gesamten Parkstraße das Tempolimit von 50 auf 30 Kilometer pro Stunde herabgesetzt werden.

Am 20. Juli 2022 habe ich dann mal den zuständigen Bausenator Holger Matthäus per Mail angeschrieben und nach dem Stand der Umsetzung gefragt, da ja das 2. Quartal mit Ende Juni abgelaufen war, aber immer noch nichts passiert ist.

Ein paar Tage später, am 26. Juli 2022, antwortete der Bausenator auch (vermutlich) persönlich und schrieb:

mir wurde nun endlich heute die Verkehrsrechtliche Anordnung zu T30 vorgelegt.
Ich habe die Anordnung unterschrieben.
Mit der Inkraftsetzung vor Ort können Sie die nächsten Tage rechnen.

Bausenator Holger Matthäus am 26.07.22 per Mail

Stand heute, d. 15. August 2022, sind die Zusatzschilder, die Tempo 30 nur für LKW und Busse vorschreiben, immer noch nicht demontiert und damit die Anordnung des Bausenators immer noch nicht umgesetzt worden.

Als Anwohner, die bei tropischen Temperaturen gezwungen sind, die Fenster zwecks Kühlung zu öffnen, ist das Problem des lauten Verkehrs gleich doppelt schwer zu ertragen: sowohl tagsüber als auch nachts, sind wir durch Verkehrslärm betroffen und leiden darunter. Insbesondere auch deswegen, weil der verwendete Asphalt auch besonders laut ist.

Ich werde sicherlich jetzt nochmal beim Bausenator nachhaken und nach einem festern Zieltermin für die Umsetzung warten.

Interesting in this is, that for some time xmpp.social seemed to be the domain of choice for many users, maybe because of “xmpp” and “social” in the domain name – or because it is easier to name it than “hookipa” with “double-oh” and “kay”… who knows…

So, the user counts on xmpp.social were rising in a steeper curve than for hookipa.net, so much that I even considered to move over to xmpp.social for the main domain of that website.

But then something happened: the curated list of XMPP providers, which is now available under https://providers.xmpp.net. Since then some client software apps included the list, e.g. uwpx, and the user count was rising on hookipa.net – while it slowed down on xmpp.social.

Here you can see that the red curve of xmpp.social was for a long time above the blue curve of hookipa.net. Approx. in August 2021 something changed and hookipa.net was steadily increasing its user count. After a year, roughly in August 2022 hookipa.net surpassed the user count of xmpp.social.

Reason for this might be that hookipa.net is listed under Class A provider on providers.xmpp.net. Class A just means that some certain criteria is met like open registration and such. It doesn’t say anything whether or not it is a well-operated service. Well, at least not directly.

You can also look at the graphs on https://the-federation.info about the increase on hookipa.net and the stagnation on xmpp.social. There you can see the difference between a service that is listed on such a providers list and one that isn’t listed. Both domains are equally operated on my server and when you visit the Hookipa website you can register for both domains. But currently a downside (for me) of providers.xmpp.net is, that you need to provide the data for your classification on a website. Hookipa has a website, xmpp.social not, because it redirects to hookipa.net. Therefor xmpp.social is not included on providers.xmpp.net and thus is not gaining that many new users than hookipa.net.

I find that quite interesting, how registration counts are shifted from one domain to the other over time and what leads to that shift.

If you want to know which XMPP apps make use of the providers list, you can have a look at https://providers.xmpp.net/apps/.

For years I’m self-hosting lots of stuff and also for family, friends and others. The user management was either standalone for each service like Nextcloud, Mastodon, Friendica, XMPP, or based on the mail auth backend on PostgreSQL, for example by authenticating against Dovecot IMAP server. This became complex over time and I was looking for a centralized auth backend. Basically this means: LDAP backend.

For that I took a look into 389ds, FusionDirectory and Univention Corporate Server (UCS). With 389ds I had installation or setup issues. FusionDirectory was way better and easier to use, but also very complex. In the end I went with UCS because of the UI experienced, the ease of use and having a working self-care portal for the users. And UCS comes with some kind of App Center with some nice apps in it that are preconfigured for the LDAP directory. So, really easy to use and to get started.

So, last year (even before I got employed by Univention) I migrated my users from PostgreSQL backend to UCS and LDAP backend. The migration was smooth and nice and worked like a charme. Only issue: things like my public XMPP server, Friendica or Mastodon are still using open registration and therefor internal auth backend. It will be much more difficult to migrate these with an existing user base of 3000-4000 users in total. So, having LDAP as user/auth backend is nice, but you should consider this at an early stage and not when you already have a plenty of services and users on your services. 😉

However, it’s even nicer when you can use Single Sign On (SSO) with your apps. I got a taste of SSO when I was doing Cisco UC stuff and liked it that I don’t need to enter my credentials over and over again. SSO is also possible with UCS, so I wanted to give it a try on my own server.

The official HowTo on setting up SAML SSO basically covers the process of setting it up, but my impression was, that this process can be made better, less error-prone and more reproducible by automatting the setup.

So I wrote in my spare time a small shell script to follow the instructions from the official HowTo and after many tests and enhancements, I released the script on Codeberg: setupSSO.sh.

So, what are the benefits of setupSSO.sh over the official HowTo?

• reproducible
• reduces copy & paste errors by automation
• works from Primary Directory Node (PDN) by logging into other Directory Nodes via SSH
• asks you some simple configuration settings instead of ing you to change the code snippets from the HowTo
• asks you for the passwords beforehand, e.g. if you are using a single password for all servers or different passwords.
• checks for hostnames in SSL certs
• checks your Identity Provider (IdP) SSL certs for validity and create a new IdP SSL cert if validity is shorter than 1 year, otherwise it will reuse the existing cert.
• maybe some more benefits.

So, if you are using UCS as well, you might want to have a look into setupSSO.sh when you are using SSO as well or plan to do so.

Disclaimer 2: As said I did this script in my spare time, so there is no support for it by Univention. Feedback via Codeberg is appreciated, of course.

The new server is a used/refurbished Supermicro server with 2x 14-core Xeon E5-2683 and 256 GB RAM and 4x 3.5″ hot-swappable drives. It also came with a Hardware-RAID SAS/SATA 8-port controller with BBU. I also ordered two slim drive kits (MCP-220-81504-0N & MCP-220-81506-0N) to be able to use 2x 3.5″ slots for rotational HDDs as a cheap storage. Right now I added 2x 128 GB Supermicro SATA DOMs, 4x WD Red 4 TB SSDs and a Sonnet Fusion 4×4 Silent and 4x 1 TB Seagate Firecuda 520 NVMe disks.

And here the issue starts:

The NVMe should be capable of 4-5 GB/s, but they are connected to a PCIe 3.0 x16 port via the Sonnet Fusion 4×4, which itself features a PCIe bridge, so bifurbacation is not necessary.

When doing some tests with bonnie++ I get around 1 GB/s transfer rates out of a RAID10 setup with all 4 NVMes. In fact, regardless of the RAID level there are only transfer rates of about 1 – 1.2 GB/s with bonnie++. (All software RAIDs with mdadm.)

But also when constructing a RAID each NVMe gives around 300-600 MB/s in sync speed – except for one exception: RAID1.

Regardless of how many NVMe disks in a RAID1 setup the sync speed is up to 2.5 GB/s for each of the NVMe disks. So the lower transfer rates with bonnie++ or other RAID levels shouldn’t be limited by bus speed nor by CPU speed. Alas, atop shows upto 100% CPU usage for all tests. I even tested

In my understanding RAID10 should perform similar to RAID1 in terms of syncing and better and while bonnie++ tests (up to 2x write and 4x read speed compared to a single disk).

For the bonnie++ tests I even made some tests that are available here. You can find the test parameters listed in the hostname column: Baldur is the hostname, then followed by the layout (near-2, far-2, offset-2), chunk size and concurrency of bonnie++. In the end there was no big impact of the chunk size of the RAID.

So, now I’m wondering what the reason for the “slow” performance of those 4x NVMe disks is? Bus speed of the PCIe 3.0 x16 shouldn’t be the cause, because I assume that the software RAID will need to transfer the blocks in RAID1 as well as in RAID10 over the bus. Same goes for the CPU: the amount of CPU work should be roughly the same for RAID1 and for RAID10. RAID10 should even have an advantage because the blocks only need to be synced to 2 disks in a stripe set.

Bonnie++ tests are a different topic for sure. But when testing reading with dd from the md-devices I “only” get around 1-1.5 GB/s as well. Even when using LVM RAID instead of LVM on top of md RAID.

All NVMe disks are already set to 4k and IO scheduler is set to mq-deadline.

Is there anything I could do to improve the performance of the NVMe disks? On the other head, pure transfer rates are not that important to a server that runs a dozen of VMs. Here the improved IOPS performance over rotation disks is a clear performance gain. But I’m still curious if I could get maybe 2 GB/s out of a RAID10 setup with the NVMe disks. Then again having two independent RAID1 setups for MariaDB and for PostgreSQL databases might be a better choice over a single RAID10 setup?

Back in that time there was some small website running on Kullervo to display some information about the Debian autobuilder. After some time we (as m68k porters) moved that webpage away from Kullervo to my root server. Step by step this site evolved to Buildd.Net and extended to other archs and “suites” beside unstable like backports or non-volatile. The project got more and more complex and beyond my ability to do a complete necessary rewrite.

So, in 2016 I asked for adoption of the project and in 2018 I shut it down, because (apparently) there was nobody taking over. From November 2005 until January 2018 I do have entries in my PostgreSQL database for Buildd.Net.

I think the data in the database might be interesting for those that want to examine that data. You can use the data to see how build times have increased over time, which e.g. led to the expulsion of m68k as release arch, because the arch couldn’t keep up anymore. I could imagine that you could do other interesting analysis with that data. For example how new versions of the toolchain increased the build times, maybe even if a specific version of e.g. binutils or gcc had a positive effect on certain archs, but a negative effect on other archs.

If there is interest in this data I could open the database to the public or even upload the dump of the database so that you can download and install it on your own.