Before the weekend I announced a downtime/maintenance windows to upgrade PostgreSQL from v15 to v17 – because of the Debian upgrade from Bookworm to Trixie. After some tests with a cloned VM I decided use the quick path of pg_ugradecluster 15 main -v 17 -m upgrade –clone. As this would be my first time to upgrade PostgreSQL that way, I made several backups. In the end everything went smooth and the database is now on v17.

However, there was also a new Proxmox kernel and packages, so I also upgrade one Proxmox node and rebootet it. And then the issues began:

But before that I also encountered an issue with Redis for Mastodon. It complained about this:

Unable to obtain the AOF file appendonly.aof.4398.base.rdb

Solution to this was to change redis configuration to autoappend no.

And then CephFS was unavailable again, complaining about laggy MDS or no MDS at all, which – of course – was totally wrong. I search for solutions and read many forum posts in the Proxmox forum, but nothing helped. I also read the official Ceph documentation. After a whole day offline for all of the services to my thousands of users, I somehow managed to get systemctl reset-failed mnt-pve-cephfs && systemctl start mnt-pve-cephfs again. Shortly before that I followed the advice in the Ceph docs for RADOS Health and there especially section about Troubleshooting Monitors.

In the end, I can’t say which step exactly did the trick that CephFS was working again. But as it seems, I will have one or two more chances to find out, because only one server out of three is currently updated.

Another issue during the downtime also was that one server crashed/rebooted and didn’t came back. It hang in the midst of an upgrade at the point of upgrade-grub. Usually it wouldn’t be a big deal: just go the IPMI website and reboot the server.

Nah! That’s too simple!

For some unknow reason the IPMI interfaces lost their DHCP leases: the DHCP server at the colocation was not serving IPs. So I opened a ticket, got some acknowledgement from the support but also a statement “maybe tomorrow or on Monday…”. Hmpf!

On Sunday evening I managed to bring back CephFS. As said: no idea what specific step did the trick. But the story continues: On Monday before lunch time the IPMI DHCP was working again and I could access the web interfaces again, logged in…. and was forcefully locked out again:

Your session has timed out. You will need to open a new session

I hit the problem described here. But cold resetting the BMC didn’t work. So still no working web interface to deal with the issue. But on my phone I got “IPMIView” as app and that still worked and showed the KVM console. But what I saw there didn’t make me happy as well:

The reason for this is apparently the crash while running update-grub. Anyway, using the Grub bootloader and selecting an older kernel works fine. The server boots, Proxmox is showing the node as up and…. the working CephFS is stalled again! Fsck!

Rebooting the node or stopping Ceph on that node results immediatedly in a working CephFS again.

Currently I’m moving everything off of Ceph to the local disks of the two nodes. If everything is on local disks I can work on debugging CephFS without interrupting the service for the users (hopefully). But this also means that there will be no redundancy for Mastodon and mail.

When I have more detailled information about possible reasons and such, I may post to the Proxmox forum.

Overview

Shortly after the update from Debian 12 “Bookworm” to Debian 13 “Trixie” I also updated the Debian-based Proxmox installations. And then the issues began and I had sleepless nights, many downtimes and frustrated users, because the usually rock-stable Ceph storage became unstable. The OSDs went off the net, the Ceph Filesystem got degraded and everything became slow. The Ceph Filesystems (CephFS) also holds the mail storage as well as the shared storage (code & data) for my Nerdculture.de Mastodon instance.

Just to outline what I’m about to discuss, here’s the cabling plan for my 3-node hyperconverged Proxmox server setup:

Basically you see 3 types of connections:
1) Internet connection to the colocation switch
2) Internal Proxmox connections between the 3 nodes
3) Internal Ceph connections between the 3 nodes

The internal, directly wired connections are necessary because the colocation provider have had no additional Copper 10 Gbit/s ports (10GbaseT) available. So I had to wire up all those by directly attached patch cables.

Ceph has a backend and frontend network. You can run Ceph with just one network, but well, then Proxmox and Ceph would need to share the same network and access to Ceph would slow down when Virtual Machines (VMs) were migrated between the nodes.

What happened the last weeks?

The problem started, as said, after updating the Proxmox nodes. On Sept. 24th the first outage happened. You can read my summary here. Somehow the network Ceph connection between didn’t work anymore. The setup that was running for years now didn’t work anymore. The Ceph backend network couldn’t see the disks (OSD) any longer, so added manual routes between the nodes instead of relying on FRR with OSPF (a dynamic routing protocol). This solved the problem back then.

The next issue happened a week later on Oct. 2nd: since the last issue a week before I discovered that CephFS was awkfully slow. Loading the mails took like 10 seconds instead of being instantly there. So I tried to find the reason. My best assumption was: the WD Red SA500 2 TB that are holding the WAL/DB for the Ceph cluster are reaching the wear level end. These SSDs are not made for that kind of workload.

Another reason might be that the Ceph frontend network, which uses the Proxmox network, because the VMs need to access the Ceph frontend, is a OpenVSwitch bridge and traffic from Baldur is using the link via Pepper to Gate, for example, instead of using the direct connection, which adds some latency and reducing bandwidth.

And with that being said, this was the reason why there was an outage yesterday as well:

For the Ceph backend network, I use an internal Linux bridge in Proxmox to hold the IP for the Ceph backend on each node. Then there are two network cards, as described in the drawing. On the link between the nodes I configured Point to Point connections and added a route for the direct neighbor with a lower metric and a route for the other node with a higher metric. The other link vice versa. This works pretty well for the Ceph backend.

Yesterday I wanted to deploy those changes as well to the Proxmox network and get rid off of that Layer 2 network via OpenVSwitch. Settings this up in the operating system was no big deal, but unfortunately Proxmox complained later about the nodes having more than one IP. And there the issue started again.

But there was another problem, because even when reverting that network change, the Ceph cluster had issues again and couldn’t find its peers. I restarted services, rebooted nodes, etc… whatever to make it work again. But still OSDs were failing, coming online again, and failing again. The service mnt-pve-cephfs.mount was not able to mount the CephFS and thus CephFS was not available for the VMs and therefore the services like mail and Mastodon failed to load as well as nearly all services that need SSL certificates which – you guess it! – lies on CephFS as well. No CephFS available, no SSL certs and no service.

But why was it not possible to mount the CephFS on the Proxmox host nodes? I had a look onto the syslog and other logs while restarting services, but the output was that much and fast, that I couldn’t find the root cause for it.

At one time I was lucky and spotted this line:
2025-10-12T01:05:28.209050+02:00 baldur ceph-mgr[40880]: ERROR:root:Module 'xmltodict' is not installed.
And the solution was as simple as searching the web for that error message and stumble across this post in the Proxmox forum:

I was able to correct this with python3-xmltodict, that resolved one issue
So, after installing that package the Ceph cluster was happy again and Proxmox could finally mount CephFS with restarting mnt-pve-cephfs.mount.

Then it was just a matter of restarting VMs and services and finally Mastodon on https://nerdculture.de/ was available again as well as mail started to come in.

Lessons learned

For one I’m going to buy new SSDs for WAL/DB in Ceph, most likely Micron 5400 MAX. This should bringt he latency down with Ceph and increase the overall speed, because data is only written for the client, when all 3 nodes have written their data to disk. The slowest node or disk is the resulting speed of Ceph. WD Red SSDs might be good enough for NAS systems, but for constant disk writes like in the case of WAL/DB in Ceph, they seem to hit their limit rather soon.

Another thing I could improve is the network. It is a complex setup and prone to errors. I need to talk to the colocation if I can get 6x 10 Gbps ports on their switch or if I can bring in my own switch and what that would cost?

Speaking of: what switch would you recommend?

Interesting in this is, that for some time xmpp.social seemed to be the domain of choice for many users, maybe because of “xmpp” and “social” in the domain name – or because it is easier to name it than “hookipa” with “double-oh” and “kay”… who knows…

So, the user counts on xmpp.social were rising in a steeper curve than for hookipa.net, so much that I even considered to move over to xmpp.social for the main domain of that website.

But then something happened: the curated list of XMPP providers, which is now available under https://providers.xmpp.net. Since then some client software apps included the list, e.g. uwpx, and the user count was rising on hookipa.net – while it slowed down on xmpp.social.

Here you can see that the red curve of xmpp.social was for a long time above the blue curve of hookipa.net. Approx. in August 2021 something changed and hookipa.net was steadily increasing its user count. After a year, roughly in August 2022 hookipa.net surpassed the user count of xmpp.social.

Reason for this might be that hookipa.net is listed under Class A provider on providers.xmpp.net. Class A just means that some certain criteria is met like open registration and such. It doesn’t say anything whether or not it is a well-operated service. Well, at least not directly.

You can also look at the graphs on https://the-federation.info about the increase on hookipa.net and the stagnation on xmpp.social. There you can see the difference between a service that is listed on such a providers list and one that isn’t listed. Both domains are equally operated on my server and when you visit the Hookipa website you can register for both domains. But currently a downside (for me) of providers.xmpp.net is, that you need to provide the data for your classification on a website. Hookipa has a website, xmpp.social not, because it redirects to hookipa.net. Therefor xmpp.social is not included on providers.xmpp.net and thus is not gaining that many new users than hookipa.net.

I find that quite interesting, how registration counts are shifted from one domain to the other over time and what leads to that shift.

If you want to know which XMPP apps make use of the providers list, you can have a look at https://providers.xmpp.net/apps/.

For years I’m self-hosting lots of stuff and also for family, friends and others. The user management was either standalone for each service like Nextcloud, Mastodon, Friendica, XMPP, or based on the mail auth backend on PostgreSQL, for example by authenticating against Dovecot IMAP server. This became complex over time and I was looking for a centralized auth backend. Basically this means: LDAP backend.

For that I took a look into 389ds, FusionDirectory and Univention Corporate Server (UCS). With 389ds I had installation or setup issues. FusionDirectory was way better and easier to use, but also very complex. In the end I went with UCS because of the UI experienced, the ease of use and having a working self-care portal for the users. And UCS comes with some kind of App Center with some nice apps in it that are preconfigured for the LDAP directory. So, really easy to use and to get started.

So, last year (even before I got employed by Univention) I migrated my users from PostgreSQL backend to UCS and LDAP backend. The migration was smooth and nice and worked like a charme. Only issue: things like my public XMPP server, Friendica or Mastodon are still using open registration and therefor internal auth backend. It will be much more difficult to migrate these with an existing user base of 3000-4000 users in total. So, having LDAP as user/auth backend is nice, but you should consider this at an early stage and not when you already have a plenty of services and users on your services. 😉

However, it’s even nicer when you can use Single Sign On (SSO) with your apps. I got a taste of SSO when I was doing Cisco UC stuff and liked it that I don’t need to enter my credentials over and over again. SSO is also possible with UCS, so I wanted to give it a try on my own server.

The official HowTo on setting up SAML SSO basically covers the process of setting it up, but my impression was, that this process can be made better, less error-prone and more reproducible by automatting the setup.

So I wrote in my spare time a small shell script to follow the instructions from the official HowTo and after many tests and enhancements, I released the script on Codeberg: setupSSO.sh.

So, what are the benefits of setupSSO.sh over the official HowTo?

• reproducible
• reduces copy & paste errors by automation
• works from Primary Directory Node (PDN) by logging into other Directory Nodes via SSH
• asks you some simple configuration settings instead of ing you to change the code snippets from the HowTo
• asks you for the passwords beforehand, e.g. if you are using a single password for all servers or different passwords.
• checks for hostnames in SSL certs
• checks your Identity Provider (IdP) SSL certs for validity and create a new IdP SSL cert if validity is shorter than 1 year, otherwise it will reuse the existing cert.
• maybe some more benefits.

So, if you are using UCS as well, you might want to have a look into setupSSO.sh when you are using SSO as well or plan to do so.

Disclaimer 2: As said I did this script in my spare time, so there is no support for it by Univention. Feedback via Codeberg is appreciated, of course.

The new server is a used/refurbished Supermicro server with 2x 14-core Xeon E5-2683 and 256 GB RAM and 4x 3.5″ hot-swappable drives. It also came with a Hardware-RAID SAS/SATA 8-port controller with BBU. I also ordered two slim drive kits (MCP-220-81504-0N & MCP-220-81506-0N) to be able to use 2x 3.5″ slots for rotational HDDs as a cheap storage. Right now I added 2x 128 GB Supermicro SATA DOMs, 4x WD Red 4 TB SSDs and a Sonnet Fusion 4×4 Silent and 4x 1 TB Seagate Firecuda 520 NVMe disks.

And here the issue starts:

The NVMe should be capable of 4-5 GB/s, but they are connected to a PCIe 3.0 x16 port via the Sonnet Fusion 4×4, which itself features a PCIe bridge, so bifurbacation is not necessary.

When doing some tests with bonnie++ I get around 1 GB/s transfer rates out of a RAID10 setup with all 4 NVMes. In fact, regardless of the RAID level there are only transfer rates of about 1 – 1.2 GB/s with bonnie++. (All software RAIDs with mdadm.)

But also when constructing a RAID each NVMe gives around 300-600 MB/s in sync speed – except for one exception: RAID1.

Regardless of how many NVMe disks in a RAID1 setup the sync speed is up to 2.5 GB/s for each of the NVMe disks. So the lower transfer rates with bonnie++ or other RAID levels shouldn’t be limited by bus speed nor by CPU speed. Alas, atop shows upto 100% CPU usage for all tests. I even tested

In my understanding RAID10 should perform similar to RAID1 in terms of syncing and better and while bonnie++ tests (up to 2x write and 4x read speed compared to a single disk).

For the bonnie++ tests I even made some tests that are available here. You can find the test parameters listed in the hostname column: Baldur is the hostname, then followed by the layout (near-2, far-2, offset-2), chunk size and concurrency of bonnie++. In the end there was no big impact of the chunk size of the RAID.

So, now I’m wondering what the reason for the “slow” performance of those 4x NVMe disks is? Bus speed of the PCIe 3.0 x16 shouldn’t be the cause, because I assume that the software RAID will need to transfer the blocks in RAID1 as well as in RAID10 over the bus. Same goes for the CPU: the amount of CPU work should be roughly the same for RAID1 and for RAID10. RAID10 should even have an advantage because the blocks only need to be synced to 2 disks in a stripe set.

Bonnie++ tests are a different topic for sure. But when testing reading with dd from the md-devices I “only” get around 1-1.5 GB/s as well. Even when using LVM RAID instead of LVM on top of md RAID.

All NVMe disks are already set to 4k and IO scheduler is set to mq-deadline.

Is there anything I could do to improve the performance of the NVMe disks? On the other head, pure transfer rates are not that important to a server that runs a dozen of VMs. Here the improved IOPS performance over rotation disks is a clear performance gain. But I’m still curious if I could get maybe 2 GB/s out of a RAID10 setup with the NVMe disks. Then again having two independent RAID1 setups for MariaDB and for PostgreSQL databases might be a better choice over a single RAID10 setup?

Back in that time there was some small website running on Kullervo to display some information about the Debian autobuilder. After some time we (as m68k porters) moved that webpage away from Kullervo to my root server. Step by step this site evolved to Buildd.Net and extended to other archs and “suites” beside unstable like backports or non-volatile. The project got more and more complex and beyond my ability to do a complete necessary rewrite.

So, in 2016 I asked for adoption of the project and in 2018 I shut it down, because (apparently) there was nobody taking over. From November 2005 until January 2018 I do have entries in my PostgreSQL database for Buildd.Net.

I think the data in the database might be interesting for those that want to examine that data. You can use the data to see how build times have increased over time, which e.g. led to the expulsion of m68k as release arch, because the arch couldn’t keep up anymore. I could imagine that you could do other interesting analysis with that data. For example how new versions of the toolchain increased the build times, maybe even if a specific version of e.g. binutils or gcc had a positive effect on certain archs, but a negative effect on other archs.

If there is interest in this data I could open the database to the public or even upload the dump of the database so that you can download and install it on your own.

To address this I would like to see XMPP support in mail clients (MUAs). So when you reply to a mail or write a new one, the client will do a lookup in your addressbook if the address has an XMPP field associated with it and (if not) do a DNS lookup for _xmpp-server._tcp.example.com (or the matching domain part of recipients address). If there is an XMPP address listed in mail header, that JID will be used. When the lookup is successful and an xmpp: protocol handler is configured in the system, the MUA offers an option to begin a chat with the recipient and/or displays the presence status of the recipients (depending on available web-presence or presence subscription).

Basically a good candidate could be Thunderbird, because it already has XMPP support built in, albeit not a good implementation and lacking many modern features like OMEMO. But for basic functions (like presence status and such) it should be sufficient for a start.

Other candidates could be Evolution, Kmail (as KDE MUA and Kaidan as a native KDE XMPP client) or even Apple Mail.app, because Apples addressbook supports XMPP fields for each contact.

Basically the same could be done for SIP contacts: if a SIP SRV record for that domain does exist, the MUA could offer an option to call the recipient.

I would be willing to give some money via Bountysource or similar platforms. Is anyone aware of such a project or willing to write some addons? Maybe within the GSoC?

PS: there is RFC7259 about Jabber/XMPP JID in mail headers and there is also a page in the XMPP.org wiki.

So, please update your git settings to https://github.com/ingoj to https://codeberg.org/Windfluechter (or the specific repo).

Honestly, I think the solution needs to be provided by LetsEncrypt…

I was having some strange issues on my ejabberd XMPP server the other day: some users complained that they couldn’t connect anymore to the MUC rooms on my server and in the logfiles I discovered some weird warnings about LetsEncrypt certificates being expired – although they were just new and valid until end of December.

It looks like this:

[warning] <0.368.0>@ejabberd_pkix:log_warnings/1:393 Invalid certificate in /etc/letsencrypt.sh/certs/buildd.net/fullchain.pem: at line 37: certificate is no longer valid as its expiration date has passed

and…

[warning] <0.18328.2>@ejabberd_s2s_out:process_closed/2:157 Failed to establish outbound s2s connection nerdica.net -> forum.friendi.ca: Stream closed by peer: Your server's certificate is invalid, expired, or not trusted by forum.friendi.ca (not-authorized); bouncing for 237 seconds

When checking out with some online tools like SSLlabs or XMPP.net the result was strange, because SSLlabs reported everything was ok while XMPP.net was showing the chain with X3 and D3 certs as having a short term validity of a few days:

After some days of fiddling around with the issue, trying to find a solution, it appears that there is a problem in Ejabberd when there are some old SSL certifcates being found by Ejabberd that are using the old CA chain. Ejabberd has a really nice feature where you can just configure a SSL cert directory (or a path containing wildcars. Ejabberd then reads all of the SSL certs and compare them to the list of configured domains to see which it will need and which not.

What helped (for me at least) was to delete all expired SSL certs from my directory, downloading the current CA file pems from LetsEncrypt (see their blog post from September 2020), run update-ca-certificates and ejabberdctl restart (instead of just ejabberdctl reload-config). UPDATE: be sure to use dpkg-reconfigure ca-certificates to uncheck the DST Root X3 cert (and others if necessary) before renewing the certs or running update-ca-certificates. Otherwise the update will bring in the expired cert again.

Currently I see at least two other XMPP domains in my server logs having certicate issues and in some MUCs there are reports of other domains as well.

Disclaimer: Again: this helped me in my case. I don’t know if this is a bug in Ejabberd or if this procedure will help you in your case nor if this is the proper solution. But maybe my story will help you solving your issue if you experience SSL certs issues in the last few days, especially now that the R3 cert has already expired and the X3 cert following in a few hours.

Early this year there was already a strong movement away from WhatsApp towards other solutions. Mainly to Signal, but also some other services like the Fediverse gained some new users. And also XMPP got their fair share of new users.

So, what to do about the WhatsApp ToS change then? Shall we go all to Signal? Surely not. Signal is another vendor lock-in silo. It’s centralistic and recent development plans want to implement some crypto payment system. Even Bruce Schneier thinks that this is a bad idea.

Other alternatives often named include Matrix/Element or XMPP. Today, Don di Dislessia in the (german) Fediverse asked about power consumption of the Fediverse incl. Matrix and XMPP and how much renewable energy is being used. Of course there is no easy answer to this question, but I tried my best – at least for my own server.

Here are my findings and conclusions…

Power

Currently my server in the colocation is using about 93W in average with 6c Xeon E5-2630L, 128 GB RAM, 4x 2 TB WD Red + 1 Samsung 960pro NVMe. The server is 7 years old. When I started with that server the power consumption was about 75W, but back then there were far less users on the server. So, 20W more over the past year…

Users

I’m running my Friendica node on Nerdica.net since 2013. Over the years it became one of the largest Friendica servers in the Fediverse, for some time it was the largest one. It has currently like 700 total users and 180 monthly active users. My Mastodon instance on Nerdculture.de has about 1000 total users and about 300 monthly active users.

Since last year I also run a Matrix-Synapse server. Although I invited my family I’m in fact the only active user on that server and have joined some channels.

My XMPP server is even older than my Friendica node. For long time I had like maybe 20 users. Now I setup a new website and added some domains like hookipa.net and xmpp.social the user count increased and currently I have like 130 users on those two domains and maybe like 50 monthly active users. Also note that all my Friendica and Mastodon users can use XMPP with their accounts, but won’t be counted the same way as on “native” users on ejabberd, because the auth backend is different.

So, let’s assume I do have like 2000 total users and 500 monthly active users.

CPU, Database Sizes and Disk I/O

Let’s have a look about how many resources are being used by those users.

Database Sizes:

• Friendica (MariaDB): 31 GB for 700 users
• Mastodon (PostgreSQL): 15 GB for 1000 users
• Matrix-Synapse (PostgreSQL): 5 GB for 1 user
• XMPP (PostgreSQL): 0.4 GB for 200 users

CPU times according to xentop:

• Webserver VM (Matrix, Friendica & Mastodon): 13410130 s / 130%
• XMPP VM: 944275 s / 5.4%

Friendica does use the largest database and causes most disk I/O on NVMe, but it’s difficult to differentiate between the load between the web apps on the webserver. So, let’s have a quick look on an simple metric:

Number of lines in webserver logfile:

• Friendica: 11575 lines
• Matrix: 8174 lines
• Mastodon: 3212 lines

These metrics correlate to some degree with the database I/O load, at least for Friendica. If you take into account the number of users, things look quite different.

Conclusion

Overall, and my personal impression, is that Matrix is really bad in regards of resource usage. Given that I’m the only active user it uses exceptionally many resources. When you also consider that Matrix is using a distributed database for its chat rooms, you can assume that the resource usage is multiplied across the network, making things even worse.

Friendica is using a large database and many disk accesses, but has a fairly large user base, so it seems ok, but of course should be improved.

Mastodon seems to be quite good, considering the database size, the number of log lines and the user count.

XMPP turns out to be the most efficient contestant in this comparison: it uses much less CPU cycles and database disk I/O.

Of course, Mastdon/Friendica are different services than XMPP or Matrix. So, coming back to the initial question about alternatives to WhatsApp, the answer for me is: you should prefer XMPP over Matrix alone for reasons of saving resources and thus reducing power consumption. Less power consumption also means a smaller ecological footprint and fewer CO₂ emissions for your communication with your family and friends.

XMPP is surely not the perfect replacement for WhatsApp, but I think it is the best thing to recommend. As said above, I don’t think that Signal is an viable option. It’s just another proprierary silo with all the problems that come with it. Matrix is a resource hog and not a messenger but a MS Teams replacement. Element as the main Matrix client is laggy and not multi-account/multi-server capable. Other Matrix clients do support multiple accounts but are not as feature-complete as Element. In the end the Matrix ecosystem will suffer from the same issues as XMPP did already a decade ago. But XMPP has learned to deal with it.

Also XMPP is proceeding fast in the last years and it has solved many problems many people are still complaining about. Sure, there still some open issues. The situation on IOS is still not as good as on Android with Conversations, but it is fairly close to it.

There are many efforts to improve XMPP. There is Quicksy IM, which is a service that will use your phone number as Jabber ID/JID and is thus comparable to Signal which uses phone numbers as well as unique identifier. But Quicksy is compatible with XMPP standards. Snikket is another new XMPP ecosystem aiming at smaller groups hosting their own server by simply installing a Docker container and setup some basic SRV records in the DNS. Or there is Mailcow, a Docker based mailserver setup that added XMPP server in their setup as well, so you can have the same mail and XMPP address. Snikket even got EU based funding for implementing XMPP Account Portability which also will improve the decentralization even further. Additionally XMPP helps vaccination in Canada and USA with vaxbot by Monal.

Be smart and use ecofriendly infrastructure.