slideshow 1 slideshow 2 slideshow 3

Jitsi Meet and ejabberd

Since the more or less global lockdown caused by Covid-19 there was a lot talk about video conferencing solutions that can be used for e.g. those people that try to coordinate with coworkers while in home office. One of the solutions is Jitsi Meet, which is NOT packaged in Debian. But there are Debian packages provided by Jitsi itself.

Jitsi relies on an XMPP server. You can see the network overview in the docs. While Jitsi itself uses Prosody as XMPP and their docs only covers that one. But it's basically irrelevant which XMPP you want to use. Only thing is that you can't follow the official Jitsi documentation when you are not using Prosody but instead e.g. ejabberd. As always, it's sometimes difficult to find the correct/best non-official documentation or how-to, so I try to describe what helped me in configuring Jitsi Meet with ejabberd as XMPP server and my own coturn STUN/TURN server...

This is not a step-by-step description, but anyway... here we go with some links:

One of the first issues I stumpled across was that my Java was too old, but this can be quickly solved by update-alternatives:

There are 3 choices for the alternative java (providing /usr/bin/java).

Selection Path Priority Status
* 0 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1111 auto mode
1 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1111 manual mode
2 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java 1081 manual mode
3 /usr/lib/jvm/jre-7-oracle-x64/bin/java 316 manual mode

It was set to jre-7, but I guess this was from years ago when I ran OpenFire as XMPP server.

If something is not working with Jitsi Meet, it helps to not watch the log files only, but also to open the Debug Console in your web browser. That way I catched some XMPP Failures and saw that it tries to connect to some components where the DNS records were missing:

meet IN A yourIP IN A yourIP IN A yourIP IN A yourIP

Of course you also need to add some config to your ejabberd.yml:

auth_password_format: scram
## Disable s2s to prevent spam
s2s_access: none
auth_method: anonymous
allow_multiple_connections: true
anonymous_protocol: both
mod_bosh: {}
mod_caps: {}
mod_carboncopy: {}
#mod_disco: {}
host: yourIP # Your coturn's public address.
type: stun
transport: udp
host: yourIP # Your coturn's public address.
type: stun
transport: tcp
host: yourIP # Your coturn's public address.
type: turn
transport: udp
access: all
access_create: local
access_persistent: local
access_admin: admin
host: "chat.@"
mod_muc_admin: {}
mod_ping: {}
access_createnode: local
db_type: sql
host: "pubsub.@"
ignore_pep_from_offline: false
last_item_cache: true
max_items_node: 5000 # For Jappix this must be set to 1000000
- "flat"
- "pep" # requires mod_caps
access_model: open
## Avoid buggy clients to make their bookmarks public
access_model: whitelist

There is more config that needs to be done, but one of the XMPP Failures I spotted from debug console in Firefox was that it tried to connect to while I prefer to use for its brevity. If you prefer conference instead of chat, then you shouldn't be affected by this. Just make just that your config is consistent with what Jitsi Meet wants to connect to. Maybe not all of the above lines are necessary, but this works for me.

Jitsi config is like this for me with short domains (/etc/jitsi/meet/

var config = {

hosts: {
domain: '',
anonymousdomain: '',
authdomain: '',
focus: '',
muc: ''

bosh: '//',
//websocket: 'wss://',
clientNode: '',
focusUserJid: '',

useStunTurn: true,

p2p: {
// Enables peer to peer mode. When enabled the system will try to
// establish a direct connection when there are exactly 2 participants
// in the room. If that succeeds the conference will stop sending data
// through the JVB and use the peer to peer connection instead. When a
// 3rd participant joins the conference will be moved back to the JVB
// connection.
enabled: true,

// Use XEP-0215 to fetch STUN and TURN servers.
useStunTurn: true,

// The STUN servers that will be used in the peer to peer connections
stunServers: [
//{ urls: '' },
//{ urls: '' },
//{ urls: '' },
//{ urls: '' },
{ urls: '' },
{ urls: '' }


In the above config snippet one of the issues solved was to add the port to the bosh setting. Of course you can also take care that your bosh requests get forwarded by your webserver as reverse proxy. Using the port in the config might be a quick way to test whether or not your config is working. It's easier to solve one issue after the other and make one config change at a time instead of needing to make changes in several places.






Sometimes there might be stupid errors like (in my case) wrong hostnames like "" (a double dot in the domain), but you can spot those easily in the debug console of your browser.

There tons of config options where you can easily make mistakes, but watching your logs and your debug console should really help you in sorting out these kind of errors. The other URLs above are helpful as well and more elaborate then my few lines here. Especially Mike Kuketz has some advanced configuration tips like disabling third party requests with "disableThirdPartyRequests: true" or limiting the number of video streams and such.

Hope this helps...


Jabber vs. XMPP

XMPP is widely - and mabye better - known as Jabber. This was more or less the same until Cisco bought Jabber Inc and the trademark. You can read more about the story on the website. But is there still a Jabber around? Yes, it is!

But Cisco Jabber is a whole infrastructure environment: you can't use Cisco Jabber client on its own without the other required Cisco infrastructure as Cisco CUCM and CIsco IM&P servers. So you can't just setup Prosody or ejabberd on your Debian server and connect Cisco Jabber to it. But what are the differences of Cisco Jabber to "standard" XMPP clients?

Cisco Jabber

The above screenshot from the official Cisco Jabber product webpage shows the new, single view layout of the Cisco Webex Teams client, but you can configure the client to have the old, classic split view layout of Contact List and Chat Window. But as you can already see from above screenshot audio & video calls is one of the core functions of Cisco Jabber whereas this feature has been added only lately to the well-known Conversations XMPP client on Android. Conversations is using Jingle extension to XMPP whereas Jabber uses SIP for voice/video calls. You can even use Cisco Jabber to control your deskphone via CTI, which is a quite common setup for Jabber. In fact you can configure Jabber to be just a CTI client to you phone or a fully featured UC client.

When you don't want to have Ciscos full set of on-premise servers, you can also use Cisco Jabber in conjunction with Cisco Webex as Cisco Webex Messenger. Or in conjunction with Webex Teams in Teams Messaging Mode. Last month Cisco announced general availability of XMPP federation for Webex Teams/Jabber in Teams Messaging Mode. With that you have basic functionality in Webex Teams. And when I say "basic" I really mean basic: you can have 1:1 chat only, no group chats (MUC) and no Presence status will be possible. Hopefully this is just the beginning and not the end of XMPP support in Webex Teams.

XMPP Clients

Well, I'm sure many of you know "normal" XMPP clients such as Gajim or Dino on Linux, Conversations on Android or Siskin/Monal/ChatSecure on Apple IOS. There are plenty of other clients of course and maybe you used an XMPP client in the past without knowing it. For example Jitsi Meet is based on XMPP and you can still download the Jitsi Desktop client and use it as a full-featured multi-protocol client, e.g. for XMPP and SIP. In fact Jitsi Desktop is maybe the client that comes closest to Cisco Jabber as a chat/voice/video client. In fact I already connected Jitsi Desktop to Cisco CUCM/IM&P infrastructure, but of course you won't be able to use all those Cisco proprietary extensions, but you can see the benefit of open, standardized protocols such as XMPP and SIP: you are free to use any standard compliant client that you want.

So, while Jitsi supported voice/video calls for a long time, even before they focussed on Jitsi Meet as a WebRTC based conference service, Conversations added this feature last month, as already stated. This had a huge effect to the whole XMPP federation, because you need an XMPP server that supports XEP-0215 to make these audio/video calls work. The well-known Compliance Tester listed the STUN/TURN features first as "Informational Tests", but quickly made this a mandatory test to pass tests and gain 100% on the Compliance Tester. But you cannot place SIP calls to other sides, because that's a different thing.

As many of you are familiar with standard XMPP clients, I'll focus now on some similarity and differences between Cisco Jabber and standard XMPP...

Similarities & Differences

First, you can federate with Cisco Jabber users. Cisco IM&P can use standard XMPP federation with all other XMPP standard compliant servers. This is really a big benefit and way better than other solutions that usually results in vendor lock-in. Depending on the setup, you can even join from your own XMPP client in MUCs (Multi User Chats), which Cisco calls "Persistent Chat Room". The other way is not that simple: basically it is possible to join with Cisco Jabber in a MUC on a random server, but it is not as easy as you might thing. Cisco Jabber simply lacks a way to enter a room JID (as you can find them on Instead you need to be added as participant by a moderator or an admin in that 3rd party MUC.

Managed File Transfers is another issue. Cisco Jabber supports Peer-to-Peer file transfers and Managed File Transfers, where the uploaded file get transferred to an SFTP server as storage backend and where the IM&P server is handling the transfer via HTTPS. You can find a schematic drawing in the Configuration Guides. Although it appears similar to HTTP Upload as defined in XEP-0363, it is not very likely that it will work. I haven't tested it yet, because in my test scenario there is a gatekeeper in the path: Cisco Expressway doesn't support (yet) Managed File Transfer, but you can upvote the idea in the ideas management of Cisco or other ideas such as OMEMO support.

OMEMO support? Yes, there is no end-to-end encryption (E2EE) currently planned for Cisco Jabber, while it is common nowadays for most modern XMPP clients. I think it would be good for Cisco Jabber to also (optionally) support OMEMO or its successor. Messaging clients without E2EE are not state of the art anymore.

Whereas Conversations is the de-facto standard on Android, Apple IOS devices are still lacking a similar well-working client. See my blog post "XMPP - Fun with Clients" for a summary. In that regard Cisco Jabber might be the best XMPP client for IOS to some degree: you have working messaging, voice/video calls, Push Notifications and integration into Apples Call Kit.

There are most likely many, many more differences and issues between Cisco Jabber and standard compliant XMPP servers and clients. But basically Cisco Jabber is still based on XMPP and extends that by proprietary extensions.


While I have the impression that the free clients and servers are well doing and increased development in the past years (thanks to Conversations and the Compliance Tester), the situation of Cisco Jabber is a little different. As a customer you can sometimes get the impression that Cisco has lost interest in developing Cisco Jabber. It got better in the last years, but when Cisco Spark was introduced some years ago, the impression was that Cisco is heavily focussed on Spark (now: Webex Teams). It's not like Cisco is not listening to customers or the development has been stopped on Jabber, but my impression is that most customers don't give feedback or tell Cisco as the vendor what they want. You can either submit ideas via the Colaboration Customer Ideas Tool or provide feedback via your Cisco and partner channels.

I think it is important for the XMPP community to also have a large enterprise level vendor like Cisco. Otherwise the Internet will become more and more an Internet of closed silos like MS Teams, Slack, Facebook, etc. Of course there are other companies like ProcessOne (ejabberd) or Tigase, but I think you agree that Cisco is another level.


XMPP: ejabberd Project on

For those interested in alternative social networks there is that website that is called, which collects some statistics of "The Fediverse". The biggest part of the fediverse is Mastodon, but there are other projects (or parts) like Friendica or Matrix that do "federation". One of the oldest projects doing federation is XMPP. You could find some Prosody servers for some time now, because there is a Prosody module "mod_nodeinfo2" that can be used. But for ejabberd there is no such module (yet?) so far, which makes it a little bit difficult to get listed on

Some days ago I wrote a small script to export the needed values to x-nodeinfo2 that is queried by It's surely not the best script or solution for that job and is currently limited to ejabberd servers that use a PostgreSQL database as backend, although it should be fairly easy to adapt the script for use with MySQL. Well, at least it does its job. At least as there is no native ejabberd module for this task.

You can find the script on Github:

Enjoy it and register your ejabberd server on! :-)


XMPP - Getting Started...

When you a new to XMPP and just installed your client and connected to some XMPP server, you will notice: there is no contact for you to chat with!

This is contrary to Facebook, WhatsApp or Signal where you upload your personal address book with all of your known contacts to a central server. And if you ask me, this is a good thing that XMPP is not doing that stuff. There are some websites where users voluntarily publish their Jabber IDs. And there is Quicksy, an XMPP service/client that uses phone numbers as Jabber IDs where you also can publish your own Jabber ID.

So, when there is no central directory, how can you find contacts? Well, the answer is simple: with XMPP you can not only have 1:1 chats with other people, but also join chat rooms, so called "Multi User Chats", or in short: MUCs. And for MUCs there is a search engine and a directory for that on:

For a start how about joining the MUC for the client you are using or some MUCs based on your interests:

You got the point, I guess.

And did you know that some mail providers such as also provide XMPP access? So, when you have contacts with such addresses, you can contact them not only by mail but also by direct messaging them via your XMPP clients and add them as your XMPP contacts as well. 

That way you can easily grow your network of XMPP contacts without the need of a central directory server or uploading your personal addressbook to some servers.

Oh! And when you say that you have already tried XMPP a few years ago: give it another try! There are many huge improvements being made in the last years!


XMPP - Fun with Clients

As I already wrote in my last blog post there's much development in XMPP, not only on the server side, but also on the client side. It's surely not exaggerated to say that Conversations on Android is the de-facto standard client-wise. So, if you have an Android phone, that's the client you want to try&use. As I don't have Android, I can't comment on it. The situation on Linux is good as well: there are such clients as Gajim, which is an old player in the "market" and is available on other platforms as well, but there is with Dino a new/modern client as well that you may want to try out.

The situation for macOS and iOS users are not that good as for Windows, Linux or Android users. But in the end all clients have their pro and cons... I'll try to summarize a few clients on Linux, macOS and iOS...


Fully featured multiprotol client with lots of available plugins. If you want to use OMEMO with Gajim you need to enable it in your plugin settings. There is even a plugin for letting the keyboard LED blink when there are new/unread messages. I found that a little bit annoying, so I disabled that. Gajim is an old-style client with the well-known layout of a contact list window and one for the actual chats. Gajim has some nice features like service discovery on your or remote servers.


Dino is available in Debian as dino-im and is a quite new client, which you will find out at first start: it's a single window app where the focus is on the chats. There is no contact list at first glance where you can see whether or not your contacts are online. You can find your contacts when you want to start a conversation with your contact. I don't find this bad or good. It's just different and puts the chat into focus, as said, maybe similar to WhatApp, Signal or other messengers nowadays where you just sent messages back and forth and don't care if the contact is online. The contact will receive and read your message when the contact is online and will eventually answer your message.


Monal is an actively developed and maintained client for macOS and iOS. If you want to try out Monal, don't waste your time with the older client, but focus on Monal Catalyst (direct download link). Catalyst shares the same code as the iOS version of Monal and will become the default Monal download within the next few weeks. It's far easier for the developers to focus on one codebase than on two different ones. Monal has great potential, but also has some issues. For some reason it seems that some messages from time to time will be sent multiple times to a contact or a MUC. The developers are very helpful and supportive. So when you find a bug or issue, please report back. 


BeagleIM is a free XMPP client by Tigase, which business is to sell their XMPP Communication suite and professional support for it. They provide clients for Android, macOS and iOS. Maybe for that reason their clients seems to be very mature, but of course will work best with their own server software. That doesn't mean that the clients won't work well with other 3rd party XMPP servers, just that their main focus will be their own server software, However, BeagleIM seems to work well with Prosody and ejabberd and when you have issues you can also reach out to Tigase on Mastodon, which I find a very big plus! They are really helpful there as well. The BeagleIM client is currently my main client on macOS and it works quite well. As you can see it's more or less chat-focused as well by default, but you can open a contact list window if you want to see all of available/all clients. Only issue I personally have at the moment is, that it seems to have problems with ejabberd: in the contact list and account preferences I see the accounts/contacts with ejabberd going offline/online every few minutes. There are some log entries in ejabberd that seem to be timeout related. I'm not sure whether this is an issue with ejabberd or with BeagleIM - or a rare combination of both.


ChatSecure is one of my first XMPP clients on iOS I installed and used for a long time. It works mostly very well, supports OMEMO (like all the other clients I mention here) and it seems to be able to work well with bookmarks (i.e. use a list of MUCs to join). Only issues I have with ChatSecure currently are: 1) when ChatSecure comes to front after a deep sleep state of the client on iPhone it presents an empty screen and no accounts in settings. You need to quit ChatSecure and restart it to have it working again. Quite annoying. 2) when restarted it polls all messages again from MAM (message archives) over and over again. A small annoying where you can decide if it's better to have duplicated messages or may miss a message.


What was valid for Monal Catalyst on macOS is also (more or less) true for Monal on iOS. As said: they share the same code base. It is usuable, but has sometimes issues with joining MUCs: it appears as if you can't join a MUC, but suddenly you receive a message in the MUC and then it is listed under Chats and you can use it.


Siskin is the iOS client from Tigase. It also seems to be very mature, but has some special caveats: in the settings you can configure Push Notifications and HTTP Uploads for each and every clients. Other clients make this automatically and I leave it to you to decide whether this is a nice feature that you can configure it or if it is a little bit annoying, because when you don't know that, you will be wondering why you can't upload files/fotos in a chat. Maybe uploading files will work with Tigase XMPP server, but it doesn't seem to work on my servers.


So, in the end, there are good/promising clients on iOS and macOS, but every client seem to have its own pitfalls. On iOS all three clients do support and use Apple Push Notifications, but you should choose carefully for which one you want to enable it. I can tell you it's a little bit annoying to test three clients and have Push Notifications turned on for all of them and have joined in several MUCs and get all notifications three times for every message... ;-)

MUCs are a special topic I need to investigate a little more in the future: when your server supports Bookmarks for MUCs I would assume that it should be working for all supporting clients and you only need to join MUC on one client and have that MUC at least in your list of bookmarks. If you want to join that MUC on every client might be another story. But I don't know if this is the intended behaviour of the XEP in question or if my assumption how it should work is just wrong.

In the end the situation for XMPP clients on macOS and iOS is much better than it was 1-2 years ago. Though, it is not as good as on Android, but you can help improving the situation by testing the available clients and give feedback to the developers by either joining the approrpriate MUCs or - even better - file issues on their Github pages!


XMPP - Prosody & Ejabberd

In my day job I'm responsible of maintaining the VoIP and XMPP infrastructure. That's about approx. 40.000 phones and several thousand users using Enterprise XMPP software. Namely it is Cisco CUCM and IM&P on the server side and Cisco Jabber on the client side. There is also Cisco Webex and Cisco Telepresence infrastructure to maintain.

On the other hand I'm running an XMPP server myself for a few users. It all started with ejabberd more than a decade ago or so. Then I moved to Openfire, because it was more modern and had a nice web GUI for administration. At some point there was Prosody as a new shiny star. This is now running for many users, mostly without any problems, but without much love and attention as well.

It all started as "Let's see what this Jabber stuff is..." on a subdomain like - it was later that I discovered the benefits of SRV records and the possibility of having the same address for mail, XMPP and SIP. So I began to provide XMPP acounts as well for some of my mail domains.

A year ago I enabled XMPP for my Friendica node on, the second largest Friendica node according to Although there are hundreds of monthly active users on Friendica, only a handful of users are using XMPP. XMPP has a hard stand since Google and Facebook went from open federation to closing in their user base.

My personal impression is that there is a lot of development in the last years in regards of XMPP - thanks to the Conversations client on Android - and its Compliance Tester. With that tool it is quite easy to have a common ground for the most needed features of todays user expectation in a mobile world. There is also some news in regards to XMPP clients on Apple iOS, but that's for another article.

This is about the server side, namely Prosody and Ejabberd. Of course there are already several excellent comparisons between these two server softwares. So, this is just my personal opinion and personal impressions about the two softwares I got in the past two weeks.

As I have the most experience with Prosody I'll start with it. Prosody has the advantage of being actively maintained and having lots of community modules to extend its functionality. This is a big win - but there is also the other side of truth: you'll need to install and configure many contrib modules to pass 100% in the Compliance Tester. Some modules might be not that well maintained. Another obstacle I faced with Prosody is the configuration style: usually you have the main config file where you can configure common settings, modules for all virtual hosts and components like PubSub, MUC, HTTP Upload and such. And then there are the config files for the virtual hosts, which feature the same kind of configuration. Important to all is (apparently): order does matter! This can get confusing: Components are similar to loading modules, using both for the same purpose can be, well, interesting. and configuration of modules and components can be challenging as well. When trying to get mod_http_upload working in the last days I experienced that a config on one virtual host was working, but the same config on a different host was not working. This was when I thought I might give Ejabberd a chance...

Contrary to Prosody there is a company behind Ejabberd. And this is often perceived as being good and bring some stability to Ejabberd. However, when I joined Ejabberd chat room, I learned in the first minutes by regarding the chat log that the main developer of that company left and the company itself seemed to have lost interest in Ejabberd. However the people in the chat room were relaxed: it's not the end of the world and there are other developers working on the code. So, no issue in the end, but that's not something you expect to read when you join a chat room for the first time. ;)
Contrary to Prosody Ejabberd seems to be well-prepared to pass the Compliance Tester without installing (too many) modules. Large sites such as are running on Ejabberd. It is also said that Ejabberd doesn't need restarts of the server for certain config changes as Prosody does. The config file itself appears to be more straightforward and doesn't differentiate between modules and components which makes it a little more easy to understand.

Currently I haven't been able to deal much with Ejabberd, but one other difference is: there is a Debian repository on, but for Ejabberd there is no such repository. You'll have to use for a newer version of Ejabberd on Debian Buster. It's up to you to decide what is better for you.

I'm still somewhat undecided whether or not to proceed with Ejabberd and migrate from Prosody. The developer of Prosody is very helpful and responsive and I like that. On the other hand, the folks in the Ejabberd chat rooms are very supportive as well. I like the flexibility and the various number of contrib modules for Prosody, but then again it's hard to find the correct/best one to load and to configure for a given task and to satisfy the Compliance Tester. Then again, both servers do feature a Web GUI for some basic tasks, but I like the one of Ejabberd more.

So, in the end, I'm also open for suggestions about either one. Some people will state of course that neither is the best way and I should consider Matrix, Briar or some other solutions, but that's maybe another article comparing XMPP and other options. This one is about XMPP server options: Prosody or Ejabberd. What do you prefer and why?



EU-Wahl, Klima, CDU und Youtube

Nächste Woche am 26. Mai ist EU-Wahl (und in Rostock auch noch Kommunal- und Oberbürgermeisterwahl).

Am 24. Mai sind wieder weltweit Demos von "Fridays for Future", bei dem die Schülerinnen und Schüler, die größtenteils nicht an der EU-Wahl teilnehmen dürfen, obwohl der Ausgang der Wahl ihr zukünftiges Leben am längsten beeinflussen dürfte, gegen die Untätigkeit der Politik in Bezug auf den Klimawandel protestieren.

Nun mischt sich auch noch ein (mir bisher unbekannter) Youtuber ein:

In dem Video wird eigentlich recht anschaulich dargelegt, warum man nicht mehr die CDU, die SPD und die AfD wählen sollte. Insbesondere bei CDU & SPD ist zu bemängeln, daß sie nachweislich nichts gegen den Klimawandel tun - seit vielen Jahrzehnten. Auch die Beteiligungen an Kriegen spricht gegen die Wahl von CDU & SPD.

Wen sollte man also dann bei der EU-Wahl wählen?

Da es in Deutschland bei der EU-Wahl keine 5%-Hürde existiert, gibt es viele Alternativen. Natürlich die Grünen und die Linken, aber auch solche Parteien wie ÖkoLinx. Oder aber auch Piratenpartei oder "Die Partei". Alle haben so ihre Vor- und Nachteile, aber allemal besser als AfD oder andere extremistischen Parteien zu wählen.

Wer SPD und CDU wählt, wählt ein "Weiter so!" bei der katastrophalen Klimapolitik der vergangenen Jahrzehnte, die uns erst in diese Situation gebracht hat, daß nun Schülerinnen und Schüler weltweit demonstrieren müssen. Für ein solches "Weiter so!" hat die Menschheit aber keine Zeit mehr. Wir brauchen mehr erneuerbaren Energien und einen zügigen Ausstieg aus den fossilen Brennstoffen, aber genau das verhindern CDU & SPD seit langem.

Schaut euch das Video an und wählt verantwortungsvoll: für einen Wechsel in der Politik und gegen rechtspopulistische Parteien!


#FridaysForFuture in #Rostock

Heute fanden in vielen, vielen Städten weltweit Demos gegen den Klimawandel unter dem Motto #FridaysForFuture statt. Auch in Rostock. Und zwar quasi arbeitnehmerfreundlich um 12 Uhr am Kröpeliner Tor, so daß ich heute einfach ein bißchen die Mittagspause ausdehnte musste,um die vielen Schülerinnen und Schülern zu unterstützen, die ihr Grundrecht auf Demonstrationsfreiheit in Anspruch nahmen.

Kritiker prangern an, daß die Schüler dies während der Schulzeit machen.

Das kann man natürlich kritisieren, aber dann hat man halt nichts verstanden. Denn zum einen haben die Schülerinnen und Schüler Recht, wenn sie nun einfordern, daß gehandelt werden soll, zum anderen sind Proteste, insbesondere ein Streik wie bei diesem Schulstreik, eigentlich nur dann wirksam, wenn sie quasi weh tun. Deshalb sind die Forderungen, daß die Klimastreiks in der Freizeit stattfinden sollen, absurd. Da würde kein Hahn nach krähen.

Greta Thunberg hätte niemals die weltweite Aufmerksamkeit bekommen, wenn sie sich einfach in ihrer Freizeit mit einem Schild vor das schwedische Parlament gestellt hätte. Häufig kann man nur eine Veränderung bewirken, wenn man bereit ist, dafür etwas in Kauf zu nehmen. Im Grunde kann man genau das nämlich auch bei der Politik sehen: echte Veränderung, die eben nicht bequem zu erreichen ist, findet nicht statt, weil viele Politiker nicht bereit sind, dafür etwaige Konsequenzen (keine Wiederwahl, keine weitere Karriere in der Politik) zu tragen. Also wird weichgespülte Politik gemacht, die aber oftmals hinter den eigenen politischen Zielen, aber auch hinter den Erwartungen der Bevölkerung zurückbleiben. Die Politiker, die unbequem sind und sich etwas trauen, werden häufig in der eigenen Partei isoliert. Beispiel hierfür ist Marco Bülow aus Dortmund in der SPD.

Aber zurück zur Demo in Rostock: es war toll zu sehen, daß sich da Jugendliche organisieren, eine Demo auf die Beine stellen und für ihre Überzeugung auf die Straße gehen - oder sollte man eher sagen: ihren Frust über die Untätigkeit der Politik?. Klar, wenn man bereits einige Demos wie "Freiheit statt Angst" seitens der Organisation mitbekommen hat. dann weiß man, was eine solche Organisation braucht. Zum Beispiel eine leistungsfähigere Lautsprecheranlage und ein Podest, damit man die Rednerinnen und Redner hören und sehen kann. Aber solche Kleinigkeiten sind geschenkt! Da haben sich Jugendliche organisiert und haben etwas gemeinsam auf die Beine gestellt. Das ist uneingeschränkt großartig!

Es bleibt zu hoffen, daß sich diese Demonstrationen nun verstetigen und die Politik vor sich hertreiben, damit endlich und schnell gehandelt wird.


Too much disk IO on sda in RAID10 setup - part 2

Some days ago I blogged about my issue with one of the disks in my server having a high utilization and latency. There have been several ideas and guesses what the reason might be, but I think I found the root cause today:

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%     50113         60067424
# 2  Short offline       Completed without error       00%      3346   

After the RAID-Sync-Sunday last weekend I removed sda from my RAIDs today and started a "smartctl -t long /dev/sda". This test was quickly aborted because it already ran into an read error after just a few minutes. Currently I'm still running a "badblocks -w" test and this is the result so far:

# badblocks -s -c 65536 -w /dev/sda
Testing with pattern 0xaa: done
Reading and comparing: 42731372done, 4:57:27 elapsed. (0/0/0 errors)
42731373done, 4:57:30 elapsed. (1/0/0 errors)
42731374done, 4:57:33 elapsed. (2/0/0 errors)
42731375done, 4:57:36 elapsed. (3/0/0 errors)
 46.82% done, 6:44:41 elapsed. (4/0/0 errors)

Long story short: I already ordered a replacement disk!

But what's also interesting is this:

I removed the disk today at approx. 12:00 and you can see the immediate effect on the other disks/LVs (the high blue graph from sda shows the badblocks test), although the RAID10 is now in degraded mode. Interesting what effect (currently) 4 defect blocks can have to a RAID10 performance without smartctl taking notice of this. Smartctl only reported an issue after issueing the selftest. It's also strange that the latency and high utilization slowly increased over time, like 6 months or so.


Too much disk IO on sda in RAID10 setup

I have a RAID10 setup with 4x 2 TB WD Red disks in my server. Although the setup works fairly well and has enough throughput there is one strange issue with that setup: /dev/sda has more utilzation/load than the other 3 disks. See the blue line in the following graph which represents utilization by week for sda:

As you can see from the graphs and from the numbers below sda has a 2-3 times higher utilization than sdb, sdc or sdd, especially when looking at disk latency graph by Munin:

Although the graphs are a little confusing you can easily spot the big difference from the below values. And it's not only Munin showing this strange behaviour of sda, but also atop: 

Here you see that sda is 94% busy although the writes to the disks are a little bit lower than on the other disks. The screenshot of atop was before I moved MySQL/MariaDB to my NVMe disk 4 weeks ago. But you can also spot that sda is slowing down the RAID10.

So the big question is: why is utilization and latency of sda that high? it's the same disk model as the other disks. All disks are connected to a Supermicro X9SRi-F mainboard. The first two SATA ports are 6 Gbit/s, the other 4 ports are 3 Gbit/s ports:

sda sdb sdc sdd
Model Family:     Western Digital Red
Device Model:     WDC WD20EFRX-68AX9N0
Serial Number:    WD-WMC301414725
LU WWN Device Id: 5 0014ee 65887fe2c
Firmware Version: 80.00A80
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sat Jan  5 17:24:58 2019 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Model Family:     Western Digital Red
Device Model:     WDC WD20EFRX-68AX9N0
Serial Number:    WD-WMC301414372
LU WWN Device Id: 5 0014ee 65887f374
Firmware Version: 80.00A80
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sat Jan  5 17:27:01 2019 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Model Family:     Western Digital Red
Device Model:     WDC WD20EFRX-68AX9N0
Serial Number:    WD-WMC301411045
LU WWN Device Id: 5 0014ee 603329112
Firmware Version: 80.00A80
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sat Jan  5 17:30:15 2019 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Model Family:     Western Digital Red
Device Model:     WDC WD20EFRX-68AX9N0
Serial Number:    WD-WMC301391344
LU WWN Device Id: 5 0014ee 60332a8aa
Firmware Version: 80.00A80
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sat Jan  5 17:30:26 2019 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

There is even the same firmware version of the disks. Usually I would have expected a slower disk IO from the 3 Gbit/s disks (sdc & sdd), but not from sda. All disks are configured in BIOS to use AHCI.

I cannot explain why sda has higher latency and more utilization than the other disks. Any ideas are welcome and appreciated. You can also reach me in the Fediverse (Friendica: & Mastodon: "> or via XMPP at



Theme by Danetsoft and Danang Probo Sayekti inspired by Maksimer