Sie sind hier

Debian

Jitsi Meet and ejabberd

Since the more or less global lockdown caused by Covid-19 there was a lot talk about video conferencing solutions that can be used for e.g. those people that try to coordinate with coworkers while in home office. One of the solutions is Jitsi Meet, which is NOT packaged in Debian. But there are Debian packages provided by Jitsi itself.

Jitsi relies on an XMPP server. You can see the network overview in the docs. While Jitsi itself uses Prosody as XMPP and their docs only covers that one. But it's basically irrelevant which XMPP you want to use. Only thing is that you can't follow the official Jitsi documentation when you are not using Prosody but instead e.g. ejabberd. As always, it's sometimes difficult to find the correct/best non-official documentation or how-to, so I try to describe what helped me in configuring Jitsi Meet with ejabberd as XMPP server and my own coturn STUN/TURN server...

This is not a step-by-step description, but anyway... here we go with some links:

One of the first issues I stumpled across was that my Java was too old, but this can be quickly solved by update-alternatives:

There are 3 choices for the alternative java (providing /usr/bin/java).

Selection Path Priority Status
------------------------------------------------------------
* 0 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1111 auto mode
1 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1111 manual mode
2 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java 1081 manual mode
3 /usr/lib/jvm/jre-7-oracle-x64/bin/java 316 manual mode

It was set to jre-7, but I guess this was from years ago when I ran OpenFire as XMPP server.

If something is not working with Jitsi Meet, it helps to not watch the log files only, but also to open the Debug Console in your web browser. That way I catched some XMPP Failures and saw that it tries to connect to some components where the DNS records were missing:

meet IN A yourIP
chat.meet IN A yourIP
focus.meet IN A yourIP
pubsub.meet IN A yourIP

Of course you also need to add some config to your ejabberd.yml:

host_config:
domain.net:
auth_password_format: scram
meet.domain.net:
## Disable s2s to prevent spam
s2s_access: none
auth_method: anonymous
allow_multiple_connections: true
anonymous_protocol: both
modules:
mod_bosh: {}
mod_caps: {}
mod_carboncopy: {}
#mod_disco: {}
mod_stun_disco:
secret: "YOURSECRETTURNCREDENTIALS"
services:
-
host: yourIP # Your coturn's public address.
type: stun
transport: udp
-
host: yourIP # Your coturn's public address.
type: stun
transport: tcp
-
host: yourIP # Your coturn's public address.
type: turn
transport: udp
mod_muc:
access: all
access_create: local
access_persistent: local
access_admin: admin
host: "chat.@"
mod_muc_admin: {}
mod_ping: {}
mod_pubsub:
access_createnode: local
db_type: sql
host: "pubsub.@"
ignore_pep_from_offline: false
last_item_cache: true
max_items_node: 5000 # For Jappix this must be set to 1000000
plugins:
- "flat"
- "pep" # requires mod_caps
force_node_config:
"eu.siacs.conversations.axolotl.*":
access_model: open
## Avoid buggy clients to make their bookmarks public
"storage:bookmarks":
access_model: whitelist

There is more config that needs to be done, but one of the XMPP Failures I spotted from debug console in Firefox was that it tried to connect to conference.domain.net while I prefer to use chat.domain.net for its brevity. If you prefer conference instead of chat, then you shouldn't be affected by this. Just make just that your config is consistent with what Jitsi Meet wants to connect to. Maybe not all of the above lines are necessary, but this works for me.

Jitsi config is like this for me with short domains (/etc/jitsi/meet/meet.domain.net-config.js):

var config = {

hosts: {
domain: 'domain.net',
anonymousdomain: 'meet.domain.net',
authdomain: 'meet.domain.net',
focus: 'focus.meet.domain.net',
muc: 'chat.hookipa.net'
},

bosh: '//meet.domain.net:5280/http-bind',
//websocket: 'wss://meet.domain.net/ws',
clientNode: 'http://jitsi.org/jitsimeet',
focusUserJid: 'focus@domain.net',

useStunTurn: true,

p2p: {
// Enables peer to peer mode. When enabled the system will try to
// establish a direct connection when there are exactly 2 participants
// in the room. If that succeeds the conference will stop sending data
// through the JVB and use the peer to peer connection instead. When a
// 3rd participant joins the conference will be moved back to the JVB
// connection.
enabled: true,

// Use XEP-0215 to fetch STUN and TURN servers.
useStunTurn: true,

// The STUN servers that will be used in the peer to peer connections
stunServers: [
//{ urls: 'stun:meet-jit-si-turnrelay.jitsi.net:443' },
//{ urls: 'stun:stun.l.google.com:19302' },
//{ urls: 'stun:stun1.l.google.com:19302' },
//{ urls: 'stun:stun2.l.google.com:19302' },
{ urls: 'stun:turn.domain.net:5349' },
{ urls: 'stun:turn.domain.net:3478' }
],

....

In the above config snippet one of the issues solved was to add the port to the bosh setting. Of course you can also take care that your bosh requests get forwarded by your webserver as reverse proxy. Using the port in the config might be a quick way to test whether or not your config is working. It's easier to solve one issue after the other and make one config change at a time instead of needing to make changes in several places.

/etc/jitsi/jicofo/sip-communicator.properties:

org.jitsi.jicofo.auth.URL=XMPP:meet.domain.net
org.jitsi.jicofo.BRIDGE_MUC=jvbbrewery@chat.meet.domain.net

 /etc/jitsi/videobridge/sip-communicator.properties:

org.jitsi.videobridge.ENABLE_STATISTICS=true
org.jitsi.videobridge.STATISTICS_TRANSPORT=muc
org.jitsi.videobridge.STATISTICS_INTERVAL=5000

org.jitsi.videobridge.xmpp.user.shard.HOSTNAME=localhost
org.jitsi.videobridge.xmpp.user.shard.DOMAIN=domain.net
org.jitsi.videobridge.xmpp.user.shard.USERNAME=jvb
org.jitsi.videobridge.xmpp.user.shard.PASSWORD=SECRET
org.jitsi.videobridge.xmpp.user.shard.MUC_JIDS=JvbBrewery@chat.meet.domain.net
org.jitsi.videobridge.xmpp.user.shard.MUC_NICKNAME=videobridge1

org.jitsi.videobridge.DISABLE_TCP_HARVESTER=true
org.jitsi.videobridge.TCP_HARVESTER_PORT=4443
org.ice4j.ice.harvest.NAT_HARVESTER_LOCAL_ADDRESS=yourIP
org.ice4j.ice.harvest.NAT_HARVESTER_PUBLIC_ADDRESS=yourIP
org.ice4j.ice.harvest.DISABLE_AWS_HARVESTER=true
org.ice4j.ice.harvest.STUN_MAPPING_HARVESTER_ADDRESSES=turn.domain.net:3478
org.ice4j.ice.harvest.ALLOWED_INTERFACES=eth0

Sometimes there might be stupid errors like (in my case) wrong hostnames like "chat.meet..domain.net" (a double dot in the domain), but you can spot those easily in the debug console of your browser.

There tons of config options where you can easily make mistakes, but watching your logs and your debug console should really help you in sorting out these kind of errors. The other URLs above are helpful as well and more elaborate then my few lines here. Especially Mike Kuketz has some advanced configuration tips like disabling third party requests with "disableThirdPartyRequests: true" or limiting the number of video streams and such.

Hope this helps...

Kategorie: 

Jabber vs. XMPP

XMPP is widely - and mabye better - known as Jabber. This was more or less the same until Cisco bought Jabber Inc and the trademark. You can read more about the story on the XMPP.org website. But is there still a Jabber around? Yes, it is!

But Cisco Jabber is a whole infrastructure environment: you can't use Cisco Jabber client on its own without the other required Cisco infrastructure as Cisco CUCM and CIsco IM&P servers. So you can't just setup Prosody or ejabberd on your Debian server and connect Cisco Jabber to it. But what are the differences of Cisco Jabber to "standard" XMPP clients?

Cisco Jabber

The above screenshot from the official Cisco Jabber product webpage shows the new, single view layout of the Cisco Webex Teams client, but you can configure the client to have the old, classic split view layout of Contact List and Chat Window. But as you can already see from above screenshot audio & video calls is one of the core functions of Cisco Jabber whereas this feature has been added only lately to the well-known Conversations XMPP client on Android. Conversations is using Jingle extension to XMPP whereas Jabber uses SIP for voice/video calls. You can even use Cisco Jabber to control your deskphone via CTI, which is a quite common setup for Jabber. In fact you can configure Jabber to be just a CTI client to you phone or a fully featured UC client.

When you don't want to have Ciscos full set of on-premise servers, you can also use Cisco Jabber in conjunction with Cisco Webex as Cisco Webex Messenger. Or in conjunction with Webex Teams in Teams Messaging Mode. Last month Cisco announced general availability of XMPP federation for Webex Teams/Jabber in Teams Messaging Mode. With that you have basic functionality in Webex Teams. And when I say "basic" I really mean basic: you can have 1:1 chat only, no group chats (MUC) and no Presence status will be possible. Hopefully this is just the beginning and not the end of XMPP support in Webex Teams.

XMPP Clients

Well, I'm sure many of you know "normal" XMPP clients such as Gajim or Dino on Linux, Conversations on Android or Siskin/Monal/ChatSecure on Apple IOS. There are plenty of other clients of course and maybe you used an XMPP client in the past without knowing it. For example Jitsi Meet is based on XMPP and you can still download the Jitsi Desktop client and use it as a full-featured multi-protocol client, e.g. for XMPP and SIP. In fact Jitsi Desktop is maybe the client that comes closest to Cisco Jabber as a chat/voice/video client. In fact I already connected Jitsi Desktop to Cisco CUCM/IM&P infrastructure, but of course you won't be able to use all those Cisco proprietary extensions, but you can see the benefit of open, standardized protocols such as XMPP and SIP: you are free to use any standard compliant client that you want.

So, while Jitsi supported voice/video calls for a long time, even before they focussed on Jitsi Meet as a WebRTC based conference service, Conversations added this feature last month, as already stated. This had a huge effect to the whole XMPP federation, because you need an XMPP server that supports XEP-0215 to make these audio/video calls work. The well-known Compliance Tester listed the STUN/TURN features first as "Informational Tests", but quickly made this a mandatory test to pass tests and gain 100% on the Compliance Tester. But you cannot place SIP calls to other sides, because that's a different thing.

As many of you are familiar with standard XMPP clients, I'll focus now on some similarity and differences between Cisco Jabber and standard XMPP...

Similarities & Differences

First, you can federate with Cisco Jabber users. Cisco IM&P can use standard XMPP federation with all other XMPP standard compliant servers. This is really a big benefit and way better than other solutions that usually results in vendor lock-in. Depending on the setup, you can even join from your own XMPP client in MUCs (Multi User Chats), which Cisco calls "Persistent Chat Room". The other way is not that simple: basically it is possible to join with Cisco Jabber in a MUC on a random server, but it is not as easy as you might thing. Cisco Jabber simply lacks a way to enter a room JID (as you can find them on https://search.jabber.network/. Instead you need to be added as participant by a moderator or an admin in that 3rd party MUC.

Managed File Transfers is another issue. Cisco Jabber supports Peer-to-Peer file transfers and Managed File Transfers, where the uploaded file get transferred to an SFTP server as storage backend and where the IM&P server is handling the transfer via HTTPS. You can find a schematic drawing in the Configuration Guides. Although it appears similar to HTTP Upload as defined in XEP-0363, it is not very likely that it will work. I haven't tested it yet, because in my test scenario there is a gatekeeper in the path: Cisco Expressway doesn't support (yet) Managed File Transfer, but you can upvote the idea in the ideas management of Cisco or other ideas such as OMEMO support.

OMEMO support? Yes, there is no end-to-end encryption (E2EE) currently planned for Cisco Jabber, while it is common nowadays for most modern XMPP clients. I think it would be good for Cisco Jabber to also (optionally) support OMEMO or its successor. Messaging clients without E2EE are not state of the art anymore.

Whereas Conversations is the de-facto standard on Android, Apple IOS devices are still lacking a similar well-working client. See my blog post "XMPP - Fun with Clients" for a summary. In that regard Cisco Jabber might be the best XMPP client for IOS to some degree: you have working messaging, voice/video calls, Push Notifications and integration into Apples Call Kit.

There are most likely many, many more differences and issues between Cisco Jabber and standard compliant XMPP servers and clients. But basically Cisco Jabber is still based on XMPP and extends that by proprietary extensions.

Summary

While I have the impression that the free clients and servers are well doing and increased development in the past years (thanks to Conversations and the Compliance Tester), the situation of Cisco Jabber is a little different. As a customer you can sometimes get the impression that Cisco has lost interest in developing Cisco Jabber. It got better in the last years, but when Cisco Spark was introduced some years ago, the impression was that Cisco is heavily focussed on Spark (now: Webex Teams). It's not like Cisco is not listening to customers or the development has been stopped on Jabber, but my impression is that most customers don't give feedback or tell Cisco as the vendor what they want. You can either submit ideas via the Colaboration Customer Ideas Tool or provide feedback via your Cisco and partner channels.

I think it is important for the XMPP community to also have a large enterprise level vendor like Cisco. Otherwise the Internet will become more and more an Internet of closed silos like MS Teams, Slack, Facebook, etc. Of course there are other companies like ProcessOne (ejabberd) or Tigase, but I think you agree that Cisco is another level.

Kategorie: 

XMPP: ejabberd Project on the-federation.info

For those interested in alternative social networks there is that website that is called the-federation.info, which collects some statistics of "The Fediverse". The biggest part of the fediverse is Mastodon, but there are other projects (or parts) like Friendica or Matrix that do "federation". One of the oldest projects doing federation is XMPP. You could find some Prosody servers for some time now, because there is a Prosody module "mod_nodeinfo2" that can be used. But for ejabberd there is no such module (yet?) so far, which makes it a little bit difficult to get listed on the-federation.info.

Some days ago I wrote a small script to export the needed values to x-nodeinfo2 that is queried by the-federation.info. It's surely not the best script or solution for that job and is currently limited to ejabberd servers that use a PostgreSQL database as backend, although it should be fairly easy to adapt the script for use with MySQL. Well, at least it does its job. At least as there is no native ejabberd module for this task.

You can find the script on Github: https://github.com/ingoj/ejabberd-nodeinfo2

Enjoy it and register your ejabberd server on the-federation.info! :-)

Kategorie: 

XMPP - Fun with Clients

As I already wrote in my last blog post there's much development in XMPP, not only on the server side, but also on the client side. It's surely not exaggerated to say that Conversations on Android is the de-facto standard client-wise. So, if you have an Android phone, that's the client you want to try&use. As I don't have Android, I can't comment on it. The situation on Linux is good as well: there are such clients as Gajim, which is an old player in the "market" and is available on other platforms as well, but there is with Dino a new/modern client as well that you may want to try out.

The situation for macOS and iOS users are not that good as for Windows, Linux or Android users. But in the end all clients have their pro and cons... I'll try to summarize a few clients on Linux, macOS and iOS...

LinuxGajim

Fully featured multiprotol client with lots of available plugins. If you want to use OMEMO with Gajim you need to enable it in your plugin settings. There is even a plugin for letting the keyboard LED blink when there are new/unread messages. I found that a little bit annoying, so I disabled that. Gajim is an old-style client with the well-known layout of a contact list window and one for the actual chats. Gajim has some nice features like service discovery on your or remote servers.

Dino

Dino is available in Debian as dino-im and is a quite new client, which you will find out at first start: it's a single window app where the focus is on the chats. There is no contact list at first glance where you can see whether or not your contacts are online. You can find your contacts when you want to start a conversation with your contact. I don't find this bad or good. It's just different and puts the chat into focus, as said, maybe similar to WhatApp, Signal or other messengers nowadays where you just sent messages back and forth and don't care if the contact is online. The contact will receive and read your message when the contact is online and will eventually answer your message.

macOSMonal

Monal is an actively developed and maintained client for macOS and iOS. If you want to try out Monal, don't waste your time with the older client, but focus on Monal Catalyst (direct download link). Catalyst shares the same code as the iOS version of Monal and will become the default Monal download within the next few weeks. It's far easier for the developers to focus on one codebase than on two different ones. Monal has great potential, but also has some issues. For some reason it seems that some messages from time to time will be sent multiple times to a contact or a MUC. The developers are very helpful and supportive. So when you find a bug or issue, please report back. 

BeagleIM

BeagleIM is a free XMPP client by Tigase, which business is to sell their XMPP Communication suite and professional support for it. They provide clients for Android, macOS and iOS. Maybe for that reason their clients seems to be very mature, but of course will work best with their own server software. That doesn't mean that the clients won't work well with other 3rd party XMPP servers, just that their main focus will be their own server software, However, BeagleIM seems to work well with Prosody and ejabberd and when you have issues you can also reach out to Tigase on Mastodon, which I find a very big plus! They are really helpful there as well. The BeagleIM client is currently my main client on macOS and it works quite well. As you can see it's more or less chat-focused as well by default, but you can open a contact list window if you want to see all of available/all clients. Only issue I personally have at the moment is, that it seems to have problems with ejabberd: in the contact list and account preferences I see the accounts/contacts with ejabberd going offline/online every few minutes. There are some log entries in ejabberd that seem to be timeout related. I'm not sure whether this is an issue with ejabberd or with BeagleIM - or a rare combination of both.

iOSChatSecure

ChatSecure is one of my first XMPP clients on iOS I installed and used for a long time. It works mostly very well, supports OMEMO (like all the other clients I mention here) and it seems to be able to work well with bookmarks (i.e. use a list of MUCs to join). Only issues I have with ChatSecure currently are: 1) when ChatSecure comes to front after a deep sleep state of the client on iPhone it presents an empty screen and no accounts in settings. You need to quit ChatSecure and restart it to have it working again. Quite annoying. 2) when restarted it polls all messages again from MAM (message archives) over and over again. A small annoying where you can decide if it's better to have duplicated messages or may miss a message.

Monal

What was valid for Monal Catalyst on macOS is also (more or less) true for Monal on iOS. As said: they share the same code base. It is usuable, but has sometimes issues with joining MUCs: it appears as if you can't join a MUC, but suddenly you receive a message in the MUC and then it is listed under Chats and you can use it.

SiskinIM

Siskin is the iOS client from Tigase. It also seems to be very mature, but has some special caveats: in the settings you can configure Push Notifications and HTTP Uploads for each and every clients. Other clients make this automatically and I leave it to you to decide whether this is a nice feature that you can configure it or if it is a little bit annoying, because when you don't know that, you will be wondering why you can't upload files/fotos in a chat. Maybe uploading files will work with Tigase XMPP server, but it doesn't seem to work on my servers.

 

So, in the end, there are good/promising clients on iOS and macOS, but every client seem to have its own pitfalls. On iOS all three clients do support and use Apple Push Notifications, but you should choose carefully for which one you want to enable it. I can tell you it's a little bit annoying to test three clients and have Push Notifications turned on for all of them and have joined in several MUCs and get all notifications three times for every message... ;-)

MUCs are a special topic I need to investigate a little more in the future: when your server supports Bookmarks for MUCs I would assume that it should be working for all supporting clients and you only need to join MUC on one client and have that MUC at least in your list of bookmarks. If you want to join that MUC on every client might be another story. But I don't know if this is the intended behaviour of the XEP in question or if my assumption how it should work is just wrong.

In the end the situation for XMPP clients on macOS and iOS is much better than it was 1-2 years ago. Though, it is not as good as on Android, but you can help improving the situation by testing the available clients and give feedback to the developers by either joining the approrpriate MUCs or - even better - file issues on their Github pages!

Kategorie: 

XMPP - Prosody & Ejabberd

In my day job I'm responsible of maintaining the VoIP and XMPP infrastructure. That's about approx. 40.000 phones and several thousand users using Enterprise XMPP software. Namely it is Cisco CUCM and IM&P on the server side and Cisco Jabber on the client side. There is also Cisco Webex and Cisco Telepresence infrastructure to maintain.

On the other hand I'm running an XMPP server myself for a few users. It all started with ejabberd more than a decade ago or so. Then I moved to Openfire, because it was more modern and had a nice web GUI for administration. At some point there was Prosody as a new shiny star. This is now running for many users, mostly without any problems, but without much love and attention as well.

It all started as "Let's see what this Jabber stuff is..." on a subdomain like jabber.domain.com - it was later that I discovered the benefits of SRV records and the possibility of having the same address for mail, XMPP and SIP. So I began to provide XMPP acounts as well for some of my mail domains.

A year ago I enabled XMPP for my Friendica node on Nerdica.net, the second largest Friendica node according to the-federation.info. Although there are hundreds of monthly active users on Friendica, only a handful of users are using XMPP. XMPP has a hard stand since Google and Facebook went from open federation to closing in their user base.

My personal impression is that there is a lot of development in the last years in regards of XMPP - thanks to the Conversations client on Android - and its Compliance Tester. With that tool it is quite easy to have a common ground for the most needed features of todays user expectation in a mobile world. There is also some news in regards to XMPP clients on Apple iOS, but that's for another article.

This is about the server side, namely Prosody and Ejabberd. Of course there are already several excellent comparisons between these two server softwares. So, this is just my personal opinion and personal impressions about the two softwares I got in the past two weeks.

Prosody:
As I have the most experience with Prosody I'll start with it. Prosody has the advantage of being actively maintained and having lots of community modules to extend its functionality. This is a big win - but there is also the other side of truth: you'll need to install and configure many contrib modules to pass 100% in the Compliance Tester. Some modules might be not that well maintained. Another obstacle I faced with Prosody is the configuration style: usually you have the main config file where you can configure common settings, modules for all virtual hosts and components like PubSub, MUC, HTTP Upload and such. And then there are the config files for the virtual hosts, which feature the same kind of configuration. Important to all is (apparently): order does matter! This can get confusing: Components are similar to loading modules, using both for the same purpose can be, well, interesting. and configuration of modules and components can be challenging as well. When trying to get mod_http_upload working in the last days I experienced that a config on one virtual host was working, but the same config on a different host was not working. This was when I thought I might give Ejabberd a chance...

Ejabberd:
Contrary to Prosody there is a company behind Ejabberd. And this is often perceived as being good and bring some stability to Ejabberd. However, when I joined Ejabberd chat room, I learned in the first minutes by regarding the chat log that the main developer of that company left and the company itself seemed to have lost interest in Ejabberd. However the people in the chat room were relaxed: it's not the end of the world and there are other developers working on the code. So, no issue in the end, but that's not something you expect to read when you join a chat room for the first time. ;)
Contrary to Prosody Ejabberd seems to be well-prepared to pass the Compliance Tester without installing (too many) modules. Large sites such as conversations.im are running on Ejabberd. It is also said that Ejabberd doesn't need restarts of the server for certain config changes as Prosody does. The config file itself appears to be more straightforward and doesn't differentiate between modules and components which makes it a little more easy to understand.

Currently I haven't been able to deal much with Ejabberd, but one other difference is: there is a Debian repository on Prosody.im, but for Ejabberd there is no such repository. You'll have to use backports.debian.org for a newer version of Ejabberd on Debian Buster. It's up to you to decide what is better for you.

I'm still somewhat undecided whether or not to proceed with Ejabberd and migrate from Prosody. The developer of Prosody is very helpful and responsive and I like that. On the other hand, the folks in the Ejabberd chat rooms are very supportive as well. I like the flexibility and the various number of contrib modules for Prosody, but then again it's hard to find the correct/best one to load and to configure for a given task and to satisfy the Compliance Tester. Then again, both servers do feature a Web GUI for some basic tasks, but I like the one of Ejabberd more.

So, in the end, I'm also open for suggestions about either one. Some people will state of course that neither is the best way and I should consider Matrix, Briar or some other solutions, but that's maybe another article comparing XMPP and other options. This one is about XMPP server options: Prosody or Ejabberd. What do you prefer and why?

 

Kategorie: 

Too much disk IO on sda in RAID10 setup - part 2

Some days ago I blogged about my issue with one of the disks in my server having a high utilization and latency. There have been several ideas and guesses what the reason might be, but I think I found the root cause today:

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%     50113         60067424
# 2  Short offline       Completed without error       00%      3346   

After the RAID-Sync-Sunday last weekend I removed sda from my RAIDs today and started a "smartctl -t long /dev/sda". This test was quickly aborted because it already ran into an read error after just a few minutes. Currently I'm still running a "badblocks -w" test and this is the result so far:

# badblocks -s -c 65536 -w /dev/sda
Testing with pattern 0xaa: done
Reading and comparing: 42731372done, 4:57:27 elapsed. (0/0/0 errors)
42731373done, 4:57:30 elapsed. (1/0/0 errors)
42731374done, 4:57:33 elapsed. (2/0/0 errors)
42731375done, 4:57:36 elapsed. (3/0/0 errors)
 46.82% done, 6:44:41 elapsed. (4/0/0 errors)

Long story short: I already ordered a replacement disk!

But what's also interesting is this:

I removed the disk today at approx. 12:00 and you can see the immediate effect on the other disks/LVs (the high blue graph from sda shows the badblocks test), although the RAID10 is now in degraded mode. Interesting what effect (currently) 4 defect blocks can have to a RAID10 performance without smartctl taking notice of this. Smartctl only reported an issue after issueing the selftest. It's also strange that the latency and high utilization slowly increased over time, like 6 months or so.

Kategorie: 

Too much disk IO on sda in RAID10 setup

I have a RAID10 setup with 4x 2 TB WD Red disks in my server. Although the setup works fairly well and has enough throughput there is one strange issue with that setup: /dev/sda has more utilzation/load than the other 3 disks. See the blue line in the following graph which represents utilization by week for sda:

As you can see from the graphs and from the numbers below sda has a 2-3 times higher utilization than sdb, sdc or sdd, especially when looking at disk latency graph by Munin:

Although the graphs are a little confusing you can easily spot the big difference from the below values. And it's not only Munin showing this strange behaviour of sda, but also atop: 

Here you see that sda is 94% busy although the writes to the disks are a little bit lower than on the other disks. The screenshot of atop was before I moved MySQL/MariaDB to my NVMe disk 4 weeks ago. But you can also spot that sda is slowing down the RAID10.

So the big question is: why is utilization and latency of sda that high? it's the same disk model as the other disks. All disks are connected to a Supermicro X9SRi-F mainboard. The first two SATA ports are 6 Gbit/s, the other 4 ports are 3 Gbit/s ports:

sda sdb sdc sdd
=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD20EFRX-68AX9N0
Serial Number:    WD-WMC301414725
LU WWN Device Id: 5 0014ee 65887fe2c
Firmware Version: 80.00A80
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sat Jan  5 17:24:58 2019 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD20EFRX-68AX9N0
Serial Number:    WD-WMC301414372
LU WWN Device Id: 5 0014ee 65887f374
Firmware Version: 80.00A80
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sat Jan  5 17:27:01 2019 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD20EFRX-68AX9N0
Serial Number:    WD-WMC301411045
LU WWN Device Id: 5 0014ee 603329112
Firmware Version: 80.00A80
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sat Jan  5 17:30:15 2019 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD20EFRX-68AX9N0
Serial Number:    WD-WMC301391344
LU WWN Device Id: 5 0014ee 60332a8aa
Firmware Version: 80.00A80
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sat Jan  5 17:30:26 2019 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

There is even the same firmware version of the disks. Usually I would have expected a slower disk IO from the 3 Gbit/s disks (sdc & sdd), but not from sda. All disks are configured in BIOS to use AHCI.

I cannot explain why sda has higher latency and more utilization than the other disks. Any ideas are welcome and appreciated. You can also reach me in the Fediverse (Friendica: https://nerdica.net/profile/ij & Mastodon: ">https://nerdculture.de/) or via XMPP at ij@nerdica.net.

Kategorie: 

Adding NVMe to Server

My server runs on a RAID10 of 4x WD RAID 2 TB disks. Basically those disks are fast enough to cope with the disk load of the virtual machines (VMs). But since many users moved away from Facebook and Google, my Friendica installation on Nerdica.Net has a growing user count putting a large disk I/O load with many small reads & writes on the disks, resulting a slowing down the general disk I/O for all the VMs and the server itself. On mdraid-sync-Sunday this month the server needed two full days to sync its RAID10.

So the idea was to remove the high disk I/O load from the rotational disks the something different. For that reason I bought a Samsung Pro 970 512 GB NVMe disk and a matching PCIe 3.0 card to be put into my server in the colocation. On Thursday the Samsung has been installed by the rrbone staff in the colocation. I moved the PostgreSQL and MySQL databases from the RAID10 to the NVMe disk and restarted services again.

Here are some results from Munin monitoring: 

Disk Utilization

Here you can see how the disk utilization dropped after NVMe installation. The red coloured bar symbolizes the average utilization on RAID10 disks and the green bar symbolizes the same RAID10 after the databases were moved to the NVMe disk. There's roughly 20% less utilization now, whch is good.

Disk Latency

Here you can see the same coloured bars for the disk latency. As you can see the latency dropped by 1/3 now.

CPU I/O wait

The most significant graph is maybe the CPU graph where you can see a large portion of iowait of the CPUs. This is no longer true as there is apparently no significant iowait anymore thanks to the low latency and high IOPS nature of SSD/NVMe disks.

Overall I cannot confirm that adding the NVMe disk results in a significant faster page load of Friendica or Mastodon, maybe because other measurements like Redis/Memcached or pgbouncer already helped a lot before the NVMe disk, but it helps a lot with general disk I/O load and improving disk speeds inside of the VMs, like for my regular backups and such.

Ah, one thing to report is: in a quick test pgbench reported >2200 tps on NVMe now. That at least is a real speed improvement, maybe by order of 10 or so.

Kategorie: 

Xen & Databases

I'm running PostgreSQL and MySQL on my server that both serve different databases to Wordpress, Drupal, Piwigo, Friendica, Mastodon, whatever...

In the past the databases where colocated in my mailserver VM whereas the webserver was running on a different VM. Somewhen I moved the databases from domU to dom0, maybe because I thought that the databases would be faster running on direct disk I/O in the dom0 environment, but can't remember the exact rasons anymore.

However, in the meantime the size of the databases grew and the number of the VMs did, too. MySQL and PostgreSQL are both configured/optimized to run with 16 GB of memory in dom0, but in the last months I experienced high disk I/O especially for MySQL and slow I/O performance in all the domU VMs because of that.

Currently iotop shows something like this:

Total DISK READ :     131.92 K/s | Total DISK WRITE :    1546.42 K/s
Actual DISK READ:     131.92 K/s | Actual DISK WRITE:       2.40 M/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
 6424 be/4 mysql       0.00 B/s    0.00 B/s  0.00 % 60.90 % mysqld
18536 be/4 mysql      43.97 K/s   80.62 K/s  0.00 % 35.59 % mysqld
 6499 be/4 mysql       0.00 B/s   29.32 K/s  0.00 % 13.18 % mysqld
20117 be/4 mysql       0.00 B/s    3.66 K/s  0.00 % 12.30 % mysqld
 6482 be/4 mysql       0.00 B/s    0.00 B/s  0.00 % 10.04 % mysqld
 6495 be/4 mysql       0.00 B/s    3.66 K/s  0.00 % 10.02 % mysqld
20144 be/4 postgres    0.00 B/s   73.29 K/s  0.00 %  4.87 % postgres: hubzilla hubzi~
 2920 be/4 postgres    0.00 B/s 1209.28 K/s  0.00 %  3.52 % postgres: wal writer process
11759 be/4 mysql       0.00 B/s   25.65 K/s  0.00 %  0.83 % mysqld
18736 be/4 mysql       0.00 B/s   14.66 K/s  0.00 %  0.17 % mysqld
21768 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.02 % [kworker/1:0]
 2922 be/4 postgres    0.00 B/s   69.63 K/s  0.00 %  0.00 % postgres: stats collector process

MySQL data site is below configured max memory size for MySQL, so everything should more or less fit into memory. Yet, there is still a large amount of disk I/O by MySQL, much more than by PostgreSQL. Of course there is much I/O done by writes to the database.

However, I'm thinking of changing my setup again back to domU based database setup again, maybe one dedicated VM for both DBMS' or even two dedicated VMs for each of them? I'm not quite sure how Xen reacts to the current work load?

Back in the days when I did 3D computer graphic I did a lot of testing with different settings in regards of priorities and such. Basically one would think that giving the renderer more CPU time would speed of the rendering, but this turned out to be wrong: the higher the render tasks priority was, the slower the rendering got, because disk I/O (and other tasks that were necessary for the render task to work) got slowed down. When running the render task at lowest priority all the other necessary tasks could run on higher speed and return the CPU more quickly, which resulted in shorter render times.

So, maybe I experience something similar with the databases on dom0 here as well: dom0 is busy doing database work and this slows down all the other tasks (== domU VMs). When I would move databases back to domU this would enable dom0 again to better do its basic job of taking care of the domUs?

Of course, this is also a quite philosophical question, but what is the recommended setup? Is it better to separate the databases in two different VMs or just one? Or is running the databases on dom0 the best option?

I'm interested in your feedback, so please comment! :-)

UPDATE: you can also contact me @ij@nerdculture.de on Mastodon or on Friendica at https://nerdica.net/profile/ij

Kategorie: 

#Friendica vs #Hubzilla vs #Mastodon

I've been running a #Friendica node for several years now. Some months ago I also started to run a #Hubzilla hub as well. Some days ago I also installed #Mastodon on a virtual machine, because there was so much hype about Mastodon in the last days due to some changes Twitter made in regards of 3rd party clients.

All of those social networks do have their own focus:

Friendica: basically can connect to all other social networks, which is quite nice because there exists historically two different worlds: the Federation (Diaspora, Socialhome) and the Fediverse (GnuSocial, Mastodon, postActiv, Pleroma). Only Friendica and Hubzilla can federate with both: Federation and Fediverse.
Friendicas look&feel appears sometimes a little bit outdated and old, but it works very well and reliable.

Hubzilla: is the second player in the field of connecting both federations, but has a different focus. It is more of one-size-fits-all approach. If you need a microblogging site, a wiki, a cloud service, a website, etc. then Hubzilla is the way to go. The look&feel is a little bit more modern, but there are some quirks that appears a little odd to me. A unique feature for Hubzilla seems to be the concept of "nomadic accounts": you can move to a different hub and take all your data with you. Read more about that in the Hubzilla documentation.

Mastodon: this aims to be a replacement for Twitter as a microblogging service. It looks nice and shiny, has a bunch of nice clients for smartphones and has the largest userbase by far (which is not that important because of federation).
But the web GUI is rather limited and weird, as far as I can tell after just some days.

Technically spoken these are the main differences:
- Friendica: MySQL/MariaDB, PHP on the server, Clients: some Android clients, no iOS client
- Hubzilla: MySQL/MariaDB or PostgreSQL, PHP on the server, Clients: don't know, didn't care so far.
- Mastodon: PostgreSQL, Ruby on the server, Clients: many iOS and Android clients available

I'm not that big Ruby fan and if I remember correctly the Ruby stuff turned me away from Diaspora years ago and made me switch to Friendica, because back then it was a pain to maintain Diaspora. Mastodon addresses this by offering Docker container for the ease of installation and maintenance. But as I'm no Docker fan either, I followed the guide to install Mastodon without Docker, which works so far as well (for the last 3 days ).

So after all my Friendica node is still my favorit, because is just works and is reliable. Hubzilla has a different approach and offers a full set of webfeatures and nomadic accounts. The best I can say about Mastodon at this moment is: it runs on PostgreSQL and has nice clients on mobile devices.

Here are my instances:
- Friendica: https://nerdica.net/
- Hubzilla: https://silverhaze.eu/
- Mastodon: https://nerdculture.de/

PS: "A quick guide to The Free Network" by Sean Tilley on https://medium.com/we-distribute/a-quick-guide-to-the-free-network-c0693...

PPS: this is a cross post from my Friendica node.

Kategorie: 

Seiten

Theme by Danetsoft and Danang Probo Sayekti inspired by Maksimer