You are here

100% CPU load due to Leap Second

This morning Gregor Samsa woke up... oh, pardon! This morning I woke up and found myself puzzled, because my home server was eating up all of my 4 cores CPU cycles. Especially mysqld was high on CPU load. 100% CPU load for the mysql-server instance and 100% CPU load for akonadiservers own mysqld instance. Restarting KDE and mysql-server didn't help on my Debian unstable machine. Next step was upgrading the system. Sometimes this helps indeed, but not today.

Looking at bugs.debian.org for mysql-server didn't reveal any help as well. So my next logical step was to ask on #debian-devel in IRC. And my question was very quick answered: 

11:28 < ij> since tonight I've got two mysqld processes running at 100% CPU, one spawned by akonadi and
            the other is the mysqld from mysql-server (unstable that is). is this an already known issue?
            haven't found anything on b.d.o for mysql-server, though
11:29 < mrvn> ij: topic
11:29 < mrvn> you need to set the time
11:30 < ij> waaaaah!
11:30 < mrvn> ij: indeed.

The topic was at that time: 

 100% CPU? Reset leap second http://xrl.us/bnde4w

So, it was caused by the leap second. Although you might suspect mysql doing some nasty things (which, IMHO, is always a good guess ;)), the issue is this time within the Linux kernel itself, as a commit on git.kernel.org clearifies.

To fix this issue you need to set the time manually using the following command or just reboot: 

date -s "`date`"

So far I found these applications being hit by this kernel bug: 

  • mysql-server
  • akonadi (as it uses own mysql instances)
  • Firefox
  • Openfire Jabber server (because it's using Java, which seems to trigger the problem as mysql does)
  • Virtualbox' VBoxSVC process
  • puppetmaster from package puppet, reported by Michael
  • mythfrontend, reported by pos on #debian-devel
  • Jetty, Hudson, Puppet agent and master, reported by Christian
  • milter-greylist, reported by E. Recio
  • dovecot, reported by Diogo Resende
  • Google Chrome, reported by Erik B. Andersen
  • if you find more apps, please comment and I'll include them here...

So, hope this helps and many thanks to mrvn and infinity on #debian-devel for the help!

Kategorie: 
 

Comments

You can add puppet ( especially puppetmaster ) to this list, as I have seen before going to bed ( same fix ).

Jetty, Hudson, Puppet agent and master

milter-greylist (not 100%, but two threads at 20% each)
Afetr I stopped ntpd and ntpdate and started it again, everything immediately dropped.

My Slackware 13.0 and 13.37 systems were 100% unaffected by the issue.

dovecot just stopped for me a couple hours ago. You have to update the date and then start it.

I had Chrome using 100% CPU on 3 cores along with mysqld using 100% on one core (on a quad core processor). Set the time with both mysqld and chrome going, and it fixed then without so much as a restart of either program. I do not know if this was occurring with Chromium. This was on Ubuntu.

Thanks for the fix, it saved my uptime (47 days and counting!)

Dude this blog saved my server thanks :)

Pages

Add new comment

Theme by Danetsoft and Danang Probo Sayekti inspired by Maksimer