100% CPU load due to Leap Second

ij

This morning Gregor Samsa woke up… oh, pardon! This morning I woke up and found myself puzzled, because my home server was eating up all of my 4 cores CPU cycles. Especially mysqld was high on CPU load. 100% CPU load for the mysql-server instance and 100% CPU load for akonadiservers own mysqld instance. Restarting KDE and mysql-server didn’t help on my Debian unstable machine. Next step was upgrading the system. Sometimes this helps indeed, but not today.

Looking at bugs.debian.org for mysql-server didn’t reveal any help as well. So my next logical step was to ask on -devel in IRC. And my question was very quick answered: 

11:28 < ij> since tonight I’ve got two mysqld processes running at 100% CPU, one spawned by akonadi and
            the other is the mysqld from mysql-server (unstable that is). is this an already known issue?
            haven’t found anything on b.d.o for mysql-server, though
11:29 < mrvn> ij: topic
11:29 < mrvn> you need to set the time
11:30 < ij> waaaaah!
11:30 < mrvn> ij: indeed.

The topic was at that time: 

 100% CPU? Reset leap second http://xrl.us/bnde4w

So, it was caused by the leap second. Although you might suspect mysql doing some nasty things (which, IMHO, is always a good guess ;)), the issue is this time within the Linux kernel itself, as a commit on git.kernel.org clearifies.

To fix this issue you need to set the time manually using the following command or just reboot: 

date -s "`date`"

So far I found these applications being hit by this kernel bug: 

  • mysql-server
  • akonadi (as it uses own mysql instances)
  • Firefox
  • Openfire Jabber server (because it’s using Java, which seems to trigger the problem as mysql does)
  • Virtualbox’ VBoxSVC process
  • puppetmaster from package puppet, reported by Michael
  • mythfrontend, reported by pos on -devel
  • Jetty, Hudson, Puppet agent and master, reported by Christian
  • milter-greylist, reported by E. Recio
  • dovecot, reported by Diogo Resende
  • Google Chrome, reported by Erik B. Andersen
  • if you find more apps, please comment and I’ll include them here…

So, hope this helps and many thanks to mrvn and infinity on -devel for the help!

Uncategorized

7 thoughts on “100% CPU load due to Leap Second”

  1. Another one on the list
    You can add puppet ( especially puppetmaster ) to this list, as I have seen before going to bed ( same fix ).

  3. Another one to the list
    milter-greylist (not 100%, but two threads at 20% each)
    Afetr I stopped ntpd and ntpdate and started it again, everything immediately dropped.

  4. And yet
    My Slackware 13.0 and 13.37 systems were 100% unaffected by the issue.

  5. dovecot just stopped for me a
    dovecot just stopped for me a couple hours ago. You have to update the date and then start it.

  6. Chrome should be added to the list
    I had Chrome using 100% CPU on 3 cores along with mysqld using 100% on one core (on a quad core processor). Set the time with both mysqld and chrome going, and it fixed then without so much as a restart of either program. I do not know if this was occurring with Chromium. This was on Ubuntu.

    Thanks for the fix, it saved my uptime (47 days and counting!)

Comments are closed.