I learned a lovely thing yesterday – there’s such a thing as an XMLRPC attack on WordPress (which powers this site, as well as another I admin).
Essentially, this attack uses the XML-RPC interface present in WP to try to guess passwords, and there are variations of this attack that allow an attacker to guess multiple passwords in a single attempt, thanks to the way the XML-RPC interface is structured.
I only discovered this attack because, which was (and actually last I checked still was, but is now denying attackers) sending an average of 41 requests per second, which then hit the database, and it just became a massive backlog of data transactions needing to be processed.
My symptoms included constant database errors, other applications being slow/unusable, and incredibly long wait times for access. In my logs, though, I found the real key to my issues – hundreds of POST requests for the xmlrpc.php file. Googling this brought me to an explanation of the attack, and I was able to block access to the attackers (my apologies for any legitimate users in the Netherlands) in Apache’s configuration, leaving me more or less completely operational now.
So, good times yesterday, dealing with international attacks.