CNYO Without The .quota – A Quick Fix And Diagnostic For A Pesky WordPress Hack

From The "Hopelessly Hacked Help" Department

Below is the process I went through to clean up a WordPress site injection into www.cnyo.org – an old astronomy club I directed and a site (now unused) that I still maintain/keep active because there's lots of (a) good information and (b) CNY-based astronomy history to be read. There are many, many files in a WordPress install that can suffer an attack of some kind, many, many ways for a hacker/bot to make such an attack happen, but only a few things everyone seems to recommend doing to try to make the problem go away. In my case, the problem with the website went away after simply renaming two folders, uploading new versions of those folders, and then using the success of that replacement step to find the exact files in the site that were causing me issues. Consider this post a "First Step" if you find yourself similarly inconvenienced.

Some Light Reading (Background)

The CNYO website had, up until quite recently, a bug in the main menu that I initially chalked up to a very old (and free) theme (Vortex) running up against recent updates to WordPress. The romantic perspective on the maintaining of the site mentioned above aside, the site is really only still up because it (a) costs me nothing and (b) is now hosted in a subfolder of this website, and the site only gets a few visits each day (even with the upcoming solar eclipse in Western New York, cnyo.org is crickets). I hadn't much bothered exploring the menu issue further. The specific issue is shown below for the front page of the website, with the "About" menu much larger vertically than all the other menu items and the sub-menu items inside of "About" equally expanded.

Interestingly, this oddity started appearing in Firefox and then, after a year or more, began appearing in Chrome and Safari. Clicking on the About link produced the image at the link below, which was a full-blown insertion of unwanted content and a de-columning of the website (and, even worse, it shows two fewer people subscribed to the site!).

See: Link to a very long and unnecessary-to-render-here screencap of the old contaminated About page.

While making tweaks to the rochesterastronomy.org website and un-doing a menu expansion that shouldn't have been, I decided to check back on the CNYO site to see if I could play the same trick. And I couldn't. But, in exploring how to tailor the css in browser Developer Tools, I noticed a slew of embedded links and class="r67298bc6" calls that I had not previously for Klonopin/Clonazepam.

Klonopin is "used to treat panic disorder (including agoraphobia) in adults." I am 100% for adults overcoming anxiety about being outdoors (because I do astronomy) or with groups (because being outside in the middle of nowhere while observing by yourself can get old when the first howls are heard). That said, advertise by injecting something into someone else's WordPress, please.

Files And Tools To Have On Hand

It's a short list, things you probably have on hand or quick access to anyway if you're maintaining your own website (self-hosting or, at least, self-sufficient on your hosting service), and things you should probably always have at-the-ready for when (not if) the situation calls for them.

  1. A local copy (on your computer) of the WordPress version your site is currently using. To be even more exhaustive, have copies of EVERYTHING installed on your site. Themes, plug-ins, custom css, media, etc.
  2. An FTP client of some kind (such as Filezilla)
  3. All the needed credentials to get into your site by (s)ftp
  4. A browser's Developer Tools to do that cursory inspection of whatever looks off (all modern browsers have this option, snoop around the Tools menu)
  5. Access to your site's database (likely phpmyadmin if you're running a WordPress site on a major hosting service)
  6. A tool for comparing the contents of folder A and folder B – this is how you'll find the difference in the content of the before/after files to see which file or files has/have been modified
  7. stackoverflow open. This link, this link, and this link were early first reads on my path (the third link was insightful, but it turns out my problem was in a WordPress core folder itself).

Step 1 – Replace WordPress Files (Carefully!)

A quick check that might save you several hours. This assumes you've done nothing to the contents of wp-admin and wp-includes, which is a pretty good assumption unless you dabble in .php files. All of your themes, plugins, and uploads are in wp-content.

  1. Uncompress your local copy of WordPress to have all of the core files/folders accessible for uploading
  2. Log out of your WordPress site if you're logged in (you probably don't have to do this)
  3. Get into your site account's directory with your (s)ftp client
  4. Rename wp-admin to wp-admin-old and wp-includes to wp-includes-old
  5. Upload the local WordPress copy's wp-admin and wp-includes
  6. When done, hit RELOAD on your problematic site.

For www.cnyo.org, my problem went away completely.

Step 2 – If Needed, Replace More WordPress Files (Very Carefully!)

My next step would have been to VERY CAREFULLY inspect and/or replace the .php files in the main directory (in the image above, index.php and downward, not touching .htaccess or .user.ini until a "Step 2b" when I found past versions to compare these to).

My next-next step would have been to go into wp-content with backup copies of my themes and plugins, but I would have done this VERY, VERY CAREFULLY.

Usual reminder from other sites – make sure you have a copy of wp-config.php somewhere, else you're in for a very long evening if you do something unwanted.

Step 3 – Inspect The Database

I started here, as it seemed the most likely culprit based on lots of searching for similar issues. Consensus among several discussions was that searching for the keywords:

evalgzinflateshell_execbase64_decode

in your database would produce the maximum bang-for-buck. As it happens, only "eval" existed in the CNYO database and, sadly, "eval" is a common combination of four letters on a science-based website (so 100 hits, many of "evaluation").

I was fortunate to do this first-pass, find little that looked problematic, and move on to Step 1 above instead.

Step 4 – Transfers Worked? Now Find The Problem (Locally)

For the academically curious… the wp-admin and wp-includes replacement solved my problem. Doing the next logical step of renaming the new wp-admin to wp-admin-new and renaming the old wp-admin-old to wp-admin (alphabetically, of course), I found the site remained in its original, untouched state – placing the problem file(s) squarely in wp-includes.

To find the official location, one needs a tool that will compare the contents of files and directories. In OSX, this is FileMerge in Xcode (search for the app or Command-Space Bar and type filemerge.app (see this link on stackexchange for a how-to). If you don't have Xcode, similar comparison tools exist on all OSs.

To start this, I moved wp-includes-old to my local machine so as to perform this comparison in filemerge. I renamed it to wp-includes in the pic below.

Once open, select the wp-includes in your clean, unzipped WordPress folder as the Left folder, then select the wp-includes(-old) as the Right folder. And run Compare.

Clicking on "Identical" on filemerge's righthand side to exclude all identical files, the list of changed files from old-to-new is small, with the offending .php file identified as temdfpg.php, which calls a .quota file that looks something like below:

php $IBmtKwVMuXQGpgbUH='ur'.'ld'.'ec'.'od'.'e'; … lots of and lots of scribble ... %22%2F%3E%3C%2F%73%62%65%7A%3E'));exit;

Problem identified, problem solved. All in all, I suspect I got lucky in doing a few very simple things to narrow down the location of the problem. I will hope for similar luck next time.

Some Recent WordPress Theme Hacking Issues (Mass Emails To Non-Existent Domain Name Addresses) And A Couple Of Things To Look For

I've spent the past few weeks making several new email client filters each day, with subject lists that look like the following:

Saturday and Sunday Only! Today's Special Buy of the Day!

One day sale event – today only, [ insert date here ]

[ insert name here ], check out this weeks specials – up to 75% off on selected items

[ Insert name here ], 10% discount for Brand or Generics for purchases placed before [ insert date here ]

We appreciate your past business with us

[ insert name here ], some of your items are back in stock now – complete your order today

[ insert date here ] deals and savings from your supplier

We're talking between 2000 and 5000 emails a day of the following mishmash, with various random email addresses used, random first names and message content (always with a link or two, plus unsubscribe links), and all to email addresses that look like

[ person's first name ] + @ + somewhereville.com

This was occurring from the website despite having many popular site-security-related PlugIns running: Including Wordfence, Sucuri Scanner, and Jetpack (not that Jetpack would protect from site problems). As it turns out, Sucuri *might* have found the problem had I installed it (at least) 6 months ago.

The amount of email has gotten so bad I the past few weeks that the site itself has been taken down thrice by my +10-year-long hosting company (web.com. Can't blame them for this one). Before telling you about where the problem eventually settled, I'll note the following attempts to find the problem, listed below.

Steps To Diagnose The Problem

1. Taking the site down – which worked

2. Running diffs on all of of the files in the WordPress install (against a freshly downloaded copy from wordpress.org) – this helped greatly

3. Deleting the many files no longer use by wordpress (having run it since 2.1.something and allowing WordPress to do auto-updates) – no affect

4. Scouring my hosting folder for hidden files, modified .htaccess files, or anything else – nothing found

5. Looking for date changes on .jpg files to see if malicious code had been embedded into one of the images that always loads with the site – nothing found

6. The big one – switching from my primary theme (a heavily modified version of relaxation 3 column from 2006) to one of the provided WordPress themes.

The Problem, Localized

The problem, then, was localized to my old theme – which could have meant one of two things

1. Something in the php was causing the problem by being too old (a piece of php that WordPress recommended removing from all themes – that I never read about)

2. Something was, despite having my permissions set for read-only on the server (because this theme is never updated by WordPress), tweaked in one of the theme files (which turned out to be the case)

In my case, a few modifications had been made to theme files over 6 months ago that sat dormant in the .php files until something eventually came along to start spitting out beaucoup spam.

NOTE: Everywhere you see "->" and "< -", these have been replaced from "<" and ">" to keep anything from being read by your browser)

1. This nasty piece of work was deposited into an index.php file many moons ago (but the file date had not changed, so it went unnoticed)

2015august27_infected_1

The following had made into ALL of the theme index.php files in my WordPress install

2015august27_infected_2

These are to be contrasted with the following, more generic look for the top of an index.php file:

?php get_header(); ?

Decidedly different.

These are both interesting, but turned out to not be the problems. Instead, my relaxation-3-column theme had its 404.php, archive.php, and index.php files modified at some point in the distant past (at least as early as November of last year) with the following new lines:

404.php

2015august27_infected_3

vs.

->?php // relaxation 3 column

->?php get_header(); ?< -

archive.php

2015august27_infected_4
vs.
->div id="content"< -

index.php

2015august27_infected_5
vs.
->?php function arphabet_widgets_init()
...

These beg the question – how does one find out that this stuff isn't supposed to be in a theme file?

The answer, assuming you know a little php, is to compare and contrast you potentially older theme files with new theme files (such as those pre-installed in WordPress). In all the above cases, available themes look like the "after" files above, with unreadable code not present in the tops of the files.

So, long-short, if you find your inbox stuffed with hundreds or thousands of spam samples coming from your own domain, a good first place to look is your running theme. Much like the BSG Episode "33", you may find yourself NOT getting spam after a certain period of time if you make a simple change from one theme to another (certainly a simple way to determine if the attack is from the theme or not).

Solution And Testing

The test for the modifications was simple:

First, backup your theme files to your local machine (or make a folder in your directory tree somewhere)

Second, after checking and (if necessary) making modifications, replace your index.php file FIRST, as this is the basis for your theme (and what WordPress looks for first in the theme). Your site will load, although it may look like hell.

Third, replace all those theme files which didn't have something odd in them (like the gobbledegook above) and reload you site. Then, WAIT to see if you get spam (for my problems above, this took about, honestly, 33 seconds). Your site may still look like hell depending.

Fourth, change other problem files and upload them 1-at-a-time, then reload and wait to see if the spam starts.

Fifth – repeat #4 until your theme is all back up

Sixth – when all uploaded, change your theme permissions to READ-ONLY (although this did not help me)

With luck, your spamming problem fit the mold of the above and google brought you to a page that would help. So say we all.