Posted on

Some Recent WordPress Theme Hacking Issues (Mass Emails To Non-Existent Domain Name Addresses) And A Couple Of Things To Look For

I've spent the past few weeks making several new email client filters each day, with subject lists that look like the following:

Saturday and Sunday Only! Today's Special Buy of the Day!

One day sale event – today only, [ insert date here ]

[ insert name here ], check out this weeks specials – up to 75% off on selected items

[ Insert name here ], 10% discount for Brand or Generics for purchases placed before [ insert date here ]

We appreciate your past business with us

[ insert name here ], some of your items are back in stock now – complete your order today

[ insert date here ] deals and savings from your supplier

We're talking between 2000 and 5000 emails a day of the following mishmash, with various random email addresses used, random first names and message content (always with a link or two, plus unsubscribe links), and all to email addresses that look like

[ person's first name ] + @ + somewhereville.com

This was occurring from the website despite having many popular site-security-related PlugIns running: Including Wordfence, Sucuri Scanner, and Jetpack (not that Jetpack would protect from site problems). As it turns out, Sucuri *might* have found the problem had I installed it (at least) 6 months ago.

The amount of email has gotten so bad I the past few weeks that the site itself has been taken down thrice by my +10-year-long hosting company (web.com. Can't blame them for this one). Before telling you about where the problem eventually settled, I'll note the following attempts to find the problem, listed below.

Steps To Diagnose The Problem

1. Taking the site down – which worked

2. Running diffs on all of of the files in the WordPress install (against a freshly downloaded copy from wordpress.org) – this helped greatly

3. Deleting the many files no longer use by wordpress (having run it since 2.1.something and allowing WordPress to do auto-updates) – no affect

4. Scouring my hosting folder for hidden files, modified .htaccess files, or anything else – nothing found

5. Looking for date changes on .jpg files to see if malicious code had been embedded into one of the images that always loads with the site – nothing found

6. The big one – switching from my primary theme (a heavily modified version of relaxation 3 column from 2006) to one of the provided WordPress themes.

The Problem, Localized

The problem, then, was localized to my old theme – which could have meant one of two things

1. Something in the php was causing the problem by being too old (a piece of php that WordPress recommended removing from all themes – that I never read about)

2. Something was, despite having my permissions set for read-only on the server (because this theme is never updated by WordPress), tweaked in one of the theme files (which turned out to be the case)

In my case, a few modifications had been made to theme files over 6 months ago that sat dormant in the .php files until something eventually came along to start spitting out beaucoup spam.

NOTE: Everywhere you see "->" and "< -", these have been replaced from "<" and ">" to keep anything from being read by your browser)

1. This nasty piece of work was deposited into an index.php file many moons ago (but the file date had not changed, so it went unnoticed)

2015august27_infected_1

The following had made into ALL of the theme index.php files in my WordPress install

2015august27_infected_2

These are to be contrasted with the following, more generic look for the top of an index.php file:

?php get_header(); ?

Decidedly different.

These are both interesting, but turned out to not be the problems. Instead, my relaxation-3-column theme had its 404.php, archive.php, and index.php files modified at some point in the distant past (at least as early as November of last year) with the following new lines:

404.php

2015august27_infected_3

vs. 

->?php // relaxation 3 column

->?php get_header(); ?< -

archive.php

2015august27_infected_4
vs. 
->div id="content"< -

index.php

2015august27_infected_5
vs. 
->?php function arphabet_widgets_init()
...

These beg the question – how does one find out that this stuff isn't supposed to be in a theme file?

The answer, assuming you know a little php, is to compare and contrast you potentially older theme files with new theme files (such as those pre-installed in WordPress). In all the above cases, available themes look like the "after" files above, with unreadable code not present in the tops of the files.

So, long-short, if you find your inbox stuffed with hundreds or thousands of spam samples coming from your own domain, a good first place to look is your running theme. Much like the BSG Episode "33", you may find yourself NOT getting spam after a certain period of time if you make a simple change from one theme to another (certainly a simple way to determine if the attack is from the theme or not).

Solution And Testing

The test for the modifications was simple:

First, backup your theme files to your local machine (or make a folder in your directory tree somewhere)

Second, after checking and (if necessary) making modifications, replace your index.php file FIRST, as this is the basis for your theme (and what WordPress looks for first in the theme). Your site will load, although it may look like hell.

Third, replace all those theme files which didn't have something odd in them (like the gobbledegook above) and reload you site. Then, WAIT to see if you get spam (for my problems above, this took about, honestly, 33 seconds). Your site may still look like hell depending.

Fourth, change other problem files and upload them 1-at-a-time, then reload and wait to see if the spam starts.

Fifth – repeat #4 until your theme is all back up

Sixth – when all uploaded, change your theme permissions to READ-ONLY (although this did not help me)

With luck, your spamming problem fit the mold of the above and google brought you to a page that would help. So say we all.

Posted on

Private Internet Access, OpenVPN (2.3.2), and Ubuntu 14.04 (.2 LTS) – Yet Another Reported Way To Get Them Working (And The Only One That Works For Me)

Update: 17 May 2016 – Doesn't happen often and is always nice to see. Thanks to Lucas Nell (on 26 April – that's my bad) for taking one additional step out of the whole process with the script below. Simply replace "[put-your-password-here]" with your actual password (and no brackets), same the below as some_name.script (or whatever), chmod +x some_name.script to make it official, and you're good to go to make the additional mod.

sudo -s

cd /etc/NetworkManager/system-connections

export pwd="\n[vpn-secrets]\npassword=[put-your-password-here]"

for f in PIA*
do
  sed -i 's/password-flags=1/password-flags=0/g' "${f}"
  echo -e "${pwd}" >> "${f}"
done

exit

If you sign up for an account with Private Internet Access (and this may go for some other VPN providers as well) and follow the only prominent Ubuntu link (12.04) in the Support Section (www.privateinternetaccess.com/pages/client-support/ubuntu-openvpn), you'll be taken to a reasonably straightforward 9-step process that walks you through the OpenVPN setup – from the install_ubuntu.sh script download to the selection of PIA-points (I just made that up) in your Network Manager GUI (that radial wifi icon or arrows in the upper-right corner).

That is, for Ubuntu 12.04.

The Problem

If you try this in Ubuntu 14.04, everything more-or-less looks and runs the same way. That said, when you try to connect to a PIA-point in the Network Manager, nothing happens. Your wifi radial doesn't change, flash, or provide any indication that something has gone right or wrong. More importantly (to the lack of feedback, anyway), you are not asked for your PIA password (having put in your username in the install process). This lack of password requesting turns out to be the real kicker (and diagnostic for the fix presented down below).

If you look in /etc/NetworkManager/system-connections, you'll see that all of the PIA files have been successfully installed.

-rw——- 1 root root 326 Jul 16 16:14 PIA – AU Melbourne
-rw——- 1 root root 313 Jul 16 16:14 PIA – AU Sydney
-rw——- 1 root root 313 Jul 16 16:14 PIA – Brazil
-rw——- 1 root root 316 Jul 16 16:14 PIA – CA North York
-rw——- 1 root root 321 Jul 16 16:14 PIA – CA Toronto
-rw——- 1 root root 313 Jul 16 16:14 PIA – France
-rw——- 1 root root 315 Jul 16 16:14 PIA – Germany
-rw——- 1 root root 312 Jul 16 16:14 PIA – Hong Kong
-rw——- 1 root root 350 Jul 17 15:49 PIA – Ireland
-rw——- 1 root root 313 Jul 16 16:14 PIA – Israel
-rw——- 1 root root 311 Jul 16 16:14 PIA – Japan
-rw——- 1 root root 313 Jul 16 16:14 PIA – Mexico
-rw——- 1 root root 314 Jul 16 16:14 PIA – Netherlands
-rw——- 1 root root 310 Jul 16 16:14 PIA – Romania
-rw——- 1 root root 313 Jul 16 16:14 PIA – Russia
-rw——- 1 root root 312 Jul 16 16:14 PIA – Singapore
-rw——- 1 root root 313 Jul 16 16:14 PIA – Sweden
-rw——- 1 root root 317 Jul 16 16:14 PIA – Switzerland
-rw——- 1 root root 313 Jul 16 16:14 PIA – Turkey
-rw——- 1 root root 319 Jul 16 16:14 PIA – UK London
-rw——- 1 root root 329 Jul 16 16:14 PIA – UK Southampton
-rw——- 1 root root 327 Jul 16 16:14 PIA – US California
-rw——- 1 root root 315 Jul 16 16:14 PIA – US East
-rw——- 1 root root 321 Jul 16 16:14 PIA – US Florida
-rw——- 1 root root 321 Jul 16 16:14 PIA – US Midwest
-rw——- 1 root root 331 Jul 16 16:14 PIA – US New York City
-rw——- 1 root root 321 Jul 16 16:14 PIA – US Seattle
-rw——- 1 root root 334 Jul 16 16:14 PIA – US Silicon Valley
-rw——- 1 root root 317 Jul 16 16:14 PIA – US Texas
-rw——- 1 root root 315 Jul 16 16:14 PIA – US West

My first attempts at troubleshooting brought me to the installing privateinternetaccess on ubuntu 14.04 LTS page at askubuntu.com. The first response seems to be regurgitating the 12.04 installation process on the PIA site (which doesn't work. For me, anyway), while the second response provides a list of installs that the install_ubuntu.sh script may or may not have successfully installed.

sudo apt-get install openvpn network-manager-openvpn network-manager-openvpn-gnome

The second commenter then walks through the install process as if the .ovpn config files didn't exist (setting up from scratch, which can be laborious if you want to add all of the PIA points) but uses the contents of the openvpn.zip file downloaded by the question-asker.

The fix to the whole matter is partly in the questioner and second answer, but some additional work needs to be done. What's described below is the process I used to figure out what was going on (showing all work), including using some alternatively-official .ovpn files (and the official ca.crt and crl.pem files provided by PIA).

The Diagnosing (What May Have Brought You Here)

With the failure to get any feedback from Network Manager (or the GUI) after the install, I went straight to the syslog to see if anything revealing appears (var/log/syslog). The error report for my VPN connection attempts reads as follows:

cd /var/log/syslog
more syslog

Jul 16 08:54:04 randommachine NetworkManager[13049]: Starting VPN service 'openvpn'…
Jul 16 08:54:04 randommachine NetworkManager[13049]: VPN service 'openvpn' started (org.freedesktop.NetworkManager.openvpn), PID 13164
Jul 16 08:54:04 randommachine NetworkManager[13049]: VPN service 'openvpn' appeared; activating connections
Jul 16 08:54:04 randommachine NetworkManager[13049]: [1437051244.977042] [nm-vpn-connection.c:1374] get_secrets_cb(): Failed to request VPN secrets #2: (6) No agents were available for this request.
Jul 16 08:54:04 randommachine NetworkManager[13049]: Policy set 'randomrouter' (wlan0) as default for IPv4 routing and DNS.
Jul 16 08:54:10 randommachine NetworkManager[13049]: VPN service 'openvpn' disappeared

A google search for "Failed to request VPN secrets #2" (I can't stress enough the value of quotes in troubleshooting Linux issues) dragged me to several pages that didn't directly address my Network Manager issue, but indicated that one should consider running OpenPVN from the Terminal anyway. Extracting openvpn.zip (downloaded from the PIA website) and cd'ing into that folder (I assume you're in Downloads), the following commands:

cd Downloads
unzip openvpn.zip
openvpn US\ East.ovpn

Produces the following output – asking for username and password, but then failing to connect (and I include all the output below, assuming the error brought you here).

Thu Jul 16 09:06:55 2015 OpenVPN 2.3.2 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [eurephia] [MH] [IPv6] built on Dec 1 2014
Enter Auth Username:pXXXXXXX
Enter Auth Password:
Thu Jul 16 09:07:11 2015 UDPv4 link local: [undef]
Thu Jul 16 09:07:11 2015 UDPv4 link remote: [AF_INET]208.167.254.223:1194
Thu Jul 16 09:07:11 2015 WARNING: this configuration may cache passwords in memory — use the auth-nocache option to prevent this
Thu Jul 16 09:07:11 2015 [Private Internet Access] Peer Connection Initiated with [AF_INET]208.167.254.223:1194
Thu Jul 16 09:07:14 2015 ERROR: Cannot ioctl TUNSETIFF tun: Operation not permitted (errno=1)
Thu Jul 16 09:07:14 2015 Exiting due to fatal error

That said, when you apply root privileges:

sudo openvpn US\ East.ovpn

You get the following:

Thu Jul 16 09:07:28 2015 OpenVPN 2.3.2 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [eurephia] [MH] [IPv6] built on Dec 1 2014
Enter Auth Username:pXXXXXXX
Enter Auth Password:
Thu Jul 16 09:07:36 2015 UDPv4 link local: [undef]
Thu Jul 16 09:07:36 2015 UDPv4 link remote: [AF_INET]208.167.254.223:1194
Thu Jul 16 09:07:36 2015 WARNING: this configuration may cache passwords in memory — use the auth-nocache option to prevent this
Thu Jul 16 09:07:37 2015 [Private Internet Access] Peer Connection Initiated with [AF_INET]208.167.254.223:1194
Thu Jul 16 09:07:39 2015 TUN/TAP device tun0 opened
Thu Jul 16 09:07:39 2015 do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
Thu Jul 16 09:07:39 2015 /sbin/ip link set dev tun0 up mtu 1500
Thu Jul 16 09:07:39 2015 /sbin/ip addr add dev tun0 local 10.134.1.6 peer 10.134.1.5
Thu Jul 16 09:07:39 2015 Initialization Sequence Completed

Works great (can verify at whatsmyipaddress.com or other). And it asks for your username (because the .ovpn files haven't been configured yet from the script) and password, so we were already ahead of the game from the Network Manager GUI.

OpenVPN seems to work fine and the .ovpn files work, so the problem is somewhere in Network Manager or how it and OpenVPN are interacting (which I've not yet found the answer to). Now, you're supposed to be asked for the password when you try to establish the VPN connection with Network Manager. To see if that was the only problem with the .ovpn files, I simply added my password to the US East.ovpn file as follows (in US /East.ovpn):

nano US /East.ovpn

And add the following somewhere in the file:

password=put-your-password-here

Then restart the Network Manager (and wait a few seconds)

sudo service network-manager restart

And that didn't work. That said, there's another password flag in the file (aptly names password_flags) to play with. A search for details lead me to a post at forums.kali.org that goes into some detail about Network Manager NOT strong the VPN password correctly because the user keyring isn't root-accessible.

Changing password-flags from 1 to 0 and attempting to connect with Network Manager = success!

So, the problem is somewhere in the failed password negotiation between Network Manager and OpenVPN, and providing that info in the .ovpn files from openvpn.zip and a network-manager restart solves the problem.

Now then, the differences between the .ovpn files in openvpn.zip (download-able from https://www.privateinternetaccess.com/openvpn/openvpn.zip) and the PIA VPN files installed using install_ubuntu.sh are as follows:

US East.ovpn

client
dev tun
proto udp
remote us-east.privateinternetaccess.com 1194
resolv-retry infinite
nobind
persist-key
persist-tun
ca ca.crt
tls-client
remote-cert-tls server
auth-user-pass
comp-lzo
verb 1
reneg-sec 0
crl-verify crl.pem

PIA US East

[connection]
id=PIA – US East
uuid=856fc7ec-cd90-4ab1-96c1-2d827e46ea8f
type=vpn
autoconnect=false

[vpn]
service-type=org.freedesktop.NetworkManager.openvpn
username=p9681681
comp-lzo=yes
remote=us-east.privateinternetaccess.com
connection-type=password
password-flags=0
ca=/etc/openvpn/ca.crt

[ipv4]
method=auto

The new format is so clean! It's also the format you get if you go through the New Connection process through Network Manager. The formatting seems to be important for the fix I'm going to propose below, so I'm going to modify the newer format files below.

The Solution

The solution is, after many hours, stupid-simple – run the install_ubuntu.sh as described on the PIA website (which will also make you install a few extra programs if you don't have them already- and it places ca.crt into your OpenVPN folder, which is then called by the VPN files), modify all of the PIA files in your /etc/NetworkManager/system-connections folder by putting your password into each (in the format as below), and restart network-manager from the Terminal. That, in theory, should be it. You'll have to have root access to do this, though, as the file permissions are all (or should be) 600.

1. https://www.privateinternetaccess.com/pages/client-support/ubuntu-openvpn

2. Open Terminal

3. Move to the system-connections folder:

cd /etc/NetworkManager/system-connections

4. Edit all the PIA files. To each of the PIA files, all you have to do is add the following:

[vpn-secrets]
password=put-your-password-here

The [vpn-secrets] is important! I would have thought this to be a comment block for organizational purposes, but adding thr password line alone won't cut it.

NOTE: If you're trying to connect through the GUI and the VPN Connections DO NOT appear in the list – provided your password is in the file, your problem is very likely that the file permissions are wrong. If they're not -rw——, then Network-Manager will not read them.

5. Extra bookkeeping step: double-confirm the permissions on the PIA files: 

chmod 600 PIA*

6. Restart network-manager

sudo service network-manager restart

I do not know if/when the fix will come in between OpenVPN and Network Manager (or something else in Ubuntu) that will obviate the need for this workaround. In the meantime, the procedure above works just fine (works at all) on a clean install of 14.04.2 LTS. The problem seems to be with OpenVPN as it plays with 14.04, a recurring theme I've found from lots of people (or, perhaps more specifically, the use of the GUI to call OpenVPN). Given several reports of PIA/14.04, I'm surprised there isn't more, perhaps specifically on the PIA website, to address this issue. Hopefully a proper fix from PIA, OpenVPN, or Ubuntu developers in en route.

Happy more safe/more secure surfing. And if you're so inclined, the Litecoin bubble has not, yet, right now, burst (scroll to the bottom of http://www.somewhereville.com/?p=1896).