home

Archive for the 'scripts' Category

Stupid-Simple (*nix-Specific) Sed Scripts To Get (All Current) Gaussian09 Output Files Working With aClimax

Monday, September 1st, 2014

The following three snippets of Gaussian output are for an optimization and normal mode analysis of simple olde methane (CH4).

...
 ******************************************
 Gaussian 03:  EM64L-G03RevE.01 11-Sep-2007
                31-Aug-2014 
 ******************************************
...
 incident light, reduced masses (AMU), force constants (mDyne/A),
 and normal coordinates:
                     1                      2                      3
                     T                      T                      T
 Frequencies --  1356.0070              1356.0070              1356.0070
 Red. masses --     1.1789                 1.1789                 1.1789
 Frc consts  --     1.2771                 1.2771                 1.2771
 IR Inten    --    14.1122                14.1122                14.1122
 Atom AN      X      Y      Z        X      Y      Z        X      Y      Z
   1   1     0.02  -0.42   0.43    -0.34  -0.13  -0.08    -0.36  -0.23  -0.23
   2   6     0.00   0.08  -0.09     0.00   0.09   0.08     0.12   0.00   0.00
...
 -------------------
 - Thermochemistry -
 -------------------
 Temperature   298.150 Kelvin.  Pressure   1.00000 Atm.
 Atom  1 has atomic number  1 and mass   1.00783
...
...
 ******************************************
 Gaussian 09:  EM64L-G09RevA.02 11-Jun-2009
                31-Aug-2014 
 ******************************************
...
 incident light, reduced masses (AMU), force constants (mDyne/A),
 and normal coordinates:
                     1                      2                      3
                     T                      T                      T
 Frequencies --  1356.0058              1356.0058              1356.0058
 Red. masses --     1.1789                 1.1789                 1.1789
 Frc consts  --     1.2771                 1.2771                 1.2771
 IR Inten    --    14.1123                14.1123                14.1123
  Atom  AN      X      Y      Z        X      Y      Z        X      Y      Z
     1   1    -0.03   0.42   0.43    -0.34  -0.14   0.07    -0.36  -0.23   0.23
     2   6     0.00  -0.08  -0.10     0.01   0.10  -0.08     0.12   0.00   0.00
...
-------------------
 - Thermochemistry -
 -------------------
 Temperature   298.150 Kelvin.  Pressure   1.00000 Atm.
 Atom     1 has atomic number  1 and mass   1.00783
...
...
 ******************************************
 Gaussian 09:  EM64L-G09RevD.01 24-Apr-2013
                31-Aug-2014 
 ******************************************
...
 incident light, reduced masses (AMU), force constants (mDyne/A),
 and normal coordinates:
                      1                      2                      3
                     ?A                     ?A                     ?A
 Frequencies --   1356.0132              1356.0132              1356.0132
 Red. masses --      1.1789                 1.1789                 1.1789
 Frc consts  --      1.2771                 1.2771                 1.2771
 IR Inten    --     14.1119                14.1119                14.1119
  Atom  AN      X      Y      Z        X      Y      Z        X      Y      Z
     1   1     0.02   0.42   0.43     0.34  -0.14   0.08    -0.36   0.23  -0.23
     2   6     0.00  -0.08  -0.09    -0.01   0.09  -0.08     0.12   0.00   0.00
...
 -------------------
 - Thermochemistry -
 -------------------
 Temperature   298.150 Kelvin.  Pressure   1.00000 Atm.
 Atom     1 has atomic number  1 and mass   1.00783
...

Two of these things are not like the other. The data’s nearly identical (and thank heavens. Unfortunately, Gaussian09 D.01 didn’t see the fully-optimized methane as belonging to the Td point group – despite all three versions being run with the same exact input file – but a rigorous re-symmetrization would have taken care of that), but there are some subtle formatting differences between all three versions (including differences between both Gaussian09 versions) that cause the venerable, all-encompassing aClimax program (developed by Timmy, the venerable, all-encompassing A. J. Ramirez-Cuesta) to throw out the following errors for all three cases when you use *.log files from a *nix (UNIX, Linux) machine.

Serious Error: A-CLIMAX has encountered an unhanded error. Please Save your data and contact support
aClimax: Quote Error Number 9
Error Loading File: Error reading data. Please check and try again.
aClimax: WARNING loaded file containing no frequencies

Problem number 1 is the existence of *nix newlines (carriage returns) in the *.log files coming off a *nix machine. Performing a conversion from *nix to DOS (for myself, using LineBreak in OSX, but tofrodos works just as well), the Gaussian03 file now opens just fine in aClimax:

File Loaded: Data Loaded Succesfully [sic].

This, unfortunately, does not improve the matter with the Gaussian09 files, which produce the following error:

Error: One of the numbers you have entered is of the wrong type.Please recheck and try again
Error Loading File: Error reading data. Please check and try again.

Given how little of the .log file aClimax actually needs to produce simulated inelastic neutron scattering (INS) spectra, I ran the methane normal mode analyses in three different Gaussian versions to determine what, in G09, was changed to make it just un-G03 enough to fail to load. With those changes figured out, I had a Perl script drafted up that would have converted everything back to the original G03 format. It was awesome. That said, after a small amount of testing to see where aClimax’s sensitivities lay, I discovered that very little of the .log file contents needed to be changed out, meaning that simple sed scripts would work just as well for those of us using our Windows boxes (or VirtualBox emulations) only for that “one stupid program” that keeps us having to log in (and, by that, I mean that we have sed already on our computers).

So, the problems between G09 and aClimax not related to carriage returns lie in two places.

1. The spacing of “Atom AN” – at the top of the eigenvector lists are the column labels, beginning with “Atom AN” – or something very close to “Atom AN” (the “|” in the boxes below mark the left edge of the output):

G03 E01 | Atom AN
G09 A02 |   Atom  AN
G09 D01 |  Atom  AN

Yes, the addition of a space or two results in a read error by aClimax. I would call this an… aggressive stringency in aClimax. That said, what did the original space in G03 versions not do that they do do in G09?

2. The spacing of “Atom N” – In the “Thermochemistry” section below the eigenvectors, atomic masses are listed as “Atom N” – or something very close to “Atom N” (again, the “|” in the boxes below mark the left edge of the output):

G03 E01 |  Atom  1
G09 A02 |    Atom     1
G09 D01 |   Atom     1

This change in spacing is also enough to cause aClimax to error out.

The Solution

A small sed script performs the necessary conversions on your *nix box (including OSX) for all .log files in a directory without issue:

#!/bin/sh

# This section converts all .log files to aClimax-friendly G03-ish format
find . -type f -name '*.log' -print | while read i
do
sed 's|  Atom  AN| Atom AN |g' $i > $i.aclimaxconversion_step1
sed 's| Atom   | Atom|g' $i.aclimaxconversion_step1 > $i.aClimaxable.log
rm $i.aclimaxconversion_step1
done

# This section converts all .out files to aClimax-friendly G03-ish format
find . -type f -name '*.out' -print | while read i
do
sed 's|  Atom  AN| Atom AN |g' $i > $i.aclimaxconversion_step1
sed 's| Atom   | Atom|g' $i.aclimaxconversion_step1 > $i.aClimaxable.out
rm $i.aclimaxconversion_step1
done

But Wait! Running G0* Jobs Under *nix? Convert To DOS Carriage Returns

The final problem halting your aClimax spectrum generation is the DOS carriage return (^M). For those running DOS-based Gaussian calculations (likely with a .out suffix), your conversion with the short script above (under *nix) likely (hopefully) worked just fine. For those running under *nix, you performed the conversion and still received the following aClimax error:

Serious Error: A-CLIMAX has encountered an unhanded error. Please Save your data and contact support
aClimax: Quote Error Number 9
Error Loading File: Error reading data. Please check and try again.
aClimax: WARNING loaded file containing no frequencies

The solution is an additional line in the sed script that will globally replace all *nix newlines with proper DOS carriage returns. The .out section remains the same.

#!/bin/sh

# This section converts all .log files to aClimax-friendly G03-ish format
find . -type f -name '*.log' -print | while read i
do
sed 's|  Atom  AN| Atom AN |g' $i > $i.aclimaxconversion_step1
sed 's| Atom   | Atom|g' $i.aclimaxconversion_step1 > $i.aclimaxconversion_step2
# This section converts your *nix newlines into DOS carriage returns
CR=`echo "\0015"`  # define the Carriage Return
sed -e "s/$/${CR}/g" $i.aclimaxconversion_step2 > $i.aClimaxable.log
done
# this cleans up your folder of temp files
rm *.aclimaxconversion_step1
rm *.aclimaxconversion_step2

# This section converts all .out files to aClimax-friendly G03-ish format
find . -type f -name '*.out' -print | while read i
do
sed 's|  Atom  AN| Atom AN |g' $i > $i.aclimaxconversion_step1
sed 's| Atom   | Atom|g' $i.aclimaxconversion_step1 > $i.aClimaxable.out
rm $i.aclimaxconversion_step1
done

Q. But what if I run the *nix-to-DOS version of the script on an already DOS-output file?

A1. The simple answer is that you’ll make your text file double-spaced (which is bad enough). aClimax will then provide the following error when you try to open it:

Error Reading File: Unexpected File End. File May be incorrect or corrupt.
Error Loading File: Error reading data. Please check and try again.

A2. I will assume that your problem is that you’re running the script in DOS to try to get your G09 to read more like G03. In this case (assuming you’re generating .out files), you’ll want to use a text editor to make the replacements described above (which is to say, that Perl script might makes it way to this page eventually. If you write a DOS .bat file or similar script for all OS’s, I’d be happy to link to it).

“From Kurdistan With Love” or Some Things To Do Before And/Or After Your WordPress Site Gets Hacked

Thursday, December 12th, 2013

“Hopefully, because he’s busy.” – Commissioner Gordon, The Dark Knight

On the plus side, www.somewhereville.com received its first update in just over 5 months. On the minus side, the new post was less than useful in many ways. I received a timely email from Dr. Obi Griffith of the Washington University in St. Louis Division of Oncology noting that my entire site was differently-down (thanks to the hijacking of my Sanger (And Illumina 1.3+ (And Solexa)) Phred Score (Q) ASCII Glyph Base Error Conversion Tables page that he linked to on a biostars site thread – so my thanks to Obi for catching something I likely would have gone weeks without noticing!).

The snapshot below shows the state of swv as of 3 December 2014. On the bright side (minus a friendly conspiracy to get someone else in trouble), I can say with some certainty that Serwan performed the content-ectomy (twitter: @S3RW4N, current email (although I suspect it won’t last long): serwan_007 – at cymbal – hotmail.com, on the Facebook, etc. All sites subject to change as people try to track him/her down post-attack (he/she’s been prolific if nothing else)).

2013dec11_serwan_hack

Exhibit A. Flag is waving in the actual version.

Several problems. To begin, it’s a gaudy hack, complete with rolling text and techno music. Second, the Television New Zealand (TVNZ) news service thought this hack to be significant enough to warrant actual coverage on their website when a similar file-swap on a WordPress (or WordPress-esque) site brought down the Health and Sports Fitness Club in Sandringham (syracuse.com didn’t give me the time of day). I commend this Kurdish hacker group for their ratings. Third, the manner in which files were replaced in the blog (specifically meaning the index.php file) blocked every other post on the site from being accessed, so every link anyone had posted to a page anywhere else on the Internets was made useless.

That said, I appreciate that Serwan generally performs fairly benign attacks on websites. File replacements were clearly identified from a simple date sorting, the important MySQL database content wasn’t touched, and Serwan even went as far as to set up a second Admin account so that I could quickly retake control of the site.

So, in light of the plight of the Kurdish people, I left the hacked version up for a few hours as I pondered what to do, which I discuss below.

My Spotty Procedure For Recovery:

What follows is a list of obvious and less obvious things to consider when recovering your WordPress blog from a hack. There are plenty of websites that show how to protect your site in the first place, then others that explain how to revive it (provided you do your own due diligence and back your site up regularly enough). What’s below is not complete, but you can rest assured that google is your friend in such matters, so keep your keywords targeted and see what comes up.

General Considerations:

1. Don’t use your blog. My last post at the time dated back to June 25th, during which time I’ve made several full backups (and kept WordPress up-to-date, the last time being 7 November 2013) of my entire site. In this respect, I was well set up to quickly recover from a hacking incident.

2. Keep a copy of your current running version of WordPress handy for file replacements. In my case, index.php was written over. All I had to do to recover was uncompress my WordPress  3.7.1 download, upload index.php to my server, and the site was back and running.

3. Have you backed up lately? This phrase has been in the .sig of my emails for many, many years. If your entire life is lived in the Googleverse (email, images, documents, etc.), then you’re fine until the Earth’s magnetic poles shift and wipe all the hard drives out (just kidding. I think). If you’re a computational scientist and have TBs of data, it’s up to you to make sure you have access to it all again. Same applies to WordPress. I’ve a biweekly alarm that tells me to back up several websites and I’ve an encrypted .txt file with all of the login info and steps needed to perform this backup. You should absolutely be doing the same if you’re not.

4. Set up an additional Administrator. In my case, my admin account was hacked to change the associated user email address to Serwan’s email. Obviously, attempting to log in, change the password, or what have you simply sent little pings of your futile attempts to the hacker. Having that second admin account will allow you to reroute your login efforts (and if they’re both hacked into, there’s still a way around. Will get to below).

5. Make a real password. At the risk of de-securing my sites by providing personal info, my typical password looks something like this:

d@!25fj014or&ydoSDfu

20 characters long, upper and lower, numbers, and non-alphanumeric characters. If you care about your site security, stay the hell away from the dictionary.

6. Dry-run your SHTF moment. Are you a survivalist? Can you identify edible berries by sight, build a lean-to, or stitch an open wound? Or are you the Marty Stouffer of the camping section at Target? If you’ve never had to work your way back from a complete disaster, you likely won’t know how to do it either quickly, efficiently, or securely.

Ergo, do another WordPress installation in a sub-folder of your main installation, create a new database, make your site pretty, perform a full backup of your database and uploaded media, then break it, either by deleting core files or corrupting your database (deleting a table would do the trick). If you can put the site back together again (the uploading of the database back onto your server likely being the worst part of the whole process), you’re likely in good shape for the real deal.

7. Harden WordPress. The good people at WordPress even tell you how to (although, admittedly, I thought I did all of this, so maybe there’s something being missed that will go into a future iteration of this page).

8. Get rid of “admin.” Several of the sites discussing WordPress hacks report that having this default account (or account default’ed) is a top-5 problem when trying to keep people out of your site. So get rid of it. Easily. Set up a new account, give it administrative privileges, then delete the admin account, which will ask you to attribute the current admin posts to another admin account.

9. Delete deactivated plugins if you’re not going to use them. Plugins are developed by people. People often have lives that keep them from timely updates of security exploits. If you’re using a plugin, that’s one thing. If a deactivated plugin languishes in your plugins folder, never gets updated, and some hacker writes something specifically to exploit a security flaw in that old, poorly maintained plugin, that’s all on you. So don’t risk your pocket knife being a projectile as you walk into the MRI room and get rid of the knife before it comes a problem.

10. I know nothing about it yet, but am giving Wordfence a whirl presently.

11. Hey, check your blog every once in a while to make sure it’s still you and not Serwan.

For The Specific Attack (From Easy To Harder):

1. FTP in and check file dates. The offending .php files (index.php and a hello.php containing the techno) were both dated 3 December 2013. Everything else was, at its newest, 7 November 2013 (from my last WordPress update). This made finding the hacked and previously not-present files easy. A cluster of important files with identically modification times and dates is an easy giveaway.

2. FTP in and check ALL the file dates. One never knows when something else is going to be placed into a themes folder, plugin folder, etc., to keep track of site access (that’s why I delete all deactivated plugins). So, sort by date and scour the whole site for modifications and new files.

3. If you make it into your site, go right to your User Settings, change the email address, then change your password.

4. Check out something like Sucuri SiteCheck. Hopefully, this search will complement your initial search as well as test against known threats. I ran a Sucuri on a similarly-hacked site (in this case, indoorstinkbugtrap.com) and received the following notification of defacement (so the check worked).

2013dec11_securi_results

securi.net results for fellow victim indoorstinkbugtrap.com.

5. If you can’t make it in the front door, crawl through the plumbing. You can change your admin account from within MySQL using, for instance, phpMyAdmin (check your hosting provider for details if this is new information to you). In the case of phpMyAdmin, you can modify the admin account in six easy steps.

1. Log in to phpMyAdmin

2. Click on the Structure Button in wp_users (red circle)

2013dec11_serwan_hack_mysql_1

3. Click on Browse (told you this was easy)

2013dec11_serwan_hack_mysql_2

4. Click the edit button for your administrative account (red circle)

2013dec11_serwan_hack_mysql_3

5. Change the email address back to your email and delete the current password.

2013dec11_serwan_hack_mysql_4

6. Save and go back to our WordPress site, then request a new password.

And, While We’re At It:

Serwan’s twitter image currently features a white hat (the Gandalf-ian sign of a good guy/gal hacker) and a long list of sites that have been defaced with otherwise useless, feral medadata promoting Kurdish Hackers for google to get confused by. A search for somewhereville.com in google left the following bad taste in its results page for a week after:

Hacked By Serwan. Allah Is Greatest. Long Live Kurdistan. Thanks To All Kurdish Hackers. Follow @S3RW4N FB.com/Mr.S995

If I may be so bold (and I’ve told Serwan the same), the Kurdish people had a long history of getting steamrolled by an oppressive regime that, regretfully, first-world countries didn’t put enough into stopping or acknowledging until the tanks rolled South into Kuwait. If you’re gong to label yourself an ethical hacker, fine. Mangle the front-end of someone’s WordPress site. That said, you could be educating others on the Kurdish people by including a few links into your hack. I live in America, where certain news services use “Muslim” and “Islam” in headlines purely for shock value when they want to appeal to an audience so narrow-minded that their hearing is susceptible to the Casimir Effect. I recommend adding the wikipedia article on Kurdistan and the Al-Anfal Campaign to future hacks (and I’m sure Serwan could find more) to provide a little substance to your efforts unless, of course, your goal is just to be a stupid-ass script-kiddie hacker.

If you’re gonna hack, at least try to be productive. Meantime, this was a valuable lesson for myself on what to do to try to keep WordPress from falling into the same limbo during a time when I might not have had an hour to fix it.

The “Stone Boulders” – All 12-Group (4,096) and 16-Group (65,536) L/R Sticking Combinations In PDF Format

Wednesday, July 13th, 2011

George L. Stone's Stick Control

You should have a copy regardless (amazon.com direct link).

[Drafted as an article for somewhere, stuck here instead…]

You could spend your life on the first six pages of “Stick Control” and still not cover all the possibilities. Dynamics, accents, foot-hand, foot-foot, fast/slow, hands on top of foot patterns, feet on top of hand patterns, regroupings and accenting in 5-7-4 (regrouping of the 16 strokes per pattern), 7-5-4 (re-regrouping of the 16 strokes), yadda yadda. If you see the first six pages of Stick Control as just exercises, you miss the fantastic complexity YOU can introduce to constantly humble yourself while hovering over a practice pad.

As I look at the sets of exercises, I see what I assume most of us do – paradiddles, singles, doubles, multiple-hits of the same stick, some oddball patterns you start playing as written and then mess up without knowing, etc. The question I found myself asking was “What drove Stone to use this particular sequence?” I eventually turned that question around and decided to answer the question “What did Stone leave out?” The PDFs linked to this article are what I’ve affectionately come to call the “Stone Boulder,” providing EVERY sticking combination Stone included and every other combination he didn’t. Some intro to how and why is below, followed by a bit of explanation. I think the patterns themselves are self-explanatory.

While not the most cite-able examples in all of genomics, there have always been passing references to drumming “being in someone’s DNA.” As it happens, drumming and biology did overlap in general approach during the mid-80’s-to-early-90’s (or so) in the great heyday of linear drumming (go dig out your Murray Houllif and Gary Chaffee books). The idea is simple: no two drums/cymbals hit at the same time, producing an often staccato and generally (well, to my ears anyway) more melodic sound from the drums (and much easier to transcribe than some of the superhuman overlapping rhythms people are having fun playing today). Ignoring the complexities of 3.5 billions years of evolution, DNA works the same way as these linear patterns to convey a message. The four bases in your DNA, A (adenine), C (cytosine), G (guanine), and T (thymine), act as a code that is read like those old drum beats were played – one at a time with no doubling-up please. The identical three million base-long DNA sequences in each of your cells (see CSI) could be turned from seemingly random patterns of [A,C,G,T] into seemingly random patterns of [L,R,B,H] (that’s left hand, right hand, bass drum foot, hi-hat foot), then some experimental linear drumming composer could “play” your genome. Better still, if the transcriber was as good as your cellular machinery, the entire performance could be written down and reconverted into [A,C,G,T] format exactly so you could be cloned and double-drum with someone who rushes and slows down just as much as you do.

While most people think of a drum programmer as someone who generates patterns on a computer, I took the route of programming to generate patterns to drum. I most certainly did NOT put the pattern pages together by hand (I promise, no mistakes). A small script in the Perl programming language used to generate DNA sequences did all of the dirty work (including making sure all patterns only appear ONCE in each document). The math for figuring out the total number of left/right patterns is quite simple. The number of combinations of unique sticking patterns for a particular pattern length is 2^n, n being the number of beats. For a single beat, that’s 2^1, or just 2, that single hit being performed with either the left or right hand. For a four-stroke pattern, that’s 2^4 (2 * 2 * 2 * 2), or 16 total patterns. These are shown below out of academic interest (although I hope you could write them down from memory).

Now, consider the first six pattern pages of the Stone book. 16 beats per pattern. That’s 2^16, or 65536 total patterns. At 20 patterns per page, the complete Stone book of these first six pages would take up 3,277 pages. At 2 seconds per pattern, you could rip through all 3,277 pages in about 36 hours 30 minutes (about the perceived length of a society gig).

For you fellow jazzers out there looking for a more swingin’ set, I’ve also included the same sticking deal with a triplet-feel set (12 beats instead of 16, so you’ve only got 4096 patterns to contend with, meaning you could play through the whole set in about 2 hours 30 minutes).

4,096 patterns are bad enough. 65,536 is borderline something uncouth. On the one hand, that’s a lot of patterns either way. On the other, for the obsessive compulsive readers, these are IT. There are no other 12- or 16-stroke sticking combinations that have a stick hitting on each beat (that is, no rests). As Terry Bozzio has said in one form or another in his many clinics introducing his ostinato independence exercises “once you’ve played through the 16, you’ve played every 16th note pattern there is.”

And it could be worse! If you wanted every combination of left, right and rest, that’s 3^16, or 43,046,721 patterns. At 2 seconds per pattern, that’s 23,915 hours, or about 2 years and 8 months. I pondered doing the same thing for all 16-note linear drum patterns (L,R,B,F), which would produce 4^16, or 4,294,967,296 patterns. That’s 2,386,093 hours, or 272 years and 3 months (that’s approaching four reincarnations of “no life”).

Each full page has three columns of 40 patterns (120 per page), producing a document that’s only 547 pages long (but entirely green-friendly in PDF format). You will note that most of the pages look like the same stupid thing. This is because the mechanism of generation for the sequences involved making single changes at a single position and walking down the entire 16-stroke sets until all changes had been accounted for. I become bored to tears staying on a single page and generally scroll at random and point the stick at the screen to pick a pattern to play. Be as methodical or all-over-the-map as you will.

Is there a good reason for doing this? Not particularly. There are lots of patterns here that are a mechanical challenge for your arms, but many (many, many) of these patterns do not immediately lend themselves to the funk-ability of some of the Stone patterns (which tend to at least have groupings that, again, reflect rudiments or make you work one limb preferentially in a “usable” way). They are here mostly for completeness and, for when you want to confuse your limbs, picking a page or more at random and seeing how the patterns feel. As independence exercises teach us very early on, our brains are wired for preferential patterning (you hit the same foot as you would hand, you’re non-dominant hand sucks, your hi-hat foot is born useless, and other revelations). This document is simply another PDF you can lose on your machine somewhere or have in that hidden work folder that comes out and gets an intense few looks as you try to split your left and right hands apart more.

And, it should be obvious, the same applies for your feet.

Having fought through enough of the combinations, I began to notice something I’m sure all of us have encountered as we approach that hypnotic state of cruising through a pattern we “get.” Some patterns feel really good to play, but only after you’ve internalized them enough to “play something else,” like feeling an odd clave or taking the patterns with many doubles of one hand and ripping them into a bounce-driven frenzy (or, invariably, playing one pattern we love to play to find out it’s a pattern you heard and memorized in a more musical context on record). The one benefit I’ve found from having this PDF around is that I have all of the patterns in one place, which makes me think harder about the different ways to play the patterns (although that is only a fringe benefit). If you treat them like a journey and not just the first 5 minutes of your warm-up routine, I suspect you could spend your life on any one page and still not cover all of the bases.

So, without further ado:

2011july13_stone_boulder_sticking_series_V1.pdf

2011july13_stone_boulder_jazz_grouping_series_V1.pdf

BclConverter-1.7.1 Installation In Ubuntu 10.04 LTS (And Related)

Tuesday, December 7th, 2010

What follows is the procedure for successfully building and running BclConverter-1.7.1 under Ubuntu (specifically 10.04, but this will likely be generic for other versions) using only apt-get to install missing programs and libraries, thereby trying to keep the install process as build-friendly as possible to the general (non-coding) user.

So, What’s BCL And Why Does It Need Converting?

The newest version of the Illumina sequencing software no longer uses the QSEQ format during the sequencing run, relying now on BCL files. This 12 January 2010 post snip from www.politigenomics.com covers the intro nicely.

Gone are the QSEQ files, they are replaced by BCL files which are binary, per image, per cycle files that contain the base call and quality information. Because they are per image, per cycle files, they can be transferred cycle by cycle as they are generated (as opposed to QSEQ files which are read based). The BCL files are also more compact, requiring only 1 byte/base (B/b) as compared to QSEQ files which require about 2.5 B/b. In addition, the intensity files are also not transferred by default, so RTA output goes from 10 B/b to just 1 B/b. Thus, even though you are generating five times more sequence data than a GA, your RTA directory will actually be smaller (about 250 GB).

That is all well and good, most of the open source programs of relevance (to me, anyway) require FASTQ format as input. As there is no one-stop conversion from BCL to FASTQ from the illumina downloads, the generation of QSEQ files is still a necessary (although not significantly difficult) evil. QSEQ files are generate-able from the BCL data (with maintenance of the illumina directory structure!) with the BclConverter code.

Unfortunately, the conversion from BCL format to QSEQ format (which is a file format for which many scripts exist online for conversion into the ever-familiar FASTQ format) requires an additional installation on your network machine, this installation being the BclConverter (v1.7.1) program available from the Illumina iCom website (registration required). This BclConverter program is not a pre-compiled binary, .rpm, .deb, .etc package, meaning the build is done by you from scratch. For many Linux distributions, this is non-problematic, as the build uses fairly standard tools. If you’re running Ubuntu, you will find yourself compiling (and running) with a host of show-stopping (or eye candy-stopping) errors. What lies below takes care of these errors.

Quick Summary

If you walk through the following steps, you’ll have no issue installing BclConverter. The more exhaustive discussion is below.

> sudo aptitude update

> sudo aptitude upgrade

> sudo apt-get install build-essential mercurial cmake python2.6-dev python3.1-dev gettext
libopenal1 libopenexr-dev libavdevice52 freeglut3-dev libglew1.5-dev libxmu-dev libxi-dev
libfreeimage-dev doxygen libqt4-dev bison flex libbz2-dev libpng12-dev libxml-simple-perl
ia32-libs lib32asound2 lib32ncurses5 lib32nss-mdns lib32z1 lib32gfortran3 gcc-4.3-multilib
gcc-multilib lib32gomp1 libc6-dev-i386 lib32mudflap0 lib32gcc1 lib32gcc1-dbg lib32stdc++6
lib32stdc++6-4.3-dbg libc6-i386 csh g++ g++-4.3 libstdc++6-4.3-dev g++-multilib
g++-4.3-multilib gcc-4.3-doc libstdc++6-4.3-dbg libstdc++6-4.3-doc nfs-common
nfs-kernel-server portmap ssh gnuplot

> sudo tar xvf BclConverter-1.7.1.tar.gz

> cd BclConverter-1.7.1

> sudo make install

Installation – The Long And Sometimes Error-Filled Version

There’s a lot of error message duplication and step-wise discussion below because I assume that you found this page by searching against errors as they came up in the build process.

NOTE: The first installation attempt failed with the following packages installed additionally during the initial setup of the machine:

sudo apt-get install ia32-libs lib32asound2 lib32ncurses5 lib32nss-mdns lib32z1 lib32gfortran3
gcc-4.3-multilib gcc-multilib lib32gomp1 libc6-dev-i386 lib32mudflap0 lib32gcc1 lib32gcc1-dbg
lib32stdc++6 lib32stdc++6-4.3-dbg libc6-i386 csh g++ g++-4.3 libstdc++6-4.3-dev g++-multilib
g++-4.3-multilib gcc-4.3-doc libstdc++6-4.3-dbg libstdc++6-4.3-doc nfs-common nfs-kernel-server
portmap ssh

You may or may not need some of these (especially if you’re running a 32-bit version of Ubuntu), but I can’t say definitively that something above is NOT ALSO required beyond the apt-get list provided below, so just install them anyway (the NFS stuff may be overkill, but if you’re going to mount this machine for sequencer file transfer, you’ll need this and/or SAMBA anyway).

My first build attempt of BclConverter with a mostly fresh Ubuntu install provided the following error:

...failed updating 2 targets...
...skipped 3 targets...
...updated 7846 targets...
boost.sh: build failed: Terminating...
CMake Error at c++/CMakeLists.txt:177 (message):
  Failed to build Boost


-- Configuring incomplete, errors occurred!
make: *** [build/Makefile] Error 1

So, we know that the Boost 1.42 libraries are not installed. Part of the BclConverter build process involves building a copy of these libraries (what failed above). The problem above was not a missing Boost as much as it was missing build tools for the whole program.

If the problem is Boost 1.42, why not just install the Ubuntu package? I’m not entirely sure, but there may or may not be something about the BclConverter build that requires something in Boost 1.42 to be findable by the BclConverter in its local directory (not too likely, but I didn’t diagnose it). Also, the problem may be version-specific (more likely than not), as the Boost build one can apt-get is 2.0-m12-2. Which is to say, installing the Ubuntu package…

sudo apt-get install boost-build

…did not solve the Boost problem. The possible solutions are to (1) build Boost 1.42 yourself or (2) simply let the BclConverter build take care of this (since the Boost 1.42 library is included in the BclConverter package for building). A Boost 1.42 build attempt external to the BclConverter program did not, in fact, solve the Boost problem in a subsequent BclConverter build attempt (I spare you repeat of the same error), making the successful apt-get-based approach all the easier.

We begin by updating our aptitude database and upgrading your machine (this is a skip-able step, but I prefer keeping everything up-to-date).

sudo aptitude update
sudo aptitude upgrade

The required build programs and libraries for BclConverter-1.7.1 (that are not part of my standard lib32 et al. install-ables listed above) are install-able as below:

sudo apt-get install build-essential mercurial cmake python2.6-dev python3.1-dev gettext
libopenal1 libopenexr-dev libavdevice52 freeglut3-dev libglew1.5-dev libxmu-dev libxi-dev
libfreeimage-dev doxygen libqt4-dev bison flex libbz2-dev libpng12-dev libxml-simple-perl
gnuplot

My routine setup preference is to place installed programs into /opt (purely for organizational purposes. It really doesn’t matter where within reason). With BclConverter-1.7.1 downloaded from iCom, we’ll move the .gz/.zip file to /opt, extract, untar, and install. With a Terminal window open and cd’ed to the BclConverter-1.7.1 download location (likely ~/Downloads, maybe ~/Desktop):

sudo mv BclConverter-1.7.1.tar.gz /opt
cd /opt
sudo tar xvf BclConverter-1.7.1.tar.gz
cd BclConverter-1.7.1
sudo make install

If, for any reason, you wish to see what the install log looks like, you can download mine for this session (in the 2010dec7__bclconverter_1_7_1_logs.zip file, see 2010dec7__bclconverter_1_7_1_build3b__successful__BUILD).

The last piece of the puzzle is to add the /opt/BclConverter/bin directory to your path, which we do in .profile as follows:

cd ~/
pico .profile

In .profile, add the following to the bottom somewhere…

PATH="/opt/BclConverter-1.7.1/bin/:$PATH"

Save and exit.

source .profile

Potential Errors Along The Way

This section is the most important part as it’s likely how you found this post. Below are the few problems (and messages) that might arise that are solved by the installation of specific packages).

1. Boost Error And Attempted sudo apt-get install boost-build

The error with and without a boost-build install is the same.

...failed updating 2 targets...
...skipped 3 targets...
...updated 7846 targets...
boost.sh: build failed: Terminating...
CMake Error at c++/CMakeLists.txt:177 (message):
  Failed to build Boost


-- Configuring incomplete, errors occurred!
make: *** [build/Makefile] Error 1

The full list from the build attempts for both cases can be viewed in (in 2010dec7__bclconverter_1_7_1_logs.zip:

* 2010dec7__bclconverter_1_7_1_build1__boosterror__FAILED.txt – initial error
* 2010dec7__bclconverter_1_7_1_build2__aptgetboost__FAILED.txt – after boost-build install

Running the full apt-get (see the contents of 2010dec7__bclconverter_1_7_1_logs.zip, with the results in 2010dec7__bclconverter_1_7_1_build3a__aptgetlist__RESULTS) produces a successful BclConverter build. The log for my build is available in 2010dec7__bclconverter_1_7_1_build3b__successful__BUILD.txt.

2. XML:Simple-Related Error And libxml-simple-perl

Without either gnuplot or the libxml-simple-perl installation, a setupBclToQseq.py run will successfully generate QSEQ files. The additional tools provide you with some statistical and visual analyses of your results (so are definitely worth installing).

If you don’t have libxml-simple-perl installed, you’ll see the following error after running the BaseCalls “make”:

/opt/BclConverter-1.7.1/bin/plotIntensity_tiles.pl tiles.txt s_8 SignalMeans _all.txt 
_all.png && echo `date` > s_8_all_pngs.txt
Can't locate XML/Simple.pm in @INC (@INC contains: /opt/BclConverter-1.7.1/lib/perl /etc/perl
/usr/local/lib/perl/5.10.1 /usr/local/share/perl/5.10.1 /usr/lib/perl5 /usr/share/perl5
/usr/lib/perl/5.10 /usr/share/perl/5.10 /usr/local/lib/site_perl .) at 
/opt/BclConverter-1.7.1/lib/perl/Gerald/Common.pm line 128.
BEGIN failed--compilation aborted at /opt/BclConverter-1.7.1/lib/perl/Gerald/Common.pm
line 128.
Compilation failed in require at /opt/BclConverter-1.7.1/bin/plotIntensity_tiles.pl line 22.
...
make: *** [s_1_all_pngs.txt] Error 2
make: *** [s_2_all_pngs.txt] Error 2
make: *** [s_3_all_pngs.txt] Error 2
make: *** [s_4_all_pngs.txt] Error 2
make: *** [s_5_all_pngs.txt] Error 2
make: *** [s_6_all_pngs.txt] Error 2
make: *** [s_7_all_pngs.txt] Error 2
make: *** [s_8_all_pngs.txt] Error 2

The fix is trivial (if you’re doing it incrementally and don’t already have XML:simple installed):

sudo apt-get install libxml-simple-perl

3. gnuplot Errors And Fix

If you don’t have gnuplot already installed (and why wouldn’t you?), you’ll receive the following error during the BCL-to-QSEQ “make” process:

sh: gnuplot: not found
/opt/BclConverter-1.7.1/share/makefiles/bclToQseq/FlowCellTargets.mk:76: [IVC.htm
(s_1_all_pngs.txt s_2_all_pngs.txt s_3_all_pngs.txt s_4_all_pngs.txt s_5_all_pngs.txt
s_6_all_pngs.txt s_7_all_pngs.txt s_8_all_pngs.txt plotIntensity_for_IVC_finished.txt)
(s_1_all_pngs.txt s_2_all_pngs.txt s_3_all_pngs.txt s_4_all_pngs.txt s_5_all_pngs.txt
s_6_all_pngs.txt s_7_all_pngs.txt s_8_all_pngs.txt plotIntensity_for_IVC_finished.txt)]

/opt/BclConverter-1.7.1/bin/create_IVC_thumbnail.pl . > IVC.htm.tmp && mv IVC.htm.tmp
IVC.htm
/opt/BclConverter-1.7.1/share/makefiles/bclToQseq/FlowCellTargets.mk:82: [All.htm
(s_1_all_pngs.txt s_2_all_pngs.txt s_3_all_pngs.txt s_4_all_pngs.txt s_5_all_pngs.txt
s_6_all_pngs.txt s_7_all_pngs.txt s_8_all_pngs.txt plotIntensity_for_IVC_finished.txt)
(s_1_all_pngs.txt s_2_all_pngs.txt s_3_all_pngs.txt s_4_all_pngs.txt s_5_all_pngs.txt
s_6_all_pngs.txt s_7_all_pngs.txt s_8_all_pngs.txt plotIntensity_for_IVC_finished.txt)]

/opt/BclConverter-1.7.1/bin/create_tile_thumbnails.pl all > FullAll.htm && \
	/opt/BclConverter-1.7.1/bin/create_tile_thumbnails.pl all --maxTiles=20 --link='
_a href="FullAll.htm"_Full output (Warning: may overload your browser!)_/a_' > All.htm.tmp 
&& mv All.htm.tmp All.htm

/opt/BclConverter-1.7.1/share/makefiles/bclToQseq/FlowCellTargets.mk:109:
[BustardSummary.xml (IVC.htm All.htm tiles.txt) (IVC.htm All.htm tiles.txt)]
/opt/BclConverter-1.7.1/bin/produceIntensityStats.pl .
Unable to find file /LOCATION_OF_INTENSITIES_FOLDER/Intensities/BaseCalls/../../
../samples.xml at /opt/BclConverter-1.7.1/lib/perl/Gerald/Jerboa.pm line 387.

...

/opt/BclConverter-1.7.1/share/makefiles/bclToQseq/FlowCellTargets.mk:58: [finished.txt
(Matrix Phasing s_1 s_2 s_3 s_4 s_5 s_6 s_7 s_8 BustardSummary.xml BustardSummary.xsl
IVC.htm All.htm) (Matrix Phasing s_1 s_2 s_3 s_4 s_5 s_6 s_7 s_8 BustardSummary.xml
BustardSummary.xsl IVC.htm All.htm)]
touch finished.txt.tmp && mv finished.txt.tmp finished.txt

With a:

sudo apt-get install gnuplot

All remaining errors in the BCL-to-QSEQ “make” process should disposed of, leaving you with a Plots directory containing multiple .png files after the QSEQ generation process.

4. Just Running “make” For BCL-to-QSEQ

The successful setupBclToQseq.py run:

setupBclToQseq.py -i /LOCATION_OF_FILES/Intensities/BaseCalls -p /LOCATION_OF_FILES/Intensities
 -o /LOCATION_OF_FILES/Intensities/BaseCalls --in-place --overwrite

ends with (also in setupBclToQseq.log):

setupBclToQseq.py version 1.7.1

Configuring /opt/BclConverter-1.7.1/share/makefiles/bclToQseq/Makefile to 
/LOCATION_OF_FILES/Intensities/BaseCalls/Makefile

Creating the 'Makefile.config'

Output directory succesfully initialized. Type 'make' in 
/LOCATION_OF_FILES/Intensities/BaseCalls to start the conversion

And if you simply type “make,” you get the following error:

/opt/BclConverter-1.7.1/bin/plotIntensity_tiles.pl tiles.txt s_1 SignalMeans _all.txt
_all.png && echo `date` > s_1_all_pngs.txt
Can't locate XML/Simple.pm in @INC (@INC contains: /opt/BclConverter-1.7.1/lib/perl /etc/perl 
/usr/local/lib/perl/5.10.1 /usr/local/share/perl/5.10.1 /usr/lib/perl5 /usr/share/perl5 
/usr/lib/perl/5.10 /usr/share/perl/5.10 /usr/local/lib/site_perl .) at 
/opt/BclConverter-1.7.1/lib/perl/Gerald/Common.pm line 128.
BEGIN failed--compilation aborted at /opt/BclConverter-1.7.1/lib/perl/Gerald/Common.pm 
line 128.
Compilation failed in require at /opt/BclConverter-1.7.1/bin/plotIntensity_tiles.pl line 22.
BEGIN failed--compilation aborted at /opt/BclConverter-1.7.1/bin/plotIntensity_tiles.pl
line 22.
make: *** [s_1_all_pngs.txt] Error 2

The complete log is available in 2010dec7__bclconverter_1_7_1_build3c__make_error_wo_j8.txt in
2010dec7__bclconverter_1_7_1_logs.zip.

A brief User Guide read will hip you to a proper run command in the BaseCalls directory (obvious, read the User Guide):

make -j 8

For a simple test (and I assume that your network directory structure for the Illumina is something like /LOCATION/TO/NETWORK/DATA/Data/Intensities and /LOCATION/TO/NETWORK/DATA/Data/Intensities/Basecalls (which it should be), we’ll use the example in the BCLConverter User Guide (and be sureto download the .PDF).

This will produce a sizable logfile. You can check out a successful run in 2010dec7__bclconverter_1_7_1_build3i__with_gnuplot.txt in 2010dec7__bclconverter_1_7_1_logs.zip.

5. QSEQ-to-FASTQ Script

Not really an error, just a last little help to convert your QSEQ files into generic FASTQ format.

#!/bin/bash
for ((x=1;x< =8;x+=1)); do 
cat s_"$x"_1_*_qseq.txt | awk -F '\t' '{gsub(/\./,"N", $9); if ($11 > 0) printf("@%s_%04d:
%s:%s:%s:%s#%s/%s\n%s\n+%s_%04d:%s:%s:%s:%s#%s/%s\n%
\n",$1,$2,$3,$4,$5,$6,$7,$8,$9,$1,$2,$3,$4,$5,$6,$7,$8,$10)}' > s_
"$x"_sequence.fastq; 
done

NOTE: the “cat” contents has to be all on one line! Copy this script into a text editor and reformat (or download a copy – 2010dec7__qseq_to_fastq.script).

Internet Connection Sharing – Ubuntu 10.04 NAT Gateway Setup (Abridged Version)

Sunday, August 29th, 2010

Had I known it would be this straightforward, I would have done it much sooner.

As is the case in many posts here, what follows is a summary of what I had to do to do what it is I’m writing about, kept largely for reference purposes but around just in case someone wants to do the same and would rather not take several web pages and combine them into one procedure that they may still have the stumble through to make work.

This page would not have been possible without the procedure outlined at ubuntuforums.org/showthread.php?t=713874.

My current cluster configuration (say, as the one shown above) contains one system board dedicated to web access (with a single available IP address from my provider) and general file transfer hosting, then several boards dedicated to computation. As an Ubuntu user, every fresh install of some new program may require the addition of libraries or subordinate programs not available in the “standard” Ubuntu installation.

With a single IP address to the room, the procedure for performing a proper install of a library or program involves either running sudo apt-get install program/library on the web-accessible machine and saving all of the .deb files from a first machine, transferring those .deb files to all other machines, and installing, or, still more involved, “walking” the ethernet cable down the line of ports on each system board, reassigning IP addresses in /etc/network/interfaces, and performing installations. This long involved set of different processes are then also required for system updates. Not only is this tedious, it requires taking machines “offline” from your cluster one-at-a-time (that is, if you don’t have second ethernet cards on each system board to keep each ready for walking the ethernet cable down the line). It’s the “other” IPv4 exhaustion.

The easiest solution that keeps your hardware from otherwise ever being touched (and keeps you from having to sit in front of it) is to set up a NAT Gateway, an approach that uses one host machine to share the outside world connection to the other machines. The whole setup procedure consists of a simple hardware setup, some configuration of the machine that has the hard-coded (or not) IP address, and modifications to the /etc/network/interfaces file of all associated machines to direct their search for the outside. After, all machines attached to the switch also connected to the Host Machine can use the “world” IP address of the Host Machine to update OS, transfer files, host pages (with some port work, which I won’t go into), etc.

The steps are as below that are all I have to follow to get this working in Ubuntu 10.04 (and should be very, very similar in all cases).

Hardware Setup

The image above tells it all. This procedure uses 1 machine with 2 ethernet cards (which likely means one on the system board and a used PCI slot), 1 switch, and X number of other machines.

Software Setup On Host Machine

Starting from a fresh installation on the Host Machine (although not necessary. This first pass was set up on a machine already the recipient of several updates and software installations) which properly recognizes the two ethernet cards, the procedure is as follows.

1. sudo pico /etc/network/interfaces

First we change the interfaces file to reflect that the two ethernet cards are two be divided into “public” (the internet. Here, eth0) and “private” (the intranet. Here, eth1). The contents of my file are as below:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
address mm.nn.oo.pp
netmask qq.rr.ss.tt
gateway uu.ww.xx.yy

auto eth1
iface eth1 inet static
address aa.bb.cc.dd
netmask 255.0.0.0
network aa.bb.cc.0
broadcast aa.bb.cc.255

Several things. eth0 connects to the outside world. mm.nn.oo.pp is the IP address provided by your provider. The values for netmask and gateway (and network and broadcast if you have them) should be available from your provider (they may not be obvious).

eth1 is being used for connection to the intranet. Letter pairs can be anything (but keep all aa‘s the same, bb‘s the same, etc.).

2. sudo pico /etc/resolv.conf

nameserver ee.ff.gg.hh
nameserver ee.ff.gg.ii

nameserver information should be available from your provider. Add this info to resolv.conf.

3. sudo pico /etc/sysctl.conf

From within this long file…

#
# /etc/sysctl.conf - Configuration file for setting system variables
# See /etc/sysctl.d/ for additional system variables.
# See sysctl.conf (5) for information.
#

#kernel.domainname = example.com

# Uncomment the following to stop low-level messages on console
#kernel.printk = 4 4 1 7

##############################################################3
# Functions previously found in netbase
#

# Uncomment the next two lines to enable Spoof protection (reverse-path filter)
# Turn on Source Address Verification in all interfaces to
# prevent some spoofing attacks
#net.ipv4.conf.default.rp_filter=1
#net.ipv4.conf.all.rp_filter=1

# Uncomment the next line to enable TCP/IP SYN cookies
#net.ipv4.tcp_syncookies=1

# Uncomment the next line to enable packet forwarding for IPv4
# net.ipv4.ip_forward=1

# Uncomment the next line to enable packet forwarding for IPv6
#net.ipv6.conf.all.forwarding=1

###################################################################
# Additional settings - these settings can improve the network
# security of the host and prevent against some network attacks
# including spoofing attacks and man in the middle attacks through
# redirection. Some network environments, however, require that these
# settings are disabled so review and enable them as needed.
#
# Ignore ICMP broadcasts
#net.ipv4.icmp_echo_ignore_broadcasts = 1
#
# Ignore bogus ICMP errors
#net.ipv4.icmp_ignore_bogus_error_responses = 1
# 
# Do not accept ICMP redirects (prevent MITM attacks)
#net.ipv4.conf.all.accept_redirects = 0
#net.ipv6.conf.all.accept_redirects = 0
# _or_
# Accept ICMP redirects only for gateways listed in our default
# gateway list (enabled by default)
# net.ipv4.conf.all.secure_redirects = 1
#
# Do not send ICMP redirects (we are not a router)
#net.ipv4.conf.all.send_redirects = 0
#
# Do not accept IP source route packets (we are not a router)
#net.ipv4.conf.all.accept_source_route = 0
#net.ipv6.conf.all.accept_source_route = 0
#
# Log Martian Packets
#net.ipv4.conf.all.log_martians = 1

Uncomment the line that reads:

# net.ipv4.ip_forward=1

so it reads:

net.ipv4.ip_forward=1

And save. This change will be made upon restart but can be started up on-the-fly by typing the following:

     3a. sudo sysctl -w net.ipv4.ip_forward=1

4. sudo pico /etc/rc.local

#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

exit 0

To the rc.local file add the following two lines at the bottom (above the exit 0):

/sbin/iptables -P FORWARD ACCEPT
/sbin/iptables --table nat -A POSTROUTING -o eth0 -j MASQUERADE

And save. This change will be made upon restart but can be started up on-the-fly by typing the following:

     4a. sudo iptables -P FORWARD ACCEPT

     4b. sudo iptables –table nat -A POSTROUTING -o eth0 -j MASQUERADE

5. sudo apt-get install dhcp3-server

This is the online machine after all. As one of the ubuntuforum pages (ubuntuforums.org/showthread.php?t=713874) I worked from stated: “The start will fail, but that is nothing to worry about.”

6. sudo pico /etc/dhcp3/dhcpd.conf

Now we configure the dhcp server on this Host Machine so it knows how to assign addresses.

The dhcpd.conf example at ubuntuforums.org/showthread.php?t=713874 worked good enough.

ddns-update-style none;
option domain-name "mynetwork";
option domain-name-servers ee.ff.gg.hh, ee.ff.gg.ii;
option routers nn.mm.oo.pp;

default-lease-time 42300;
max-lease-time 84600;
authoritative;

log-facility local7;

subnet aa.bb.cc.0 netmask 255.0.0.0 {
  range aa.bb.cc.AA aa.bb.cc.ZZ;
}

Note: letter pairs here need to match up with the interfaces file. The range is for the Client Machines looking for IP addresses. My strong preference is to hard-code IP addresses for these Client Machines, so just have the range account for the number of machines from AA to ZZ.

7. sudo pico /etc/default/dhcp3-server

Finally, one last change on this Host Machine, to configure the dhcp server interface (for this cluster, eth1).

# Defaults for dhcp initscript
# sourced by /etc/init.d/dhcp
# installed at /etc/default/dhcp3-server by the maintainer scripts

#
# This is a POSIX shell fragment
#

# On what interfaces should the DHCP server (dhcpd) serve DHCP requests?
#	Separate multiple interfaces with spaces, e.g. "eth0 eth1".

Simply add the following to the bottom of this file

INTERFACES="eth1"

My strong bias at this point is to restart this machine and confirm that the network works (that you can open web pages and ping accordingly), but you can restart the dhcp server by typing:

     7a. sudo /etc/init.d/dhcp3-server start

Client Machines

This could not be easier.

1. sudo pico /etc/network/interfaces

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
 address aa.bb.cc.AA (up to ZZ)
 netmask 255.0.0.0
 network aa.bb.cc.0
 broadcast aa.bb.cc.255
 gateway aa.bb.cc.dd

Note! The gateway should be the IP address of the Host Machine eth1 card (aa.bb.cc.dd). That is all of the direction that is needed.

2. sudo pico /etc/resolv.conf

nameserver ee.ff.gg.hh
nameserver ee.ff.gg.ii

3. sudo /etc/init.d/networking restart

Either reboot the machine or use the above to restart the network. Following the above, the Client Machine can see the outside world, run system updates (that is, sudo aptitude update and sudo aptitude upgrade), etc.

Obligatory

  • CNYO

  • Sol. Sys. Amb.

  • Ubuntu 4 Nano

  • NMT Review

  • N-Fact. Collab.

  • Pres. Asn. CNY

  • T R P Nanosys

  • Nano Gallery

  • nano gallery
  • Aerial Photos

    More @ flickr.com

    Syracuse Scenes

    More @ flickr.com