home

Archive for the 'scripts' Category

Led Astray By (A) Photon – WordPress, Jetpack, and The Perils Of Embedded Clear Sky Charts (And Other)

Friday, May 1st, 2015

A re-post from the CNY Observers website (www.cnyo.org).

Greetings fellow astrophiles,

CNYO has been anticipating our first observing session at Beaver Lake for this year, with the first of our two Spring dates (April 23rd) already clouded/snowed out. The forecast for April 30th hadn’t looked too much better based on Monday estimates, leaving us to wonder if attendees would be stuck indoors with a lecture instead of outdoors with the rest of the universe.

I woke up early on the 30th to blue skies and a very bright Sun, certainly already exceeding the expectations of the past few days. But what of the afternoon and evening?

As I am prone to do on the day of an observing session, I headed right for the CNYO Cheat Sheet, where one can find the sky conditions for a large part of Central New York in the form of several Clear Sky Charts (CSCs – and, based on the different cloud cover at different locations, even begin to piece together how the skies at your location may change). The morning’s CSCs are shown in the image below.

2015april30_photon_before

You will note that the bars to the far left (representing the morning) are not the dark blue squares that would indicate an almost cloud-less sky. As the red text at the bottom notes, sometimes the CSC images from a previous session are still sitting in your browser’s cache and, to make sure you’re looking at the newest data, you should hit Page Reload. Well, 5 or 10 of those didn’t change matters at all. I clicked on the Downtown Syracuse image in order to see what the actual CSC website said about today. An almost perfect band of dark blue – prime observing weather (when the wind is mild, that is).

So, what happened?

The first clue came when I right-clicked on one of the images in order to see just the image in my browser. When you do this, you should see something like: cleardarksky.com/c/SyrcsNYcs0.gif?1

What I saw for the link was the following: i1.wp.com/cleardarksky.com/c/SyrcsNYcs0.gif?1

Something is afoot in Boötes.

A quick google search indicated that the i1.wp.com (which might also be i0.wp.com, i2.wp.com, maybe more) site is, in fact, an image (maybe other) repository for wordpress.com that is supposed to speed up your page downloading process (by being faster than the same image you might load somewhere else) and is called upon, specifically, by Photon – one of the functions built into Jetpack (itself a large suite of plugins for WordPress that very generally make my life much easier by providing Site Stats, Contact Forms, etc.). That said, this is no good for the Clear Sky Chart, as you don’t know how many days ago that i1.wp.com image was saved (and it clearly ain’t today’s!).

To disable this feature (if it was turned on, anyway), go to your WordPress Dashboard and click on Jetpack on the right-hand side.

2015april30_photon_jetback

At present, Photon is the first clickable item at upper left. Click on “Photon” to reveal the following image:

2015april30_photon_deactivate

Click on Deactivate and go back to your Clear Sky Chart-containing page:

2015april30_photon_after

You’ll note that the Clear Sky Charts are fixed (revealing an excellent day for Solar and Night Observing) and you’ll also see that the NASA/SOHO image is different, the SWPC/NOAA image is different, and event the Wunderground logo is different. Quite the site fix!

If you have the same problem, I hope the above fixes it. If you know of a site running the Clear Sky Chart and it doesn’t reflect what you see outside, let the site admin know.

Stupid-Simple (*nix-Specific) Sed Scripts To Get (All Current) Gaussian09 Output Files Working With aClimax

Monday, September 1st, 2014

The following three snippets of Gaussian output are for an optimization and normal mode analysis of simple olde methane (CH4).

...
 ******************************************
 Gaussian 03:  EM64L-G03RevE.01 11-Sep-2007
                31-Aug-2014 
 ******************************************
...
 incident light, reduced masses (AMU), force constants (mDyne/A),
 and normal coordinates:
                     1                      2                      3
                     T                      T                      T
 Frequencies --  1356.0070              1356.0070              1356.0070
 Red. masses --     1.1789                 1.1789                 1.1789
 Frc consts  --     1.2771                 1.2771                 1.2771
 IR Inten    --    14.1122                14.1122                14.1122
 Atom AN      X      Y      Z        X      Y      Z        X      Y      Z
   1   1     0.02  -0.42   0.43    -0.34  -0.13  -0.08    -0.36  -0.23  -0.23
   2   6     0.00   0.08  -0.09     0.00   0.09   0.08     0.12   0.00   0.00
...
 -------------------
 - Thermochemistry -
 -------------------
 Temperature   298.150 Kelvin.  Pressure   1.00000 Atm.
 Atom  1 has atomic number  1 and mass   1.00783
...
...
 ******************************************
 Gaussian 09:  EM64L-G09RevA.02 11-Jun-2009
                31-Aug-2014 
 ******************************************
...
 incident light, reduced masses (AMU), force constants (mDyne/A),
 and normal coordinates:
                     1                      2                      3
                     T                      T                      T
 Frequencies --  1356.0058              1356.0058              1356.0058
 Red. masses --     1.1789                 1.1789                 1.1789
 Frc consts  --     1.2771                 1.2771                 1.2771
 IR Inten    --    14.1123                14.1123                14.1123
  Atom  AN      X      Y      Z        X      Y      Z        X      Y      Z
     1   1    -0.03   0.42   0.43    -0.34  -0.14   0.07    -0.36  -0.23   0.23
     2   6     0.00  -0.08  -0.10     0.01   0.10  -0.08     0.12   0.00   0.00
...
-------------------
 - Thermochemistry -
 -------------------
 Temperature   298.150 Kelvin.  Pressure   1.00000 Atm.
 Atom     1 has atomic number  1 and mass   1.00783
...
...
 ******************************************
 Gaussian 09:  EM64L-G09RevD.01 24-Apr-2013
                31-Aug-2014 
 ******************************************
...
 incident light, reduced masses (AMU), force constants (mDyne/A),
 and normal coordinates:
                      1                      2                      3
                     ?A                     ?A                     ?A
 Frequencies --   1356.0132              1356.0132              1356.0132
 Red. masses --      1.1789                 1.1789                 1.1789
 Frc consts  --      1.2771                 1.2771                 1.2771
 IR Inten    --     14.1119                14.1119                14.1119
  Atom  AN      X      Y      Z        X      Y      Z        X      Y      Z
     1   1     0.02   0.42   0.43     0.34  -0.14   0.08    -0.36   0.23  -0.23
     2   6     0.00  -0.08  -0.09    -0.01   0.09  -0.08     0.12   0.00   0.00
...
 -------------------
 - Thermochemistry -
 -------------------
 Temperature   298.150 Kelvin.  Pressure   1.00000 Atm.
 Atom     1 has atomic number  1 and mass   1.00783
...

Two of these things are not like the other. The data’s nearly identical (and thank heavens. Unfortunately, Gaussian09 D.01 didn’t see the fully-optimized methane as belonging to the Td point group – despite all three versions being run with the same exact input file – but a rigorous re-symmetrization would have taken care of that), but there are some subtle formatting differences between all three versions (including differences between both Gaussian09 versions) that cause the venerable, all-encompassing aClimax program (developed by Timmy, the venerable, all-encompassing A. J. Ramirez-Cuesta) to throw out the following errors for all three cases when you use *.log files from a *nix (UNIX, Linux) machine.

Serious Error: A-CLIMAX has encountered an unhanded error. Please Save your data and contact support
aClimax: Quote Error Number 9
Error Loading File: Error reading data. Please check and try again.
aClimax: WARNING loaded file containing no frequencies

Problem number 1 is the existence of *nix newlines (carriage returns) in the *.log files coming off a *nix machine. Performing a conversion from *nix to DOS (for myself, using LineBreak in OSX, but tofrodos works just as well), the Gaussian03 file now opens just fine in aClimax:

File Loaded: Data Loaded Succesfully [sic].

This, unfortunately, does not improve the matter with the Gaussian09 files, which produce the following error:

Error: One of the numbers you have entered is of the wrong type.Please recheck and try again
Error Loading File: Error reading data. Please check and try again.

Given how little of the .log file aClimax actually needs to produce simulated inelastic neutron scattering (INS) spectra, I ran the methane normal mode analyses in three different Gaussian versions to determine what, in G09, was changed to make it just un-G03 enough to fail to load. With those changes figured out, I had a Perl script drafted up that would have converted everything back to the original G03 format. It was awesome. That said, after a small amount of testing to see where aClimax’s sensitivities lay, I discovered that very little of the .log file contents needed to be changed out, meaning that simple sed scripts would work just as well for those of us using our Windows boxes (or VirtualBox emulations) only for that “one stupid program” that keeps us having to log in (and, by that, I mean that we have sed already on our computers).

So, the problems between G09 and aClimax not related to carriage returns lie in two places.

1. The spacing of “Atom AN” – at the top of the eigenvector lists are the column labels, beginning with “Atom AN” – or something very close to “Atom AN” (the “|” in the boxes below mark the left edge of the output):

G03 E01 | Atom AN
G09 A02 |   Atom  AN
G09 D01 |  Atom  AN

Yes, the addition of a space or two results in a read error by aClimax. I would call this an… aggressive stringency in aClimax. That said, what did the original space in G03 versions not do that they do do in G09?

2. The spacing of “Atom N” – In the “Thermochemistry” section below the eigenvectors, atomic masses are listed as “Atom N” – or something very close to “Atom N” (again, the “|” in the boxes below mark the left edge of the output):

G03 E01 |  Atom  1
G09 A02 |    Atom     1
G09 D01 |   Atom     1

This change in spacing is also enough to cause aClimax to error out.

The Solution

A small sed script performs the necessary conversions on your *nix box (including OSX) for all .log files in a directory without issue:

#!/bin/sh

# This section converts all .log files to aClimax-friendly G03-ish format
find . -type f -name '*.log' -print | while read i
do
sed 's|  Atom  AN| Atom AN |g' $i > $i.aclimaxconversion_step1
sed 's| Atom   | Atom|g' $i.aclimaxconversion_step1 > $i.aClimaxable.log
rm $i.aclimaxconversion_step1
done

# This section converts all .out files to aClimax-friendly G03-ish format
find . -type f -name '*.out' -print | while read i
do
sed 's|  Atom  AN| Atom AN |g' $i > $i.aclimaxconversion_step1
sed 's| Atom   | Atom|g' $i.aclimaxconversion_step1 > $i.aClimaxable.out
rm $i.aclimaxconversion_step1
done

But Wait! Running G0* Jobs Under *nix? Convert To DOS Carriage Returns

The final problem halting your aClimax spectrum generation is the DOS carriage return (^M). For those running DOS-based Gaussian calculations (likely with a .out suffix), your conversion with the short script above (under *nix) likely (hopefully) worked just fine. For those running under *nix, you performed the conversion and still received the following aClimax error:

Serious Error: A-CLIMAX has encountered an unhanded error. Please Save your data and contact support
aClimax: Quote Error Number 9
Error Loading File: Error reading data. Please check and try again.
aClimax: WARNING loaded file containing no frequencies

The solution is an additional line in the sed script that will globally replace all *nix newlines with proper DOS carriage returns. The .out section remains the same.

#!/bin/sh

# This section converts all .log files to aClimax-friendly G03-ish format
find . -type f -name '*.log' -print | while read i
do
sed 's|  Atom  AN| Atom AN |g' $i > $i.aclimaxconversion_step1
sed 's| Atom   | Atom|g' $i.aclimaxconversion_step1 > $i.aclimaxconversion_step2
# This section converts your *nix newlines into DOS carriage returns
CR=`echo "\0015"`  # define the Carriage Return
sed -e "s/$/${CR}/g" $i.aclimaxconversion_step2 > $i.aClimaxable.log
done
# this cleans up your folder of temp files
rm *.aclimaxconversion_step1
rm *.aclimaxconversion_step2

# This section converts all .out files to aClimax-friendly G03-ish format
find . -type f -name '*.out' -print | while read i
do
sed 's|  Atom  AN| Atom AN |g' $i > $i.aclimaxconversion_step1
sed 's| Atom   | Atom|g' $i.aclimaxconversion_step1 > $i.aClimaxable.out
rm $i.aclimaxconversion_step1
done

Q. But what if I run the *nix-to-DOS version of the script on an already DOS-output file?

A1. The simple answer is that you’ll make your text file double-spaced (which is bad enough). aClimax will then provide the following error when you try to open it:

Error Reading File: Unexpected File End. File May be incorrect or corrupt.
Error Loading File: Error reading data. Please check and try again.

A2. I will assume that your problem is that you’re running the script in DOS to try to get your G09 to read more like G03. In this case (assuming you’re generating .out files), you’ll want to use a text editor to make the replacements described above (which is to say, that Perl script might makes it way to this page eventually. If you write a DOS .bat file or similar script for all OS’s, I’d be happy to link to it).

“From Kurdistan With Love” or Some Things To Do Before And/Or After Your WordPress Site Gets Hacked

Thursday, December 12th, 2013

“Hopefully, because he’s busy.” – Commissioner Gordon, The Dark Knight

On the plus side, www.somewhereville.com received its first update in just over 5 months. On the minus side, the new post was less than useful in many ways. I received a timely email from Dr. Obi Griffith of the Washington University in St. Louis Division of Oncology noting that my entire site was differently-down (thanks to the hijacking of my Sanger (And Illumina 1.3+ (And Solexa)) Phred Score (Q) ASCII Glyph Base Error Conversion Tables page that he linked to on a biostars site thread – so my thanks to Obi for catching something I likely would have gone weeks without noticing!).

The snapshot below shows the state of swv as of 3 December 2014. On the bright side (minus a friendly conspiracy to get someone else in trouble), I can say with some certainty that Serwan performed the content-ectomy (twitter: @S3RW4N, current email (although I suspect it won’t last long): serwan_007 – at cymbal – hotmail.com, on the Facebook, etc. All sites subject to change as people try to track him/her down post-attack (he/she’s been prolific if nothing else)).

2013dec11_serwan_hack

Exhibit A. Flag is waving in the actual version.

Several problems. To begin, it’s a gaudy hack, complete with rolling text and techno music. Second, the Television New Zealand (TVNZ) news service thought this hack to be significant enough to warrant actual coverage on their website when a similar file-swap on a WordPress (or WordPress-esque) site brought down the Health and Sports Fitness Club in Sandringham (syracuse.com didn’t give me the time of day). I commend this Kurdish hacker group for their ratings. Third, the manner in which files were replaced in the blog (specifically meaning the index.php file) blocked every other post on the site from being accessed, so every link anyone had posted to a page anywhere else on the Internets was made useless.

That said, I appreciate that Serwan generally performs fairly benign attacks on websites. File replacements were clearly identified from a simple date sorting, the important MySQL database content wasn’t touched, and Serwan even went as far as to set up a second Admin account so that I could quickly retake control of the site.

So, in light of the plight of the Kurdish people, I left the hacked version up for a few hours as I pondered what to do, which I discuss below.

My Spotty Procedure For Recovery:

What follows is a list of obvious and less obvious things to consider when recovering your WordPress blog from a hack. There are plenty of websites that show how to protect your site in the first place, then others that explain how to revive it (provided you do your own due diligence and back your site up regularly enough). What’s below is not complete, but you can rest assured that google is your friend in such matters, so keep your keywords targeted and see what comes up.

General Considerations:

1. Don’t use your blog. My last post at the time dated back to June 25th, during which time I’ve made several full backups (and kept WordPress up-to-date, the last time being 7 November 2013) of my entire site. In this respect, I was well set up to quickly recover from a hacking incident.

2. Keep a copy of your current running version of WordPress handy for file replacements. In my case, index.php was written over. All I had to do to recover was uncompress my WordPress  3.7.1 download, upload index.php to my server, and the site was back and running.

3. Have you backed up lately? This phrase has been in the .sig of my emails for many, many years. If your entire life is lived in the Googleverse (email, images, documents, etc.), then you’re fine until the Earth’s magnetic poles shift and wipe all the hard drives out (just kidding. I think). If you’re a computational scientist and have TBs of data, it’s up to you to make sure you have access to it all again. Same applies to WordPress. I’ve a biweekly alarm that tells me to back up several websites and I’ve an encrypted .txt file with all of the login info and steps needed to perform this backup. You should absolutely be doing the same if you’re not.

4. Set up an additional Administrator. In my case, my admin account was hacked to change the associated user email address to Serwan’s email. Obviously, attempting to log in, change the password, or what have you simply sent little pings of your futile attempts to the hacker. Having that second admin account will allow you to reroute your login efforts (and if they’re both hacked into, there’s still a way around. Will get to below).

5. Make a real password. At the risk of de-securing my sites by providing personal info, my typical password looks something like this:

d@!25fj014or&ydoSDfu

20 characters long, upper and lower, numbers, and non-alphanumeric characters. If you care about your site security, stay the hell away from the dictionary.

6. Dry-run your SHTF moment. Are you a survivalist? Can you identify edible berries by sight, build a lean-to, or stitch an open wound? Or are you the Marty Stouffer of the camping section at Target? If you’ve never had to work your way back from a complete disaster, you likely won’t know how to do it either quickly, efficiently, or securely.

Ergo, do another WordPress installation in a sub-folder of your main installation, create a new database, make your site pretty, perform a full backup of your database and uploaded media, then break it, either by deleting core files or corrupting your database (deleting a table would do the trick). If you can put the site back together again (the uploading of the database back onto your server likely being the worst part of the whole process), you’re likely in good shape for the real deal.

7. Harden WordPress. The good people at WordPress even tell you how to (although, admittedly, I thought I did all of this, so maybe there’s something being missed that will go into a future iteration of this page).

8. Get rid of “admin.” Several of the sites discussing WordPress hacks report that having this default account (or account default’ed) is a top-5 problem when trying to keep people out of your site. So get rid of it. Easily. Set up a new account, give it administrative privileges, then delete the admin account, which will ask you to attribute the current admin posts to another admin account.

9. Delete deactivated plugins if you’re not going to use them. Plugins are developed by people. People often have lives that keep them from timely updates of security exploits. If you’re using a plugin, that’s one thing. If a deactivated plugin languishes in your plugins folder, never gets updated, and some hacker writes something specifically to exploit a security flaw in that old, poorly maintained plugin, that’s all on you. So don’t risk your pocket knife being a projectile as you walk into the MRI room and get rid of the knife before it comes a problem.

10. I know nothing about it yet, but am giving Wordfence a whirl presently.

11. Hey, check your blog every once in a while to make sure it’s still you and not Serwan.

For The Specific Attack (From Easy To Harder):

1. FTP in and check file dates. The offending .php files (index.php and a hello.php containing the techno) were both dated 3 December 2013. Everything else was, at its newest, 7 November 2013 (from my last WordPress update). This made finding the hacked and previously not-present files easy. A cluster of important files with identically modification times and dates is an easy giveaway.

2. FTP in and check ALL the file dates. One never knows when something else is going to be placed into a themes folder, plugin folder, etc., to keep track of site access (that’s why I delete all deactivated plugins). So, sort by date and scour the whole site for modifications and new files.

3. If you make it into your site, go right to your User Settings, change the email address, then change your password.

4. Check out something like Sucuri SiteCheck. Hopefully, this search will complement your initial search as well as test against known threats. I ran a Sucuri on a similarly-hacked site (in this case, indoorstinkbugtrap.com) and received the following notification of defacement (so the check worked).

2013dec11_securi_results

securi.net results for fellow victim indoorstinkbugtrap.com.

5. If you can’t make it in the front door, crawl through the plumbing. You can change your admin account from within MySQL using, for instance, phpMyAdmin (check your hosting provider for details if this is new information to you). In the case of phpMyAdmin, you can modify the admin account in six easy steps.

1. Log in to phpMyAdmin

2. Click on the Structure Button in wp_users (red circle)

2013dec11_serwan_hack_mysql_1

3. Click on Browse (told you this was easy)

2013dec11_serwan_hack_mysql_2

4. Click the edit button for your administrative account (red circle)

2013dec11_serwan_hack_mysql_3

5. Change the email address back to your email and delete the current password.

2013dec11_serwan_hack_mysql_4

6. Save and go back to our WordPress site, then request a new password.

And, While We’re At It:

Serwan’s twitter image currently features a white hat (the Gandalf-ian sign of a good guy/gal hacker) and a long list of sites that have been defaced with otherwise useless, feral medadata promoting Kurdish Hackers for google to get confused by. A search for somewhereville.com in google left the following bad taste in its results page for a week after:

Hacked By Serwan. Allah Is Greatest. Long Live Kurdistan. Thanks To All Kurdish Hackers. Follow @S3RW4N FB.com/Mr.S995

If I may be so bold (and I’ve told Serwan the same), the Kurdish people had a long history of getting steamrolled by an oppressive regime that, regretfully, first-world countries didn’t put enough into stopping or acknowledging until the tanks rolled South into Kuwait. If you’re gong to label yourself an ethical hacker, fine. Mangle the front-end of someone’s WordPress site. That said, you could be educating others on the Kurdish people by including a few links into your hack. I live in America, where certain news services use “Muslim” and “Islam” in headlines purely for shock value when they want to appeal to an audience so narrow-minded that their hearing is susceptible to the Casimir Effect. I recommend adding the wikipedia article on Kurdistan and the Al-Anfal Campaign to future hacks (and I’m sure Serwan could find more) to provide a little substance to your efforts unless, of course, your goal is just to be a stupid-ass script-kiddie hacker.

If you’re gonna hack, at least try to be productive. Meantime, this was a valuable lesson for myself on what to do to try to keep WordPress from falling into the same limbo during a time when I might not have had an hour to fix it.

The “Stone Boulders” – All 12-Group (4,096) and 16-Group (65,536) L/R Sticking Combinations In PDF Format

Wednesday, July 13th, 2011

George L. Stone's Stick Control

You should have a copy regardless (amazon.com direct link).

[Drafted as an article for somewhere, stuck here instead…]

You could spend your life on the first six pages of “Stick Control” and still not cover all the possibilities. Dynamics, accents, foot-hand, foot-foot, fast/slow, hands on top of foot patterns, feet on top of hand patterns, regroupings and accenting in 5-7-4 (regrouping of the 16 strokes per pattern), 7-5-4 (re-regrouping of the 16 strokes), yadda yadda. If you see the first six pages of Stick Control as just exercises, you miss the fantastic complexity YOU can introduce to constantly humble yourself while hovering over a practice pad.

As I look at the sets of exercises, I see what I assume most of us do – paradiddles, singles, doubles, multiple-hits of the same stick, some oddball patterns you start playing as written and then mess up without knowing, etc. The question I found myself asking was “What drove Stone to use this particular sequence?” I eventually turned that question around and decided to answer the question “What did Stone leave out?” The PDFs linked to this article are what I’ve affectionately come to call the “Stone Boulder,” providing EVERY sticking combination Stone included and every other combination he didn’t. Some intro to how and why is below, followed by a bit of explanation. I think the patterns themselves are self-explanatory.

While not the most cite-able examples in all of genomics, there have always been passing references to drumming “being in someone’s DNA.” As it happens, drumming and biology did overlap in general approach during the mid-80’s-to-early-90’s (or so) in the great heyday of linear drumming (go dig out your Murray Houllif and Gary Chaffee books). The idea is simple: no two drums/cymbals hit at the same time, producing an often staccato and generally (well, to my ears anyway) more melodic sound from the drums (and much easier to transcribe than some of the superhuman overlapping rhythms people are having fun playing today). Ignoring the complexities of 3.5 billions years of evolution, DNA works the same way as these linear patterns to convey a message. The four bases in your DNA, A (adenine), C (cytosine), G (guanine), and T (thymine), act as a code that is read like those old drum beats were played – one at a time with no doubling-up please. The identical three million base-long DNA sequences in each of your cells (see CSI) could be turned from seemingly random patterns of [A,C,G,T] into seemingly random patterns of [L,R,B,H] (that’s left hand, right hand, bass drum foot, hi-hat foot), then some experimental linear drumming composer could “play” your genome. Better still, if the transcriber was as good as your cellular machinery, the entire performance could be written down and reconverted into [A,C,G,T] format exactly so you could be cloned and double-drum with someone who rushes and slows down just as much as you do.

While most people think of a drum programmer as someone who generates patterns on a computer, I took the route of programming to generate patterns to drum. I most certainly did NOT put the pattern pages together by hand (I promise, no mistakes). A small script in the Perl programming language used to generate DNA sequences did all of the dirty work (including making sure all patterns only appear ONCE in each document). The math for figuring out the total number of left/right patterns is quite simple. The number of combinations of unique sticking patterns for a particular pattern length is 2^n, n being the number of beats. For a single beat, that’s 2^1, or just 2, that single hit being performed with either the left or right hand. For a four-stroke pattern, that’s 2^4 (2 * 2 * 2 * 2), or 16 total patterns. These are shown below out of academic interest (although I hope you could write them down from memory).

Now, consider the first six pattern pages of the Stone book. 16 beats per pattern. That’s 2^16, or 65536 total patterns. At 20 patterns per page, the complete Stone book of these first six pages would take up 3,277 pages. At 2 seconds per pattern, you could rip through all 3,277 pages in about 36 hours 30 minutes (about the perceived length of a society gig).

For you fellow jazzers out there looking for a more swingin’ set, I’ve also included the same sticking deal with a triplet-feel set (12 beats instead of 16, so you’ve only got 4096 patterns to contend with, meaning you could play through the whole set in about 2 hours 30 minutes).

4,096 patterns are bad enough. 65,536 is borderline something uncouth. On the one hand, that’s a lot of patterns either way. On the other, for the obsessive compulsive readers, these are IT. There are no other 12- or 16-stroke sticking combinations that have a stick hitting on each beat (that is, no rests). As Terry Bozzio has said in one form or another in his many clinics introducing his ostinato independence exercises “once you’ve played through the 16, you’ve played every 16th note pattern there is.”

And it could be worse! If you wanted every combination of left, right and rest, that’s 3^16, or 43,046,721 patterns. At 2 seconds per pattern, that’s 23,915 hours, or about 2 years and 8 months. I pondered doing the same thing for all 16-note linear drum patterns (L,R,B,F), which would produce 4^16, or 4,294,967,296 patterns. That’s 2,386,093 hours, or 272 years and 3 months (that’s approaching four reincarnations of “no life”).

Each full page has three columns of 40 patterns (120 per page), producing a document that’s only 547 pages long (but entirely green-friendly in PDF format). You will note that most of the pages look like the same stupid thing. This is because the mechanism of generation for the sequences involved making single changes at a single position and walking down the entire 16-stroke sets until all changes had been accounted for. I become bored to tears staying on a single page and generally scroll at random and point the stick at the screen to pick a pattern to play. Be as methodical or all-over-the-map as you will.

Is there a good reason for doing this? Not particularly. There are lots of patterns here that are a mechanical challenge for your arms, but many (many, many) of these patterns do not immediately lend themselves to the funk-ability of some of the Stone patterns (which tend to at least have groupings that, again, reflect rudiments or make you work one limb preferentially in a “usable” way). They are here mostly for completeness and, for when you want to confuse your limbs, picking a page or more at random and seeing how the patterns feel. As independence exercises teach us very early on, our brains are wired for preferential patterning (you hit the same foot as you would hand, you’re non-dominant hand sucks, your hi-hat foot is born useless, and other revelations). This document is simply another PDF you can lose on your machine somewhere or have in that hidden work folder that comes out and gets an intense few looks as you try to split your left and right hands apart more.

And, it should be obvious, the same applies for your feet.

Having fought through enough of the combinations, I began to notice something I’m sure all of us have encountered as we approach that hypnotic state of cruising through a pattern we “get.” Some patterns feel really good to play, but only after you’ve internalized them enough to “play something else,” like feeling an odd clave or taking the patterns with many doubles of one hand and ripping them into a bounce-driven frenzy (or, invariably, playing one pattern we love to play to find out it’s a pattern you heard and memorized in a more musical context on record). The one benefit I’ve found from having this PDF around is that I have all of the patterns in one place, which makes me think harder about the different ways to play the patterns (although that is only a fringe benefit). If you treat them like a journey and not just the first 5 minutes of your warm-up routine, I suspect you could spend your life on any one page and still not cover all of the bases.

So, without further ado:

2011july13_stone_boulder_sticking_series_V1.pdf

2011july13_stone_boulder_jazz_grouping_series_V1.pdf

BclConverter-1.7.1 Installation In Ubuntu 10.04 LTS (And Related)

Tuesday, December 7th, 2010

What follows is the procedure for successfully building and running BclConverter-1.7.1 under Ubuntu (specifically 10.04, but this will likely be generic for other versions) using only apt-get to install missing programs and libraries, thereby trying to keep the install process as build-friendly as possible to the general (non-coding) user.

So, What’s BCL And Why Does It Need Converting?

The newest version of the Illumina sequencing software no longer uses the QSEQ format during the sequencing run, relying now on BCL files. This 12 January 2010 post snip from www.politigenomics.com covers the intro nicely.

Gone are the QSEQ files, they are replaced by BCL files which are binary, per image, per cycle files that contain the base call and quality information. Because they are per image, per cycle files, they can be transferred cycle by cycle as they are generated (as opposed to QSEQ files which are read based). The BCL files are also more compact, requiring only 1 byte/base (B/b) as compared to QSEQ files which require about 2.5 B/b. In addition, the intensity files are also not transferred by default, so RTA output goes from 10 B/b to just 1 B/b. Thus, even though you are generating five times more sequence data than a GA, your RTA directory will actually be smaller (about 250 GB).

That is all well and good, most of the open source programs of relevance (to me, anyway) require FASTQ format as input. As there is no one-stop conversion from BCL to FASTQ from the illumina downloads, the generation of QSEQ files is still a necessary (although not significantly difficult) evil. QSEQ files are generate-able from the BCL data (with maintenance of the illumina directory structure!) with the BclConverter code.

Unfortunately, the conversion from BCL format to QSEQ format (which is a file format for which many scripts exist online for conversion into the ever-familiar FASTQ format) requires an additional installation on your network machine, this installation being the BclConverter (v1.7.1) program available from the Illumina iCom website (registration required). This BclConverter program is not a pre-compiled binary, .rpm, .deb, .etc package, meaning the build is done by you from scratch. For many Linux distributions, this is non-problematic, as the build uses fairly standard tools. If you’re running Ubuntu, you will find yourself compiling (and running) with a host of show-stopping (or eye candy-stopping) errors. What lies below takes care of these errors.

Quick Summary

If you walk through the following steps, you’ll have no issue installing BclConverter. The more exhaustive discussion is below.

> sudo aptitude update

> sudo aptitude upgrade

> sudo apt-get install build-essential mercurial cmake python2.6-dev python3.1-dev gettext
libopenal1 libopenexr-dev libavdevice52 freeglut3-dev libglew1.5-dev libxmu-dev libxi-dev
libfreeimage-dev doxygen libqt4-dev bison flex libbz2-dev libpng12-dev libxml-simple-perl
ia32-libs lib32asound2 lib32ncurses5 lib32nss-mdns lib32z1 lib32gfortran3 gcc-4.3-multilib
gcc-multilib lib32gomp1 libc6-dev-i386 lib32mudflap0 lib32gcc1 lib32gcc1-dbg lib32stdc++6
lib32stdc++6-4.3-dbg libc6-i386 csh g++ g++-4.3 libstdc++6-4.3-dev g++-multilib
g++-4.3-multilib gcc-4.3-doc libstdc++6-4.3-dbg libstdc++6-4.3-doc nfs-common
nfs-kernel-server portmap ssh gnuplot

> sudo tar xvf BclConverter-1.7.1.tar.gz

> cd BclConverter-1.7.1

> sudo make install

Installation – The Long And Sometimes Error-Filled Version

There’s a lot of error message duplication and step-wise discussion below because I assume that you found this page by searching against errors as they came up in the build process.

NOTE: The first installation attempt failed with the following packages installed additionally during the initial setup of the machine:

sudo apt-get install ia32-libs lib32asound2 lib32ncurses5 lib32nss-mdns lib32z1 lib32gfortran3
gcc-4.3-multilib gcc-multilib lib32gomp1 libc6-dev-i386 lib32mudflap0 lib32gcc1 lib32gcc1-dbg
lib32stdc++6 lib32stdc++6-4.3-dbg libc6-i386 csh g++ g++-4.3 libstdc++6-4.3-dev g++-multilib
g++-4.3-multilib gcc-4.3-doc libstdc++6-4.3-dbg libstdc++6-4.3-doc nfs-common nfs-kernel-server
portmap ssh

You may or may not need some of these (especially if you’re running a 32-bit version of Ubuntu), but I can’t say definitively that something above is NOT ALSO required beyond the apt-get list provided below, so just install them anyway (the NFS stuff may be overkill, but if you’re going to mount this machine for sequencer file transfer, you’ll need this and/or SAMBA anyway).

My first build attempt of BclConverter with a mostly fresh Ubuntu install provided the following error:

...failed updating 2 targets...
...skipped 3 targets...
...updated 7846 targets...
boost.sh: build failed: Terminating...
CMake Error at c++/CMakeLists.txt:177 (message):
  Failed to build Boost


-- Configuring incomplete, errors occurred!
make: *** [build/Makefile] Error 1

So, we know that the Boost 1.42 libraries are not installed. Part of the BclConverter build process involves building a copy of these libraries (what failed above). The problem above was not a missing Boost as much as it was missing build tools for the whole program.

If the problem is Boost 1.42, why not just install the Ubuntu package? I’m not entirely sure, but there may or may not be something about the BclConverter build that requires something in Boost 1.42 to be findable by the BclConverter in its local directory (not too likely, but I didn’t diagnose it). Also, the problem may be version-specific (more likely than not), as the Boost build one can apt-get is 2.0-m12-2. Which is to say, installing the Ubuntu package…

sudo apt-get install boost-build

…did not solve the Boost problem. The possible solutions are to (1) build Boost 1.42 yourself or (2) simply let the BclConverter build take care of this (since the Boost 1.42 library is included in the BclConverter package for building). A Boost 1.42 build attempt external to the BclConverter program did not, in fact, solve the Boost problem in a subsequent BclConverter build attempt (I spare you repeat of the same error), making the successful apt-get-based approach all the easier.

We begin by updating our aptitude database and upgrading your machine (this is a skip-able step, but I prefer keeping everything up-to-date).

sudo aptitude update
sudo aptitude upgrade

The required build programs and libraries for BclConverter-1.7.1 (that are not part of my standard lib32 et al. install-ables listed above) are install-able as below:

sudo apt-get install build-essential mercurial cmake python2.6-dev python3.1-dev gettext
libopenal1 libopenexr-dev libavdevice52 freeglut3-dev libglew1.5-dev libxmu-dev libxi-dev
libfreeimage-dev doxygen libqt4-dev bison flex libbz2-dev libpng12-dev libxml-simple-perl
gnuplot

My routine setup preference is to place installed programs into /opt (purely for organizational purposes. It really doesn’t matter where within reason). With BclConverter-1.7.1 downloaded from iCom, we’ll move the .gz/.zip file to /opt, extract, untar, and install. With a Terminal window open and cd’ed to the BclConverter-1.7.1 download location (likely ~/Downloads, maybe ~/Desktop):

sudo mv BclConverter-1.7.1.tar.gz /opt
cd /opt
sudo tar xvf BclConverter-1.7.1.tar.gz
cd BclConverter-1.7.1
sudo make install

If, for any reason, you wish to see what the install log looks like, you can download mine for this session (in the 2010dec7__bclconverter_1_7_1_logs.zip file, see 2010dec7__bclconverter_1_7_1_build3b__successful__BUILD).

The last piece of the puzzle is to add the /opt/BclConverter/bin directory to your path, which we do in .profile as follows:

cd ~/
pico .profile

In .profile, add the following to the bottom somewhere…

PATH="/opt/BclConverter-1.7.1/bin/:$PATH"

Save and exit.

source .profile

Potential Errors Along The Way

This section is the most important part as it’s likely how you found this post. Below are the few problems (and messages) that might arise that are solved by the installation of specific packages).

1. Boost Error And Attempted sudo apt-get install boost-build

The error with and without a boost-build install is the same.

...failed updating 2 targets...
...skipped 3 targets...
...updated 7846 targets...
boost.sh: build failed: Terminating...
CMake Error at c++/CMakeLists.txt:177 (message):
  Failed to build Boost


-- Configuring incomplete, errors occurred!
make: *** [build/Makefile] Error 1

The full list from the build attempts for both cases can be viewed in (in 2010dec7__bclconverter_1_7_1_logs.zip:

* 2010dec7__bclconverter_1_7_1_build1__boosterror__FAILED.txt – initial error
* 2010dec7__bclconverter_1_7_1_build2__aptgetboost__FAILED.txt – after boost-build install

Running the full apt-get (see the contents of 2010dec7__bclconverter_1_7_1_logs.zip, with the results in 2010dec7__bclconverter_1_7_1_build3a__aptgetlist__RESULTS) produces a successful BclConverter build. The log for my build is available in 2010dec7__bclconverter_1_7_1_build3b__successful__BUILD.txt.

2. XML:Simple-Related Error And libxml-simple-perl

Without either gnuplot or the libxml-simple-perl installation, a setupBclToQseq.py run will successfully generate QSEQ files. The additional tools provide you with some statistical and visual analyses of your results (so are definitely worth installing).

If you don’t have libxml-simple-perl installed, you’ll see the following error after running the BaseCalls “make”:

/opt/BclConverter-1.7.1/bin/plotIntensity_tiles.pl tiles.txt s_8 SignalMeans _all.txt 
_all.png && echo `date` > s_8_all_pngs.txt
Can't locate XML/Simple.pm in @INC (@INC contains: /opt/BclConverter-1.7.1/lib/perl /etc/perl
/usr/local/lib/perl/5.10.1 /usr/local/share/perl/5.10.1 /usr/lib/perl5 /usr/share/perl5
/usr/lib/perl/5.10 /usr/share/perl/5.10 /usr/local/lib/site_perl .) at 
/opt/BclConverter-1.7.1/lib/perl/Gerald/Common.pm line 128.
BEGIN failed--compilation aborted at /opt/BclConverter-1.7.1/lib/perl/Gerald/Common.pm
line 128.
Compilation failed in require at /opt/BclConverter-1.7.1/bin/plotIntensity_tiles.pl line 22.
...
make: *** [s_1_all_pngs.txt] Error 2
make: *** [s_2_all_pngs.txt] Error 2
make: *** [s_3_all_pngs.txt] Error 2
make: *** [s_4_all_pngs.txt] Error 2
make: *** [s_5_all_pngs.txt] Error 2
make: *** [s_6_all_pngs.txt] Error 2
make: *** [s_7_all_pngs.txt] Error 2
make: *** [s_8_all_pngs.txt] Error 2

The fix is trivial (if you’re doing it incrementally and don’t already have XML:simple installed):

sudo apt-get install libxml-simple-perl

3. gnuplot Errors And Fix

If you don’t have gnuplot already installed (and why wouldn’t you?), you’ll receive the following error during the BCL-to-QSEQ “make” process:

sh: gnuplot: not found
/opt/BclConverter-1.7.1/share/makefiles/bclToQseq/FlowCellTargets.mk:76: [IVC.htm
(s_1_all_pngs.txt s_2_all_pngs.txt s_3_all_pngs.txt s_4_all_pngs.txt s_5_all_pngs.txt
s_6_all_pngs.txt s_7_all_pngs.txt s_8_all_pngs.txt plotIntensity_for_IVC_finished.txt)
(s_1_all_pngs.txt s_2_all_pngs.txt s_3_all_pngs.txt s_4_all_pngs.txt s_5_all_pngs.txt
s_6_all_pngs.txt s_7_all_pngs.txt s_8_all_pngs.txt plotIntensity_for_IVC_finished.txt)]

/opt/BclConverter-1.7.1/bin/create_IVC_thumbnail.pl . > IVC.htm.tmp && mv IVC.htm.tmp
IVC.htm
/opt/BclConverter-1.7.1/share/makefiles/bclToQseq/FlowCellTargets.mk:82: [All.htm
(s_1_all_pngs.txt s_2_all_pngs.txt s_3_all_pngs.txt s_4_all_pngs.txt s_5_all_pngs.txt
s_6_all_pngs.txt s_7_all_pngs.txt s_8_all_pngs.txt plotIntensity_for_IVC_finished.txt)
(s_1_all_pngs.txt s_2_all_pngs.txt s_3_all_pngs.txt s_4_all_pngs.txt s_5_all_pngs.txt
s_6_all_pngs.txt s_7_all_pngs.txt s_8_all_pngs.txt plotIntensity_for_IVC_finished.txt)]

/opt/BclConverter-1.7.1/bin/create_tile_thumbnails.pl all > FullAll.htm && \
	/opt/BclConverter-1.7.1/bin/create_tile_thumbnails.pl all --maxTiles=20 --link='
_a href="FullAll.htm"_Full output (Warning: may overload your browser!)_/a_' > All.htm.tmp 
&& mv All.htm.tmp All.htm

/opt/BclConverter-1.7.1/share/makefiles/bclToQseq/FlowCellTargets.mk:109:
[BustardSummary.xml (IVC.htm All.htm tiles.txt) (IVC.htm All.htm tiles.txt)]
/opt/BclConverter-1.7.1/bin/produceIntensityStats.pl .
Unable to find file /LOCATION_OF_INTENSITIES_FOLDER/Intensities/BaseCalls/../../
../samples.xml at /opt/BclConverter-1.7.1/lib/perl/Gerald/Jerboa.pm line 387.

...

/opt/BclConverter-1.7.1/share/makefiles/bclToQseq/FlowCellTargets.mk:58: [finished.txt
(Matrix Phasing s_1 s_2 s_3 s_4 s_5 s_6 s_7 s_8 BustardSummary.xml BustardSummary.xsl
IVC.htm All.htm) (Matrix Phasing s_1 s_2 s_3 s_4 s_5 s_6 s_7 s_8 BustardSummary.xml
BustardSummary.xsl IVC.htm All.htm)]
touch finished.txt.tmp && mv finished.txt.tmp finished.txt

With a:

sudo apt-get install gnuplot

All remaining errors in the BCL-to-QSEQ “make” process should disposed of, leaving you with a Plots directory containing multiple .png files after the QSEQ generation process.

4. Just Running “make” For BCL-to-QSEQ

The successful setupBclToQseq.py run:

setupBclToQseq.py -i /LOCATION_OF_FILES/Intensities/BaseCalls -p /LOCATION_OF_FILES/Intensities
 -o /LOCATION_OF_FILES/Intensities/BaseCalls --in-place --overwrite

ends with (also in setupBclToQseq.log):

setupBclToQseq.py version 1.7.1

Configuring /opt/BclConverter-1.7.1/share/makefiles/bclToQseq/Makefile to 
/LOCATION_OF_FILES/Intensities/BaseCalls/Makefile

Creating the 'Makefile.config'

Output directory succesfully initialized. Type 'make' in 
/LOCATION_OF_FILES/Intensities/BaseCalls to start the conversion

And if you simply type “make,” you get the following error:

/opt/BclConverter-1.7.1/bin/plotIntensity_tiles.pl tiles.txt s_1 SignalMeans _all.txt
_all.png && echo `date` > s_1_all_pngs.txt
Can't locate XML/Simple.pm in @INC (@INC contains: /opt/BclConverter-1.7.1/lib/perl /etc/perl 
/usr/local/lib/perl/5.10.1 /usr/local/share/perl/5.10.1 /usr/lib/perl5 /usr/share/perl5 
/usr/lib/perl/5.10 /usr/share/perl/5.10 /usr/local/lib/site_perl .) at 
/opt/BclConverter-1.7.1/lib/perl/Gerald/Common.pm line 128.
BEGIN failed--compilation aborted at /opt/BclConverter-1.7.1/lib/perl/Gerald/Common.pm 
line 128.
Compilation failed in require at /opt/BclConverter-1.7.1/bin/plotIntensity_tiles.pl line 22.
BEGIN failed--compilation aborted at /opt/BclConverter-1.7.1/bin/plotIntensity_tiles.pl
line 22.
make: *** [s_1_all_pngs.txt] Error 2

The complete log is available in 2010dec7__bclconverter_1_7_1_build3c__make_error_wo_j8.txt in
2010dec7__bclconverter_1_7_1_logs.zip.

A brief User Guide read will hip you to a proper run command in the BaseCalls directory (obvious, read the User Guide):

make -j 8

For a simple test (and I assume that your network directory structure for the Illumina is something like /LOCATION/TO/NETWORK/DATA/Data/Intensities and /LOCATION/TO/NETWORK/DATA/Data/Intensities/Basecalls (which it should be), we’ll use the example in the BCLConverter User Guide (and be sureto download the .PDF).

This will produce a sizable logfile. You can check out a successful run in 2010dec7__bclconverter_1_7_1_build3i__with_gnuplot.txt in 2010dec7__bclconverter_1_7_1_logs.zip.

5. QSEQ-to-FASTQ Script

Not really an error, just a last little help to convert your QSEQ files into generic FASTQ format.

#!/bin/bash
for ((x=1;x< =8;x+=1)); do 
cat s_"$x"_1_*_qseq.txt | awk -F '\t' '{gsub(/\./,"N", $9); if ($11 > 0) printf("@%s_%04d:
%s:%s:%s:%s#%s/%s\n%s\n+%s_%04d:%s:%s:%s:%s#%s/%s\n%
\n",$1,$2,$3,$4,$5,$6,$7,$8,$9,$1,$2,$3,$4,$5,$6,$7,$8,$10)}' > s_
"$x"_sequence.fastq; 
done

NOTE: the “cat” contents has to be all on one line! Copy this script into a text editor and reformat (or download a copy – 2010dec7__qseq_to_fastq.script).

Obligatory

  • CNYO

  • Sol. Sys. Amb.

  • Ubuntu 4 Nano

  • NMT Review

  • N-Fact. Collab.

  • Pres. Asn. CNY

  • T R P Nanosys

  • Nano Gallery

  • nano gallery
  • Aerial Photos

    More @ flickr.com

    Syracuse Scenes

    More @ flickr.com