NOTE 25 March 2009: The problem with Open-MPI and Abinit is related to the version of Open-MPI. 1.3.x is used below, while 1.2.x allows you to use the .files for running batch-based Abinit calculations. See www.somewhereville.com/?p=590 for additional notes.
The purpose of the HPLIP fix reported in a previous post was to install Abinit in Ubuntu via apt-get in order to employ the procedure used by Hooper et al in Chemical Physics Letters to calculate infrared intensities in the low-frequency region for solid-state terahertz (THz) assignments (phew!). The problem is that the apt-get install of Abinit is an older and serial (non-parallel) version. Further compounding the problem, the OpenMPI version one can install via apt-get (sudo apt-get install openmpi-bin) does not have F90 (Fortran 90) support, so one cannot simply install OpenMPI, install one of the pre-compiled versions of Abinit, and start using those other processors either on the board or plugged into a gigabit switch.
As long as you're building one from source anyway, you might as well build them both. As has been the case in several of my previous posts involving code compilation, the procedure below assumes as little as possible from you (the code builder). The scientific audience that potentially serves to gain the most from the procedure is, in my past experience, in need of as much detail as possible to explain what is happening and to spell it out in enough detail that ANYONE could do it. While I'm a great fan of Open Source Software, I think the best way to make an academic software novice go to a catalog and buy GUI-based computational chemistry software is to tell them only to ./configure, make, and make install.
LEGEND
Text in black – my ramblings.
Text in bold preformatted red - things you will type in the Terminal
Text in green – text you will either see or will type into files (using pico, my preference)
Step 0: Coming To Grips With sudo (taken from HERE)
The one aspect of Ubuntu that differs from most other Linux distributions is the differentiation between root, Administrator, and user right from the installation. Whereas you set up the root user in Suse and Fedora as part of the installation process, you set up an Administrator account during Ubuntu installation that is distinct from root. As a result, if you do not set up the root account to perform installations and system-level modifications, you are left in the Administrator account to use the sudo (super-user do…) command to allow you, the Administrator, to build and install programs outside your home ($HOME) directory.
"Do I have to constantly sudo everything?" No. Accessing a pure "root" terminal for installations is straightforward after the root password is assigned, it is simply argued by many (including the Ubuntu wiki) that it is safer to use sudo. If you want to go the root route, check out help.ubuntu.com/community/RootSudo.
Step 1. Install Ubuntu 8.10 Desktop or Server (borrowed from HERE)
Default installation, simply walk through the entire installation process. For DHCP people, this should be no problem once the machine reboots. I've been having an odd issue with the fixed IP address machine I'm working on that I provide the workaround for below (which, hopefully, you do not need, but it reminds me to diagnose it properly later). Specifically, IP information was not being saved for edited connections. I don't know if this is GUI-related (not saving changes) or permission-related (not being allowed to make persistent changes). As a quick ifconfig fix described in more detail HERE (eth0 being your ethernet card (likely eth0 if you've only one card), 128.230.XXX.XXX is half my local IP address + half none of your business, and gw is the gateway for your subnet (probably XXX.XXX.XXX.1)):
sudo ifconfig eth0 down
sudo ifconfig eth0 128.230.XXX.XXX netmask 255.255.255.0 up
sudo route add default gw 128.230.XXX.XXX
Step 2. Terminal (stolen from HERE)
There's no use in driving through the Applications drop box to get to a program you'll be using as often as Terminal. If you didn't know it yet, go to Applications -> Accessories -> left-click-and-hold on the Terminal icon, then drag the icon to the panel (next to your handy Firefox icon, for instance) or right-click and "Add this launcher to panel."
Step 3. apt-get (evolved from HERE)
It is at this point that your internet connection becomes vital in Ubuntu. In the words of Bruce Byfield…
"If any single program defines the Debian Linux project, that program is apt-get."
If you're a research scientist hoping to build software from source, it's about as an important an interface between Ubuntu and your programs as your neck is between your head and your body. For building Abinit, you will need to install several programs (and associated libraries. One required program, g95, is not get available by apt-get, but we'll get to that later). apt-get makes this process (nearly) painless.
At the Terminal window:
sudo apt-get update
This updates the package list in the apt-get database (and it is recommended that you run this regularly, as it is your connection to newly available and updated programs).
sudo apt-get install g++ gfortran autoconf patch csh netcdf-bin [continued on next line]
ssh libblas3gf liblapack3gf libgfortran2
This installs seven base programs and many additional programs and libraries (for the record, I did not need to install libblas3gf, liblapack3gf, or libgfortran2 to successfully compile both OpenMPI and Abinit on my first attempt, but the Abinit version available by apt-get lists these libraries at dependencies and they did not appear to be installed on the original Ubuntu installation but are clearly there upon compilation (which I assume means they are provided and compiled as needed by Abinit), so this is just an extra "what if?" safety on my part). For those that do not know, you discover which of these programs are required by running an installation, reading the output when the build fails, installing the necessary programs, then cycling through the install-fail step until you get to the end.
Why gfortran? or, why not fort77? In the compilation of Amber10, I had some issue with running the tests using gfortran that required script modification I was not inclined to attempt. In trying to build OpenMPI, I received the following error that, you guessed it, went away using gfortran (yes, only the bottom line is of any relevance, but using the whole section does fill the page out a bit).
*** Fortran 77 compiler
checking for gfortran… no
checking for g77… no
checking for f77… f77
checking whether we are using the GNU Fortran 77 compiler… yes
checking whether f77 accepts -g… yes
checking if Fortran 77 compiler works… yes
checking f77 external symbol convention… double underscore
checking if C and Fortran 77 are link compatible… yes
checking to see if F77 compiler likes the C++ exception flags… skipped (no C++ exceptions flags)
checking if Fortran 77 compiler supports LOGICAL… yes
checking size of Fortran 77 LOGICAL… 4
checking for C type corresponding to LOGICAL… int
checking alignment of Fortran LOGICAL… 4
checking if Fortran 77 compiler supports LOGICAL*1… yes
checking size of Fortran 77 LOGICAL*1… 1
checking for C type corresponding to LOGICAL*1… char
checking alignment of Fortran LOGICAL*1… 1
checking if Fortran 77 compiler supports LOGICAL*2… yes
checking size of Fortran 77 LOGICAL*2… 2
checking for C type corresponding to LOGICAL*2… short
checking alignment of Fortran LOGICAL*2… 2
checking if Fortran 77 compiler supports LOGICAL*4… yes
checking size of Fortran 77 LOGICAL*4… 4
checking for C type corresponding to LOGICAL*4… int
checking alignment of Fortran LOGICAL*4… 4
checking if Fortran 77 compiler supports LOGICAL*8… no
checking if Fortran 77 compiler supports INTEGER… yes
checking size of Fortran 77 INTEGER… 4
checking for C type corresponding to INTEGER… int
checking alignment of Fortran INTEGER… 4
checking if Fortran 77 compiler supports INTEGER*1… yes
checking size of Fortran 77 INTEGER*1… 1
checking for C type corresponding to INTEGER*1… char
checking alignment of Fortran INTEGER*1… 1
checking if Fortran 77 compiler supports INTEGER*2… yes
checking size of Fortran 77 INTEGER*2… 2
checking for C type corresponding to INTEGER*2… short
checking alignment of Fortran INTEGER*2… 2
checking if Fortran 77 compiler supports INTEGER*4… yes
checking size of Fortran 77 INTEGER*4… 4
checking for C type corresponding to INTEGER*4… int
checking alignment of Fortran INTEGER*4… 4
checking if Fortran 77 compiler supports INTEGER*8… yes
checking size of Fortran 77 INTEGER*8… configure: error: Could not determine size of INTEGER*8
Admittedly, switching compilers is not much of a proper fix when either should work, but, since this install revolves around apt-get anyway, I decided to not dig any further into script modification than I needed to (so you don't have to, either).
autoconf and netcdf-bin are required for the Abinit build (well, autoconf for the makemake step).
You don't realize you need ssh until you try to run a parallel instance of OpenMPI and get the error below (I reproduce the error in total below because someone will invariably search based on some part of the error message. Hope this explains it! You can compile OpenMPI without ssh, but ssh is more generally useful than you might initially realize, so might as well have it installed from the get-go):
ssh: connect to host [machine] port 22: Connection refused
————————————————————————–
A daemon (pid 27364) died unexpectedly with status 255 while attempting
to launch so we are aborting.
There may be more information reported by the environment (see above).
This may be because the daemon was unable to find all the needed shared
libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
location of the shared libraries on the remote nodes and this will
automatically be forwarded to the remote nodes.
————————————————————————–
————————————————————————–
mpirun noticed that the job aborted, but has no info as to the process
that caused that situation.
————————————————————————–
mpirun: clean termination accomplished
NOTE 1: If you get an error related to hplip, see my previous post about fixing it.
Step 4: .profile Modifications
In the interest of keeping track of where I install programs on a machine, I tend to use the /opt directory, which Perry thinks is crazy. You may see /usr/local, /usr/local/bin, or various combinations thereof in your web searching travels. If you're new to Linux, you'll blindly do what I did because you're concerned that something below will rely on something being in /opt. You will note that the new directory that Abinit creates after building is (with the building in /opt) /opt/etsf/abinit/5.6/, which is fine with me (that's what the PATH specification below is for and, as long as it stays in /opt, I'm not bothered with the location).
cd
(or cd $HOME, but the default operation for cd is to return you to your home directory)
pico .profile
Into .profile, place the following (at bottom, just to keep track of the changes you make)
PATH="/opt/etsf/abinit/5.6/bin:$PATH"
LD_LIBRARY_PATH="/usr/local/lib/openmpi/"
Again, this changes if you want the Abinit executables somewhere else. As for the library specification, I'll cover that below.
Crtl-X, The Enter Key twice, and done.
To make these changes to your account, make sure you're in your home directory…
source .profile
Step 5. Installing g95 (lifted from HERE)
The g95 program is an open source fortran compiler with support for f90 and f95. I found myself NOT using CPMD a few years ago because I could not find a free f90 compiler (this was before the Intel Fortran Compiler became available for Linux) and am very happy that g95 has matured as of late. That said, there is no apt-get for it yet, so you need to install this the olde-fashioned way. With its Debian underpinnings, the Ubuntu g95 installation is trivial. You need only be cognizant of whether you installed the 32-bit or 64-bit versions of Ubuntu. If you don't know, it will be obvious when you go to build Abinit (and you can simply install the other flavor over the old one).
At g95.org/downloads.shtml#CS, you will find the current snapshots for various flavors and installation formats for g95. You either want to download the Linux x86 (Debian) or the Linux x86-64 (Debian) packages. If you're using Firefox in Ubuntu, your download folder should be your Desktop. Sticking to the Terminal, simply type the following:
cd ~/Desktop
to go to the Desktop Folder in your $HOME directory
sudo dpkg -i g95-x86.deb
if you downloaded with the intent of installing the 32-bit version, or
sudo dpkg -i g95-x86_64.deb
if you downloaded with the intent of installing the 64-bit version
The dpkg program is Debian's equivalent of Redhat's rpm (and vice versa). It will extract the contents of a .deb file and place all of the executables and libraries in their proper folders PROVIDED you remember the sudo. If you do not use the .deb package and decide to install from the .tgz file, you may find yourself having to copy libraries and executables into relevant directories. If possible, stick with dpkg.
Step 6. Installing OpenMPI 1.3
It is hoped that this step is also the missing piece for the Amber10 parallel build which may or may not be appearing shortly on this website (if it works). OpenMPI is the MPI (message passing interface) program used as the conduit through which a number of comp. chem. programs distribute calculations over multiple processors (both on the same board (SMP) and between machines on a network). I suspect MPICH and MPICH2 are just as well, but OSX uses OpenMPI and the pre-compiled Abinit binaries work through OpenMPI (which I expect will continue to be the case for future versions of the program). Once THIS version of OpenMPI is compiled (with F90 support via g95), I suspect you should be all set for future Abinit versions.
With gfortran and g95 properly installed, the build of OpenMPI 1.3 is trivial. Simply download the OpenMPI source (which, with Firefox, will likely place it on the Desktop) and move the source to /opt to begin the build.
sudo cp ~/Desktop/openmpi-1.3.tar.gz /opt/
cd /opt
sudo tar xvjf openmpi-1.3.tar.gz
will make a folder openmpi-1.3 in opt/
cd openmpi-1.3
./configure
A considerable amount of output follows. To see what an otherwise flawless installation looks like, you can view my output HERE.
sudo make all install
A considerable amount of output follows. To see what an otherwise flawless installation looks like, you can view my output HERE.
There is one more step for the Abinit build that I would have expected to be handled by the specification of LD_LIBRARY_PATH in .profile but are not. There are several MPI libraries that Abinit needs for its parallel compilation that are placed in /usr/local/lib without links to those libraries placed anywhere else. Abinit seems to force itself to look in /lib. The fix is either to place the files below into /lib or to make links (ln -s). I opted to simply copy them into /lib. And you learn this list of files the hard way (four build attempts of Abinit with errors for each of the file sets below).
sudo cp /usr/local/lib/libopen-pal.* /lib
sudo cp /usr/local/lib/libmpi* /lib
sudo cp /usr/local/lib/libopen-rte.* /lib
Step 7. Installing Abinit 5.6.5
The relative ease with which this install occurs (after the work above) from sources is a credit to the Abinit developer community. To begin, download the Abinit source
sudo cp ~/Desktop/abinit-5.6.5.tar.gz /opt/
cd /opt
sudo tar xvjf abinit-5.6.5.tar.gz
will make a folder abinit-5.6.5 in opt/. To install, we'll be going nearly by the book…
The instructions are contained in the
doc/install_notes directory .
You can find instructions in
http://www.abinit.org/developers/?text=ab5_10steps
In short, if you have bazaar installed, and got one version from
the archive, the steps are :
(1) ./config/scripts/makemake (or */*/makemake)
(2) ./configure (or first create a tmp directory, then cd tmp, then ../configure)
(3) ./make (or ./make multi for using 4 proc of an SMP machine)
Then you might also need
(4) ./make install
… with the usual sudo modifications. the makemake step requires autoconf be installed (which we did in our apt-get install list).
cd abinit-5.6.5
sudo ./config/scripts/makemake
will generate the output provided HERE. For the compilation of Abinit, we want mpi support, so we run ./configure as follows:
sudo ./configure --enable-mpi="yes"
With all of the compilers and libraries properly places as described above, this configure process provides the flawless output provided HERE. To build, it is simply:
sudo make multi
will generate the output provided HERE.
Random Aside: For some reason, I tried the sudo make multi_alt build option first and received the BigDFT_API error. Listed purely for academic reasons, but if you see this error and you used multi_alt, don't use multi_alt (because bigdft_api.mod, the associated file, shows up nowhere in google).
cd src/defs ; make
make[1]: Entering directory `/opt/abinit-5.6.5/src/defs'
/usr/local/bin/mpif90 -DHAVE_CONFIG_H -I. -Ihttp://www.somewhereville.com/ -Ihttp://www.somewhereville.com/src/defs -Ihttp://www.somewhereville.com/src/defs -I/opt/abinit-5.6.5/plugins/bigdft -g -O2 -c -o defs_basis.o defs_basis.F90
/usr/local/bin/mpif90 -DHAVE_CONFIG_H -I. -Ihttp://www.somewhereville.com/ -Ihttp://www.somewhereville.com/src/defs -Ihttp://www.somewhereville.com/src/defs -I/opt/abinit-5.6.5/plugins/bigdft -g -O2 -c -o defs_aimfields.o defs_aimfields.F90
/usr/local/bin/mpif90 -DHAVE_CONFIG_H -I. -Ihttp://www.somewhereville.com/ -Ihttp://www.somewhereville.com/src/defs -Ihttp://www.somewhereville.com/src/defs -I/opt/abinit-5.6.5/plugins/bigdft -g -O2 -c -o defs_aimprom.o defs_aimprom.F90
/usr/local/bin/mpif90 -DHAVE_CONFIG_H -I. -Ihttp://www.somewhereville.com/ -Ihttp://www.somewhereville.com/src/defs -Ihttp://www.somewhereville.com/src/defs -I/opt/abinit-5.6.5/plugins/bigdft -g -O2 -c -o defs_datatypes.o defs_datatypes.F90
/usr/local/bin/mpif90 -DHAVE_CONFIG_H -I. -Ihttp://www.somewhereville.com/ -Ihttp://www.somewhereville.com/src/defs -Ihttp://www.somewhereville.com/src/defs -I/opt/abinit-5.6.5/plugins/bigdft -g -O2 -c -o defs_berry.o defs_berry.F90
/usr/local/bin/mpif90 -DHAVE_CONFIG_H -I. -Ihttp://www.somewhereville.com/ -Ihttp://www.somewhereville.com/src/defs -Ihttp://www.somewhereville.com/src/defs -I/opt/abinit-5.6.5/plugins/bigdft -g -O2 -c -o defs_dyson.o defs_dyson.F90
/usr/local/bin/mpif90 -DHAVE_CONFIG_H -I. -Ihttp://www.somewhereville.com/ -Ihttp://www.somewhereville.com/src/defs -Ihttp://www.somewhereville.com/src/defs -I/opt/abinit-5.6.5/plugins/bigdft -g -O2 -c -o defs_elphon.o defs_elphon.F90
/usr/local/bin/mpif90 -DHAVE_CONFIG_H -I. -Ihttp://www.somewhereville.com/ -Ihttp://www.somewhereville.com/src/defs -Ihttp://www.somewhereville.com/src/defs -I/opt/abinit-5.6.5/plugins/bigdft -g -O2 -c -o defs_fftdata.o defs_fftdata.F90
/usr/local/bin/mpif90 -DHAVE_CONFIG_H -I. -Ihttp://www.somewhereville.com/ -Ihttp://www.somewhereville.com/src/defs -Ihttp://www.somewhereville.com/src/defs -I/opt/abinit-5.6.5/plugins/bigdft -g -O2 -c -o defs_interfaces.o defs_interfaces.F90
/usr/local/bin/mpif90 -DHAVE_CONFIG_H -I. -Ihttp://www.somewhereville.com/ -Ihttp://www.somewhereville.com/src/defs -Ihttp://www.somewhereville.com/src/defs -I/opt/abinit-5.6.5/plugins/bigdft -g -O2 -c -o defs_parameters.o defs_parameters.F90
/usr/local/bin/mpif90 -DHAVE_CONFIG_H -I. -Ihttp://www.somewhereville.com/ -Ihttp://www.somewhereville.com/src/defs -Ihttp://www.somewhereville.com/src/defs -I/opt/abinit-5.6.5/plugins/bigdft -g -O2 -c -o defs_scalapack.o defs_scalapack.F90
/usr/local/bin/mpif90 -DHAVE_CONFIG_H -I. -Ihttp://www.somewhereville.com/ -Ihttp://www.somewhereville.com/src/defs -Ihttp://www.somewhereville.com/src/defs -I/opt/abinit-5.6.5/plugins/bigdft -g -O2 -c -o defs_suscep.o defs_suscep.F90
/usr/local/bin/mpif90 -DHAVE_CONFIG_H -I. -Ihttp://www.somewhereville.com/ -Ihttp://www.somewhereville.com/src/defs -Ihttp://www.somewhereville.com/src/defs -I/opt/abinit-5.6.5/plugins/bigdft -g -O2 -c -o defs_time.o defs_time.F90
/usr/local/bin/mpif90 -DHAVE_CONFIG_H -I. -Ihttp://www.somewhereville.com/ -Ihttp://www.somewhereville.com/src/defs -Ihttp://www.somewhereville.com/src/defs -I/opt/abinit-5.6.5/plugins/bigdft -g -O2 -c -o defs_wannier90.o defs_wannier90.F90
/usr/local/bin/mpif90 -DHAVE_CONFIG_H -I. -Ihttp://www.somewhereville.com/ -Ihttp://www.somewhereville.com/src/defs -Ihttp://www.somewhereville.com/src/defs -I/opt/abinit-5.6.5/plugins/bigdft -g -O2 -c -o defs_wvltypes.o defs_wvltypes.F90
defs_wvltypes.F90:33.15:
use BigDFT_API
1
Fatal Error: Can't open module file 'bigdft_api.mod' for reading at (1): No such file or directory
make[1]: *** [defs_wvltypes.o] Error 1
make[1]: Leaving directory `/opt/abinit-5.6.5/src/defs'
make: *** [multi_alt] Error 2
sudo make install
will generate the output provided HERE. The result is a set of executables installed in /opt/etsf/abinit/5.6/bin, which is specified in our PATH statement in .profile.
Step 8. Running Calculations With mpirun
The above was almost pleasant. If you are running the serial version of Abinit (abinis), which is also built with the parallel installation, you simply need to type the following at a Terminal prompt.
abinis < inputfile.script >& outputfile.output
because our PATH is already specified in .profile.
MPI calculations require a bit more out of your fingers. For the specific build I have performed here, there is also one very odd issue with the parallel run that I am still trying to work out the solution to (but it is by no means mission-critical to the proper compilation AND the workaround is dumb to put it mildly, so it hasn't stopped me from this Ubuntu/OpenMPI/Abinit post).
To run parallel Abinit on a multi-processor box (that is, SMP. The actual multi-node cluster setup is in progress), the command is SUPPOSED to be follows:
mpirun -np N /opt/etsf/abinit/5.6/bin/abinip < input.file >& output
Where N is the number of processors. For mpirun, you need to specify the full path to the executable (which, for the build above, is as Abinit installs abinip when thebuild occurs in the /opt directory). The input.file specification is as per the Abinit users manual so I won't go into it here. You will also be asked to supply your password because I've done nothing to the setup of ssh (you are, in effect, logging into your machine to run the MPI calculation).
Now, when the above is run, this is the error that I get:
abinit : nproc,me= 4 0
ABINIT
Give name for formatted input file:
At line 127 of file iofn1.F90 (unit = 5, file = 'stdin')
Fortran runtime error: End of file
abinit : nproc,me= 4 1
abinit : nproc,me= 4 2
abinit : nproc,me= 4 3
————————————————————————–
mpirun has exited due to process rank 0 with PID 7131 on
node terahertz-desktop exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
————————————————————————–
What is supposed to happen is that the input.file file lists the files that Abinit requires to perform the run and provides these files by name one-at-a-time as Abinit requests them upon start-up. For some reason, the input.file file is not being read properly or is not being read at all before the job crashes. Oddities noted in the above order of the output include
(1) the abinit : nproc,me= values are not grouped above the "Give name for formatted input file:" <- Abinit does not appear to be trying to read the text from the nproc,me lines as actual input data, as you have to provide all of the files before Abinit will crash with a wrong file name.
(2) At line 127 of file iofn1.F90 <- this is an Abinit file that is repsonsible for reading the contents of input.files. So, is the problem with this fine in Abinit? Well…
(3) The serial build of Abinit (abinis) runs just fine with input.file <- which leads me to conclude that the problem is mpirun-related. I hope to resolve this (I'm sure it will be trivial) and post my error accordingly.
What's the work-around? Simple. Copy the contents of the input.file file (literally Crtl+C with the text selected) and paste it after running this command:
mpirun -np N /opt/etsf/abinit/5.6/bin/abinip
Abinit will ask for the files in order AND your Crtl+C includes the carraige returns at the end of each line, so you are effectively feeding Abinit the same content it would read from the input.file file if, in fact, it was capable of reading the input.file file.
Final Thoughts (ripped from HERE)
There will be a bit more to follow, perhaps significantly more if the Intel Fortran Compiler yields a considerable speed-up in calculation times as reported by Torsten Hoefler in his Abinit-specific survey of compilers and parallelization codes. If questions are raised, comments are thought of, speed-ups identified, etc., please either send me an email or post them here. Our concern as computational chemists should be making predictions and interpreting data, not making compilation errors and interpreting error messages.
hplipopensource.com/hplip-web/index.html
www.somewhereville.com/?p=149
www.abinit.org
www.ubuntu.com
www.debian.org/doc/manuals/apt-howto/
www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6TFN-4V166C7-9&_user=783137&_rdoc=1&_fmt=&_orig=search…
www.sciencedirect.com/science/journal/00092614
www.open-mpi.org
en.wikipedia.org/wiki/Fortran
www.opensource.org
www.opensuse.org/en/
fedoraproject.org
en.wikipedia.org/wiki/Sudo
help.ubuntu.com/community/RootSudo
en.wikipedia.org/wiki/Dhcp
en.wikipedia.org/wiki/Ifconfig
linux-ip.net/html/basic-changing.html
help.ubuntu.com/community/UsingTheTerminal
www.mozilla.com/en-US/firefox/
members.axion.net/~bbyfield/
www.g95.org
www.somewhereville.com/?p=345
ambermd.org
www.openssh.com
www.piermont.com
www.cpmd.org
www.intel.com/cd/software/products/asmo-na/eng/282048.htm
www.us.debian.org
g95.org/downloads.shtml#CS
en.wikipedia.org/wiki/Dpkg
en.wikipedia.org/wiki/RPM_Package_Manager
www-unix.mcs.anl.gov/mpi/mpich1
www.mcs.anl.gov/research/projects/mpich2
www.apple.com/osx
www.unixer.de/research/abinit