Already a member?
Sign in
| Version | User | Scope of changes |
|---|---|---|
| May 29 2008, 4:06 PM EDT (current) | laytonjb | 2 words added, 1 word deleted |
| May 14 2008, 7:45 AM EDT | laytonjb | 15 words added, 2 words deleted |
Changes
Key: Additions Deletions
Environment Modules:
Scratching the Multi-MPI, Multi-Compiler Itch
When people first start using clusters they tend to stick with whatever compiler and MPI library came with the cluster when it was installed. As they become more comfortable with the cluster, using the compilers, using the MPI libraries, they start to look around at other options. Are there other compilers that could perhaps improve performance? Similarly, they may start looking at other MPI libraries – can they help improve performance? Do other MPI libaries have tools that can make things easier? Perhaps even more importantly, these people would like to install the next version of the compilers or MPI libraries so they can test them with their code(s). So this begs the question, how do you have multiple compilers and multiple MPI libraries on the cluster at the same time and not get them confused? I’m glad you asked.
Caveman Approach
I don’t want to get the cavemen angry with me, but the “uneducated” approach to using multiple compilers and MPI libraries on a cluster is, shall we say, kludgy. The basic idea is to change your $PATH in the .bashrc (if you are using bash), and then log out and log back in whenever you need to change your compiler/MPI combination. Initially this sounds like a pain and it is, but it works – to some degree. It doesn’t work in the situation where you want to run multiple jobs each with a different compiler/MPI combination.
For example, let’s say I have a job using the gcc 4.2.1compilers4.2.1 compilers using Open MPI. Then I have a job using gcc 4.2.1 and MPICH2. If I have both jobs in the queue at the same time, how can I control my .bashrc to make sure each job has the correct $PATH? The only way to do this is to restrict myself to one job in the queue at a time. When it’s finished I can then change my .bashrc and submit a new job. Even this is not the best approach because if you are using a different compiler/MPI combination from what is in the queue for something as simple as code development, you have to watch when the job is run to make sure your .bashrc matches your job.
Cro-Magnon Man Approach – Environment Modules
A much better way to handle compiler/MPI combinations is to use Environment Modules (Be careful not to confuse “environment modules” with “kernel modules”). According to the website, “The Environment Modules package provides for the dynamic modification of a user’s environment via modulefiles.” While this may not sound earth shattering, it actually is a quantum leap for using multiple compilers/MPI libraries. But you can use it for more than just compiler/MPI combinations, which I will talk about later.
You can use environment modules to alter or change environment variables such as $PATH, $MANPATH, $LD_LIBRARY_LOAD, and others. Since most job scripts for resource managers such as LSF , PBS-Pro, MOAB, are really shell scripts, you can incorporate environment modules into the scripts to set the appropriate $PATH your compiler/MPI combination needs or any other environment variables your application(s) require for operation.
Installing Environment Modules
How you install environment modules depends upon how your cluster is built. For those that use Platform OCS 4.x Platform has created a roll. Here is a readme describing the various rolls Platform has created. The roll should be included on the OCS DVD that came with the cluster.
If you didn’t use Platform OCS, then you should look around for something like an rpm for environment modules (assuming you used an rpm based cluster toolkit). Remember that Google is your friend when looking for rpm’s. If you can’t find an appropriate binary containing environment modules for your system, then you will have to build it from scratch. Fortunately, the instructions on the website are fairly good and you shouldn’t have too much trouble.
Some Examples for Using Environment Modules
I won’t discuss how to create or define environment modules. The website does a far better job than I in explaining how to do this. But if there is enough demand, I may do a future blog that discusses how to create modules for new compilers/MPI libraries/applications. So let’s assume that the environment modules are installed and functioning.
The first thing to check is what modules are available to you by using the “module avail” command:
home8:~> module avail
-------------------------- /usr/share/modules/modulefiles --------------------------
compiler/gcc dot null
compiler/pgi5.2 module-cvs pbs
compiler/pgi5.2-x86_64 module-info use.own
compiler/pgi6.0 modules
compiler/pgi6.0-x86_64 mpi/mpich-1.2.6
home8:~>
This commands lists what environment modules are installed. In this cases there are a number of compilers installed – gcc, pgi 5.2, pgi 5.2-x86_64, pgi 6.0, pgi 6.0-x86_64, as well as a single MPI library – mpich-1.2.6.
Let’s “load” one of the environment modules:
home8:~> module load compiler/pgi5.2-x86_64
You can just cut and past from the list of available modules to load the ones you want or need (this is what I do and it makes things easier). By loading a module you will have just changed the environmental variables defined for that module. Typically this is $PATH, $MANPATH, and $LD_LIBRARY_LOAD.
To check what modules are loaded just use the “list” option.
home8:~> module list
Currently Loaded Modulefiles:
1) compiler/pgi5.2-x86_64
To unload a module just use the “unload” option but you have to specify the complete name of the environment module.
home8:~> module unload compiler/pgi5.2-x86_64
home8:~> module list
No Modulefiles Currently Loaded.
Alternatively, to you can unload all loaded environment modules using the “purge” option.
home8:~> module load compiler/pgi5.2-x86_64
home8:~> module list
Currently Loaded Modulefiles:
1) compiler/pgi5.2-x86_64
home8:~> module purge
home8:~> module list
No Modulefiles Currently Loaded.
The command is “module purge” to unload all loaded modules. You can see above that after the “module purge” command, there are no more environment modules loaded.
Typically you will want to load a compiler and an MPI library such as the following:
home8:~> module load compiler/pgi5.2-x86_64
home8:~> module load mpi/mpich-1.2.6
home8:~> module list
Currently Loaded Modulefiles:
1) compiler/pgi5.2-x86_64 2) mpi/mpich-1.2.6
home8:~>
Notice that in the list of modules you don't see a MPI module for each version of the compiler. Environment Modules is smart enough to know which version of the MPI libraries to load based on what version of the compiler is loaded.
To run a job you need to load the modules in the job script. Typically, after the part of the script where you request resources (in the PBS world these are defined as #PBS commands), you will then load the environment modules you need.
These examples are generic and you may not have any modules defined. So let’s finish by looking at some of comments in the Platform readme file about the environment modules roll in Platform OCS 4.x.
Using Environment Modules with Platform OCS
From the readme file, the environment module roll defines some common HPC environments. In particular:
The documentation gives a couple of quick examples:
% module load hpc/mpich-ethernet-gnu
% which mpirun
/opt/mpich/gnu/bin/mpirun
The “which” command tells where the particular command is located. In this case, the mpirun executable points to the one it /opt/mpich/gnu/bin which is the correct one.
To unload the environment module, you just use the “unload” option.
% module unload hpc/mpich-ethernet-gnu
% which mpirun
This shows that this particular module has been unloaded and that you cannot find the mpirun binary any longer (Note that /opt/mpich is not in the standard path so you should not be able to find the executable unless you have modified the default path for the cluster).
Summary
I consider Environment Modules to be one of the essential tools for clusters. They allow you to easily manage multiple compiler/MPI combinations and even applications. A few simple commands and you can modify your environment for your needs.
I highly encourage you to take a look at the website and associated links then take environment modules for a whirl. You won't regret and you will wonder how life existed prior to it.
Jeff
Scratching the Multi-MPI, Multi-Compiler Itch
When people first start using clusters they tend to stick with whatever compiler and MPI library came with the cluster when it was installed. As they become more comfortable with the cluster, using the compilers, using the MPI libraries, they start to look around at other options. Are there other compilers that could perhaps improve performance? Similarly, they may start looking at other MPI libraries – can they help improve performance? Do other MPI libaries have tools that can make things easier? Perhaps even more importantly, these people would like to install the next version of the compilers or MPI libraries so they can test them with their code(s). So this begs the question, how do you have multiple compilers and multiple MPI libraries on the cluster at the same time and not get them confused? I’m glad you asked.
Caveman Approach
I don’t want to get the cavemen angry with me, but the “uneducated” approach to using multiple compilers and MPI libraries on a cluster is, shall we say, kludgy. The basic idea is to change your $PATH in the .bashrc (if you are using bash), and then log out and log back in whenever you need to change your compiler/MPI combination. Initially this sounds like a pain and it is, but it works – to some degree. It doesn’t work in the situation where you want to run multiple jobs each with a different compiler/MPI combination.
For example, let’s say I have a job using the gcc 4.2.1compilers4.2.1 compilers using Open MPI. Then I have a job using gcc 4.2.1 and MPICH2. If I have both jobs in the queue at the same time, how can I control my .bashrc to make sure each job has the correct $PATH? The only way to do this is to restrict myself to one job in the queue at a time. When it’s finished I can then change my .bashrc and submit a new job. Even this is not the best approach because if you are using a different compiler/MPI combination from what is in the queue for something as simple as code development, you have to watch when the job is run to make sure your .bashrc matches your job.
Cro-Magnon Man Approach – Environment Modules
A much better way to handle compiler/MPI combinations is to use Environment Modules (Be careful not to confuse “environment modules” with “kernel modules”). According to the website, “The Environment Modules package provides for the dynamic modification of a user’s environment via modulefiles.” While this may not sound earth shattering, it actually is a quantum leap for using multiple compilers/MPI libraries. But you can use it for more than just compiler/MPI combinations, which I will talk about later.
You can use environment modules to alter or change environment variables such as $PATH, $MANPATH, $LD_LIBRARY_LOAD, and others. Since most job scripts for resource managers such as LSF , PBS-Pro, MOAB, are really shell scripts, you can incorporate environment modules into the scripts to set the appropriate $PATH your compiler/MPI combination needs or any other environment variables your application(s) require for operation.
Installing Environment Modules
How you install environment modules depends upon how your cluster is built. For those that use Platform OCS 4.x Platform has created a roll. Here is a readme describing the various rolls Platform has created. The roll should be included on the OCS DVD that came with the cluster.
If you didn’t use Platform OCS, then you should look around for something like an rpm for environment modules (assuming you used an rpm based cluster toolkit). Remember that Google is your friend when looking for rpm’s. If you can’t find an appropriate binary containing environment modules for your system, then you will have to build it from scratch. Fortunately, the instructions on the website are fairly good and you shouldn’t have too much trouble.
Some Examples for Using Environment Modules
I won’t discuss how to create or define environment modules. The website does a far better job than I in explaining how to do this. But if there is enough demand, I may do a future blog that discusses how to create modules for new compilers/MPI libraries/applications. So let’s assume that the environment modules are installed and functioning.
The first thing to check is what modules are available to you by using the “module avail” command:
home8:~> module avail
-------------------------- /usr/share/modules/modulefiles --------------------------
compiler/gcc dot null
compiler/pgi5.2 module-cvs pbs
compiler/pgi5.2-x86_64 module-info use.own
compiler/pgi6.0 modules
compiler/pgi6.0-x86_64 mpi/mpich-1.2.6
home8:~>
This commands lists what environment modules are installed. In this cases there are a number of compilers installed – gcc, pgi 5.2, pgi 5.2-x86_64, pgi 6.0, pgi 6.0-x86_64, as well as a single MPI library – mpich-1.2.6.
Let’s “load” one of the environment modules:
home8:~> module load compiler/pgi5.2-x86_64
You can just cut and past from the list of available modules to load the ones you want or need (this is what I do and it makes things easier). By loading a module you will have just changed the environmental variables defined for that module. Typically this is $PATH, $MANPATH, and $LD_LIBRARY_LOAD.
To check what modules are loaded just use the “list” option.
home8:~> module list
Currently Loaded Modulefiles:
1) compiler/pgi5.2-x86_64
To unload a module just use the “unload” option but you have to specify the complete name of the environment module.
home8:~> module unload compiler/pgi5.2-x86_64
home8:~> module list
No Modulefiles Currently Loaded.
Alternatively, to you can unload all loaded environment modules using the “purge” option.
home8:~> module load compiler/pgi5.2-x86_64
home8:~> module list
Currently Loaded Modulefiles:
1) compiler/pgi5.2-x86_64
home8:~> module purge
home8:~> module list
No Modulefiles Currently Loaded.
The command is “module purge” to unload all loaded modules. You can see above that after the “module purge” command, there are no more environment modules loaded.
Typically you will want to load a compiler and an MPI library such as the following:
home8:~> module load compiler/pgi5.2-x86_64
home8:~> module load mpi/mpich-1.2.6
home8:~> module list
Currently Loaded Modulefiles:
1) compiler/pgi5.2-x86_64 2) mpi/mpich-1.2.6
home8:~>
Notice that in the list of modules you don't see a MPI module for each version of the compiler. Environment Modules is smart enough to know which version of the MPI libraries to load based on what version of the compiler is loaded.
To run a job you need to load the modules in the job script. Typically, after the part of the script where you request resources (in the PBS world these are defined as #PBS commands), you will then load the environment modules you need.
These examples are generic and you may not have any modules defined. So let’s finish by looking at some of comments in the Platform readme file about the environment modules roll in Platform OCS 4.x.
Using Environment Modules with Platform OCS
From the readme file, the environment module roll defines some common HPC environments. In particular:
- MPICH (GNU, Intel compilers)
- MPICH-GM
- LAM (GNU, Intel compilers)
- MVAPICH (with the Topspin IB roll)
- Open MPI
The documentation gives a couple of quick examples:
% module load hpc/mpich-ethernet-gnu
% which mpirun
/opt/mpich/gnu/bin/mpirun
The “which” command tells where the particular command is located. In this case, the mpirun executable points to the one it /opt/mpich/gnu/bin which is the correct one.
To unload the environment module, you just use the “unload” option.
% module unload hpc/mpich-ethernet-gnu
% which mpirun
This shows that this particular module has been unloaded and that you cannot find the mpirun binary any longer (Note that /opt/mpich is not in the standard path so you should not be able to find the executable unless you have modified the default path for the cluster).
Summary
I consider Environment Modules to be one of the essential tools for clusters. They allow you to easily manage multiple compiler/MPI combinations and even applications. A few simple commands and you can modify your environment for your needs.
I highly encourage you to take a look at the website and associated links then take environment modules for a whirl. You won't regret and you will wonder how life existed prior to it.
Jeff
