Setting things up in 

Param Pravega HPC for PLUTO

Last updated on 6th December 2022. 

At present, this is how we can set up our environment and use the runs efficiently

Setting up the SPACK environment

Param Pravega's webpage is a good place to start : https://www.serc.iisc.ac.in/supercomputer/for-traditional-hpc-simulations-param-pravega/

If you want to create your own environment, the following steps might be useful.


The spack environment in PARAM (use which spack to know your current SPACK_ROOT environment) needs to be loaded.

module load spack/0.17


Move to a scratch space where you want to set up your spack environment and clone from spack's git repository. Use your username at PARAM in place of <username> in the following commands.

cd /scratch/<username>/

git clone https://github.com/spack/spack.git


You can point your Spack installation to another installation, in order to use any packages that are installed there. To register the other Spack instance, you can add it as an entry to upstreams.yaml (located in spack/etc/spack/).

vi /scratch/<username>/spack/etc/spack/defaults/upstreams.yaml


Add the following text in the .yaml file.

upstreams:

  spack-instance-1:

    install_tree: /home-ext/apps/spack/opt/spack/

    modules:

          tcl: /home-ext/apps/spack/share/spack/modules


Find out the path  where the parent spack installation exists.

which spack

$ /home-ext/apps/spack/bin/spack

      Copy the compilers.yaml file to the spack/etc/spack folder. 

cp /home-ext/apps/spack/etc/spack/compilers.yaml /scratch/<username>/spack/etc/spack/


export SPACK_ROOT=/scratch/<username>/spack

. $SPACK_ROOT/share/spack/setup-env.sh

(NOTE: Don't forget the  .  in the beginning of above line)


Source your bashrc file or re-login.

Then, check if the path of your installation has been set as SPACK_ROOT using:

which spack

$ /scratch/<username>/spack/bin/spack


Here's a list of spack loads one can use for running PLUTO.

spack load intel-oneapi-compilers@2021.4.0

spack load hwloc@2.7.0%intel@2021.4.0

spack load zlib@1.2.11%intel@2021.4.0

spack load gsl@2.7%intel@2021.4.0

spack load gmp@6.2.1%intel@2021.4.0

spack load fftw@3.3.10%intel@2021.4.0/4s24afk

spack load hdf5@1.10.8%intel@2021.4.0/ssgjscn

spack load cmake@3.22.1/ozzmsd4

(NOTE: You can add these to your bashrc too, or do it on every login)



With recent updates on develop branch of spack, the paths with spack load are not added automatically to LD_LIBRARY_PATHS.

You will not see anything by echo $LD_LIBRARY_PATH if you spack loaded the packages.

But, during compiling, one would need the spack loaded package paths in LIBRARY_PATH. To do this use these commands once (you can unset the behavior using spack config remove later, if required).


spack config add modules:prefix_inspections:lib64:[LIBRARY_PATH]

spack config add modules:prefix_inspections:lib:[LIBRARY_PATH]


 Now, you would see the LD_LIBRARY_PATH to be set correctly. Add these to your ~/.bashrc now:

export LIBRARY_PATH=$LD_LIBRARY_PATH

export CPATH=/home-ext/apps/spack/opt/spack/linux-centos7-cascadelake/intel-2021.4.0/hdf5-1.10.8-ssgjscncg3xcw7ogodzjg3whjdmvx3bg/include:$CPATH

       (NOTE: the include directory is confirmed in the location given by which h5ls for its bin and then added. It's only important for people working              with  hdf5 data dumps)



Make sure that the file I/O of PLUTO is done in the scratch space (/scratch/<username>). Otherwise parallel I/O takes too much time!

    Voila! You are ready for the run!

Debugging a Test Problem

Quick run to check you have reasonable run times!

Using Catalyst with PLUTO

In-situ visualisation with ParaView Catalyst on your PLUTO runs at Param Pravega