=== Source code checkout === 

<code>
git clone https://gitlab.rlp.net/emp/primasoft.git
</code>

Alternatively (not preferred), check out from the fileserver. \\

Set up the ssh connection to ''bwec-fileserver'' through the HIM gateway following the instructions at https://www.hi-mainz.de/intern/service/ssh-gateway/ \\
Then check out:
<code>
git clone agmaas@bwec-fileserver:/home/agmaas/git/primasoft.git
</code>


\\
=== Environment setup ===
load module and set the variable FAIRSOFT_BUILD_DIR:\\
new since Nov 2020:
<code>
export MODULEPATH=$MODULEPATH:/cluster/him/modulefiles
module load fairsoft/gcc/8.3.0/may18p1
export FAIRSOFT_BUILD_DIR=/cluster/him/fairsoft/gcc_8.3.0/may18p1/
</code>

or
<code>
module load fairsoft/gcc/gcc_4.8.5/oct17p1
export FAIRSOFT_BUILD_DIR=/cluster/him/fairsoft/gcc_4.8.5/oct17p1
</code>

\\
=== Compile and build ===
Follow the instruction on the [[software:install|installation page]].

\\
=== Send jobs to the queue system slurm ===
All of the following information has been gathered by studying the ''mogonII'' documentation at  https://mogonwiki.zdv.uni-mainz.de/dokuwiki/. Refer to that page if something is unclear.

In the following a minimum script to run a single job on slurm is given.
To submit your job use 
<code>
$ sbatch myjobscript
</code>


Example for myjobscript:

<code>
#!/bin/bash
#-----------------------------------------------------------------
# Example SLURM job script to run primag4sim on HimsterII
#-----------------------------------------------------------------

# First think about the resources you need and set the job requirements:
 
#SBATCH -J g4simtest             # Job name
#SBATCH -o g4simtest.%j.out      # Specify stdout output file (%j expands to jobId)
#SBATCH -p himster2_exp          # Queue (partition) name
#SBATCH -N 1                     # Total number of nodes requested (64 cores/node per Mogon I node)
#SBATCH -n 1                     # Total number of tasks
#SBATCH -t 00:30:00              # Run time (hh:mm:ss) - 0.5 hours
 
#SBATCH -A m2_him_exp            # Specify allocation to charge against

# A script to set the environment is sourced. It must contain all 
# environment settings needed to run the program, as if you would 
# run it just after login on himsterII. Something like 
#
#  module load fairsoft/gcc/8.1.0/may18
#  export FAIRSOFT_BUILD_DIR=...
#
# should be in there. Think about what you need.
#

source mysetenv.sh
 
# Submit the executable with the needed arguments (it might also be a script)

srun <executable> <arguments>

</code>


An example script is given here to run a train of jobs with different ''.conf'' files and ''run.mac'' macros.
It is broken down in pieces with comments between them, deliberately to try and avoid unthoughtful copy-pasting. Things can be done in many different ways, this is just an example which should run and also produce  result output files, structured in a useful way.

Provide the possibility to have a "dry runs" where all scripts are prepared and can be checked, without actually starting the jobs. That is very useful to avoid having to chase running jobs with wrong settings.

The following code can be executed with:
<code>
$ ./myjobscript
</code>

<code>
#!/bin/bash
#-----------------------------------------------------------------
# Script for running a train of primag4sim jobs on HimsterII
#-----------------------------------------------------------------

# set to 1 to really submit the jobs
dorun=0
</code>

Data, logs and all used scripts will be saved with the output root files on the lustre filesystem, where there is some storage space. Create a personal directory at ''/lustre/miifs05/scratch/him-emp/'' to save your data.

<code>
# the path where scripts, confs, logs and output is stored 
outpath=/lustre/miifs05/scratch/him-emp/<...>
</code>


Base macro and config files have to be copied previously to the output directory and will be used as starting point for the job specific ones.
The directory structure for all input and output files is constructed.
<code>
# base scripts to be completed for single jobs
confbase=$outpath/test_base.conf
macrobase=$outpath/run_base.mac

# the simulation executable
exec=$G4WORKDIR/bin/Linux-g++/primag4sim


# make dirs 
mkdir -p $outpath/sbatch
mkdir -p $outpath/logs
mkdir -p $outpath/config
mkdir -p $outpath/macros
mkdir -p $outpath/rootfiles
</code>


The main loop on the jobs to be started (200 in this example) starts here. For each job config and macro files are completed, the job script is constructed and the job submitted if it is not a dry run.
<code>
for i in `seq 0 199`; do
    
    istr=`printf %04d $i`

    script=$outpath/sbatch/jobscript.$istr.sh
    log=$outpath/logs/primasim.$istr.out
    config=$outpath/config/primasim_$istr.conf
    macro=$outpath/macros/run_$istr.mac
    rootfile=$outpath/rootfiles/primaout_$istr.root

    # prepare config file
    cat $confbase > $config
    echo "
macro-file $macro
root-output-file $rootfile" >> $config

    # prepare macro file
    echo "/random/setSeeds 17 $i" > $macro
    cat $macrobase >> $macro
    
    # prepare sbatch script
    echo "#!/bin/bash" > $script
    echo "
#SBATCH -J run$istr
#SBATCH -o $log
#SBATCH -p himster2_exp
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -t 02:00:00
#SBATCH -A m2_him_exp
source $HOME/primasoft/mysetenv.sh
srun $exec $config
" >> $script

    # submit the job
    if [ $dorun -eq 1 ]; then sbatch $script;  fi

done

</code>