Differences

This shows you the differences between two versions of the page.

--- software:himster2 [2019/09/11 08:55] – himuser
+++ software:himster2 [2022/01/14 18:44] (current) – external edit 127.0.0.1
@@ Line 1: / Line 1: @@
-=== SSH connection to the git repository ===
+=== Source code checkout ===
-Set up the ssh connection to ''bwec-fileserver'' through the HIM gateway following the instructions at https://www.hi-mainz.de/intern/service/ssh-gateway/
-\\
-=== Source code checkout ===
 <code>
 git clone https://gitlab.rlp.net/emp/primasoft.git
 </code>
-Alternatively (not preferred), check out from the fileserver
+Alternatively (not preferred), check out from the fileserver. \\
+Set up the ssh connection to ''bwec-fileserver'' through the HIM gateway following the instructions at https://www.hi-mainz.de/intern/service/ssh-gateway/ \\
+Then check out:
 <code>
 git clone agmaas@bwec-fileserver:/home/agmaas/git/primasoft.git
@@ Line 18: / Line 17: @@
 \\
 === Environment setup ===
-load module and set the variable FAIRSOFT_BUILD_DIR:
+load module and set the variable FAIRSOFT_BUILD_DIR:\\
+new since Nov 2020:
 <code>
-module load fairsoft/gcc/8.1.0/may18
+export MODULEPATH=$MODULEPATH:/cluster/him/modulefiles
-export FAIRSOFT_BUILD_DIR=/cluster/him/fairsoft/gcc_8.1.0/may18/
+module load fairsoft/gcc/8.3.0/may18p1
+export FAIRSOFT_BUILD_DIR=/cluster/him/fairsoft/gcc_8.3.0/may18p1/
 </code>
 or
 <code>
@@ Line 33: / Line 35: @@
 Follow the instruction on the [[software:install|installation page]].
+\\
+=== Send jobs to the queue system slurm ===
+All of the following information has been gathered by studying the ''mogonII'' documentation at  https://mogonwiki.zdv.uni-mainz.de/dokuwiki/. Refer to that page if something is unclear.
+In the following a minimum script to run a single job on slurm is given.
+To submit your job use
+<code>
+$ sbatch myjobscript
+</code>
+Example for myjobscript:
+<code>
+#!/bin/bash
+#-----------------------------------------------------------------
+# Example SLURM job script to run primag4sim on HimsterII
+#-----------------------------------------------------------------
+# First think about the resources you need and set the job requirements:
+#SBATCH -J g4simtest             # Job name
+#SBATCH -o g4simtest.%j.out      # Specify stdout output file (%j expands to jobId)
+#SBATCH -p himster2_exp          # Queue (partition) name
+#SBATCH -N 1                     # Total number of nodes requested (64 cores/node per Mogon I node)
+#SBATCH -n 1                     # Total number of tasks
+#SBATCH -t 00:30:00              # Run time (hh:mm:ss) - 0.5 hours
+#SBATCH -A m2_him_exp            # Specify allocation to charge against
+# A script to set the environment is sourced. It must contain all
+# environment settings needed to run the program, as if you would
+# run it just after login on himsterII. Something like
+#
+#  module load fairsoft/gcc/8.1.0/may18
+#  export FAIRSOFT_BUILD_DIR=...
+#
+# should be in there. Think about what you need.
+#
+source mysetenv.sh
+# Submit the executable with the needed arguments (it might also be a script)
+srun <executable> <arguments>
+</code>
+An example script is given here to run a train of jobs with different ''.conf'' files and ''run.mac'' macros.
+It is broken down in pieces with comments between them, deliberately to try and avoid unthoughtful copy-pasting. Things can be done in many different ways, this is just an example which should run and also produce  result output files, structured in a useful way.
+Provide the possibility to have a "dry runs" where all scripts are prepared and can be checked, without actually starting the jobs. That is very useful to avoid having to chase running jobs with wrong settings.
+The following code can be executed with:
+<code>
+$ ./myjobscript
+</code>
+<code>
+#!/bin/bash
+#-----------------------------------------------------------------
+# Script for running a train of primag4sim jobs on HimsterII
+#-----------------------------------------------------------------
+# set to 1 to really submit the jobs
+dorun=0
+</code>
+Data, logs and all used scripts will be saved with the output root files on the lustre filesystem, where there is some storage space. Create a personal directory at ''/lustre/miifs05/scratch/him-emp/'' to save your data.
+<code>
+# the path where scripts, confs, logs and output is stored
+outpath=/lustre/miifs05/scratch/him-emp/<...>
+</code>
+Base macro and config files have to be copied previously to the output directory and will be used as starting point for the job specific ones.
+The directory structure for all input and output files is constructed.
+<code>
+# base scripts to be completed for single jobs
+confbase=$outpath/test_base.conf
+macrobase=$outpath/run_base.mac
+# the simulation executable
+exec=$G4WORKDIR/bin/Linux-g++/primag4sim
+# make dirs
+mkdir -p $outpath/sbatch
+mkdir -p $outpath/logs
+mkdir -p $outpath/config
+mkdir -p $outpath/macros
+mkdir -p $outpath/rootfiles
+</code>
+The main loop on the jobs to be started (200 in this example) starts here. For each job config and macro files are completed, the job script is constructed and the job submitted if it is not a dry run.
+<code>
+for i in `seq 0 199`; do
+    istr=`printf %04d $i`
+    script=$outpath/sbatch/jobscript.$istr.sh
+    log=$outpath/logs/primasim.$istr.out
+    config=$outpath/config/primasim_$istr.conf
+    macro=$outpath/macros/run_$istr.mac
+    rootfile=$outpath/rootfiles/primaout_$istr.root
+    # prepare config file
+    cat $confbase > $config
+    echo "
+macro-file $macro
+root-output-file $rootfile" >> $config
+    # prepare macro file
+    echo "/random/setSeeds 17 $i" > $macro
+    cat $macrobase >> $macro
+    # prepare sbatch script
+    echo "#!/bin/bash" > $script
+    echo "
+#SBATCH -J run$istr
+#SBATCH -o $log
+#SBATCH -p himster2_exp
+#SBATCH -N 1
+#SBATCH -n 1
+#SBATCH -t 02:00:00
+#SBATCH -A m2_him_exp
+source $HOME/primasoft/mysetenv.sh
+srun $exec $config
+" >> $script
+    # submit the job
+    if [ $dorun -eq 1 ]; then sbatch $script;  fi
+done
+</code>