2.2. First steps with Piernik code

2.2.1. System & compilation requirements

To succesfully compile the Piernik code one may need to install the following packages:

• SCM system: git to download the code,
• Fortran compiler: decent Fortran 2003 compiler (such as GNU ​>=gfortran-4.7 or Intel’s ​>=ifort-13.1 )
• MPI implementation with ROM-IO: any flavour, tested with OpenMPI, MPICH2, mvapich,
• HDF5 library: >=HDF5-1.8.8 - compiled with high-level library, Fortran 2003 interface and MPI support (–enable-shared –enable-fortran –enable-fortran2003 –enable-parallel); it allows you to use a wrapper ‘h5pfc’ as a compiler,
• python 2.7

Optionally you may also need:

• FFT library: >=FFTW3-3.0 - optional (for selfgravity),
• Lapack library: Lapack - optional (for selfgravity),
• visualization: yt , matplotlib or IDL.

2.2.2. Code download

To download PIERNIK directly from git repo type:

git clone https://github.com/piernik-dev/piernik.git

In the code main directory one can find:

• bin

  a directory of scripts and tools

• compilers

  a directory of compiler and library settings files

• doc

  a directory of doxygen documentation (will vanish soon)

• docs

  a directory of Sphinx documentation (will appear soon)

• Makefile

  a script

• make_test.py

  a python script

• problems

  a directory of simulation cases

• python

  a direcotry of python scripts

• setup

  a link to setup script

• src

  the main Piernik source directory containing base, grid, fluids, IO and several extensions modules

2.2.3. Code configuration

To prepare the code to use you will have to configure a compiler and libraries (see requirements) In compilers directory several configuration files exist. You can choose one of them or compose another that will be dedicated to your machine. A new configuration file has to be placed in the compilers directory and to have the .in extension. It is important to set the following flags:

• PROG - the name of the compiled program which is piernik. Yet you can change it for instance to myLOVLIESTandBESTcode if you want
• F90 - path to (if it is not set in the environment) and filename of the Fortran90 compiler
• F90FLAGS - default Fortran90 compiler flags
• LDFLAGS - default linker flags

In one configuration file you may specify 2 variants of settings: a normal case and a debug case. These variants may be governed by PIERNIK_DEBUG variable which is set during code compilation.

As an example of configuration file:

PROG      = piernik

### gcc-4.7
F90       = mpif90
F90FLAGS  = -fdefault-real-8 -ffree-form -std=gnu -fimplicit-none -ffree-line-length-none -g
LDFLAGS   = -Wl,--as-needed -Wl,-O1

ifeq ($(PIERNIK_DEBUG), 1)
   F90FLAGS += -ggdb -O0 -ffpe-trap=zero,overflow,invalid -fcheck=all -fno-omit-frame-pointer
   F90FLAGS += -Wall -W -Wextra -pedantic-errors -Wno-unused-function
else
   F90FLAGS += -O2 -fstack-arrays
endif

If there were compiler flags (F90FLAGS) that are not default for your configuration and you want to use them only sometimes, it is possible to pass them through the command line in the next step (see Compilation and Command line option arguments):

> setup -c <mysettings> --f90flags=F90FLAGS <FILES>

where <mysettings> is the name of a prepared compiler configuration file without the .in extension (e.g. compilers/<mysettings>.in), and F90FLAGS is an additional compiler flag to pass.

Note

To make things easier during compilation calls in the compilers directory default.in link should exist and be set to your configuration file.

2.2.4. Simulation problem case

In the directory problems simulation cases are stored. You can list existing within your code copy simulations problem cases typing the following command in the code main direcotry:

./setup --problems

That will cause listing of names and descriptions. In each problem directory several files must exist:

• info - contains a name and a description to be listed by setup --problems command
• initproblem.F90 - the main problem module
• piernik.def - a configure file for the problem; depanding on this file proper modules of the code are taken into account to be compiled
• problem.par - a parameter list file; it is read during the execution of the code (see: Input file problem.par)

There might be placed also other additional files, usage of each is optional:

• <extra_module1>.F90, <extra_module2>.F90,.. - extra modules, that are always compiled if the file extension is .F90
• problem.par.<version> - different versions of parameter list file (to use it instead of problem.par type: setup --param=problem.par.<version> FILES; see: Command line option arguments)
• user_rules.h - a macro for precompiler with problem-specific dependencies to be used across the code (it is switched on by USER_RULES precompiler declaration
• <any_file>.h - files with the .h extension to be included into problem-specific modules.

2.2.5. Compilation

To compile the configured code with a chosen problem case use the setup script in the code main directory:

./setup <problem_name>

It will cause choosing proper modules and files to be copied into obj directory and compile them using the settings chosen during Code configuration step. For non-default configuration you may use several options (see: Command line option arguments):

./setup --linkexe -c <nondefconfig> -o ala <problem_name>

This will cause choosing code modules proper to the case from problems/<problem_name> directory, linking them (not copying) to the obj_ala directory and compiling them using a configuration from compilers/<nondefconfig>.in file.

Note

You can use make command in the code main directory to recompile the code. See: MAKE calls.

After the compilation an executable piernik and parameters file problem.par are copied to the automatically created directory run/<problem_name>. You can verify the configuration of recently compiled code typing:

piernik --version

(see: Program arguments).

2.2.6. Command line option arguments

2.2.6.1. SETUP script arguments

Usage: setup [options] FILES

Options:

-h, --help

show this help message and exit

-v, --verbose

try to confuse the user with some diagnostics ;-)

-q, --laconic

compress stdout from make process

-n, --nocompile

Create object directory, check for circular dependencies, but do not compile or prepare run directory. In combination with –copy will prepare portable object directory.

--problems

print available problems and exit

-u, --units

print available units and exit

--last

call setup with last used options and arguments (require Pickles)

--copy

hard-copy source files instead of linking them

-l, --linkexe

do not copy obj/piernik to runs/ but link it instead

-p FILE, --param=FILE

use FILE instead problem.par

-d CPPFLAGS, --define=CPPFLAGS

add precompiler directives, use comma-separated list

--f90flags=F90FLAGS

pass additional compiler flags to F90FLAGS

-c FILE, --compiler=FILE

choose specified config from compilers directory

-o POSTFIX, --obj=POSTFIX

use obj_POSTFIX directory instead of obj/ and runs/_POSTFIX rather than runs/

Note

Frequently used options (like --linkexe, --laconic or -c ) can be stored in .setuprc and .setuprc.${HOSTNAME} files

2.2.6.2. MAKE calls

Makefile can be used to rebuild one or more object directories at a time.

Usage:

make

rebuilds all obj*/piernik executables calling make for all obj* directories

make obj_abc obj_xyz

rebuilds just the two specified directories

make resetup

calls setup for all obj* directories, relying on obj*/.setup.call and current .setuprc file

RS=1 make obj_abc

calls setup only for obj_abc directory

make obj_abc RS=1

same as above

make clean

calls make clean for all obj* directories

make obj_abc CL=1

calls make clean for specified directories

make allsetup

creates object directories for all valid problems, but does not compile them

make ctags

recreate ctags for {src,problems}

make dep [P=problem]

create and show dependency graph $P defaults to mcrwind

Note

Resetup will also call make for the object directories, unless you’ve specified --nocompile either in your .setuprc* files or it was stored in .setup.call file.

2.2.6.3. Program arguments

usage: piernik [OPTIONS]

Without further options, piernik prints the date and exits.

piernik options:

-v, --version

print version information and exit

-n, --namelist

read namelist from command line

-p, --param

path to the directory with problem.par and/or restarts (default: ”.”)

-w, --write

path to the directory where output will be written (default: ”.”)

-h, --help

print usage information and exit

-t, --time

print time and exit

Note

Program name piernik may be changed with PROG parameter during Code configuration.

2.2.7. Input file problem.par

After compiling the code using setup command (but not with make in obj directory) an executable program piernik is copied to run/<problem_name> directory (created or overwritten). Before running a simulation a user should check and, if necessary, update runtime parameters in the **problem.par file. It is a file of parameters divided into namelists (sets). Namelists correspond to modules of the code or functionalities.

$BASE_DOMAIN
   n_d = 100, 100, 40
   nb  = 4
   xmin   = -100.0
   xmax   =  100.0
   ymin   = -100.0
   ymax   =  100.0
   zmin   =  -40.0
   zmax   =   40.0
/

$AMR
/

Note

It is important to have all namelists, that are used in compiled modules, even empty, but present in problem.par file to be read and check during the simulation.

To check parameters meaning look at its descriptions in Runtime Parameters Descriptions or use its Namelist parameters index.

2.2.8. Running the code

Warning

The run/<problem_name> directory with its contant should be moved (or copied) to an appropriate place on a machine (designed scratch) due to potentially big data produced during simulations.

To run piernik code go to the directory you have already moved compiled source and runtime parameters file problem.par. To start the simulation as one process on 1 CPU type:

piernik

or to start multiprocess simulation (see: Parallel Runs (+)):

mpiexec <options> piernik

The name of the program (by default piernik) may be changed if you have editted the PROG variable in the used configuration file.

Note

Commonly on large computing machines and computing clusters users are are obliged to use job schedulers. In such a case just use commands described above in a script for a job scheduler.

Immediately, extracts to stdout/stderr will appear and you will be able to check whether all was intended. You also may look into the log file that will appear in the problem run directory. The name of the log file is given by two parameters from problem.par file from OUTPUT_CONTROL namelist and a number of restart: <problem_name>_<run_id>_<xxx>.log.

2.2.9. Interrupting a code run

During a simulation a user may control and govern it. The interface for this is a special file that should be placed in the simulation directory and be named as the user_message_file parameter in problem.par file (default msg). Command in this file should be from the following list:

res, dump

dump restart file (see: Types of output (+)) at the end of the current simulation step

hdf

dump regular IO file (see: Types of output (+)) at the end of the current simulation step

log

dump log file during the current simulation step

tsl

dump timeslice IO file (see: Types of output (+)) during current simulation step

wend <value>

update wend parameter with the <value>

wdtres <value>

update wdt_res parameter with the <value>

wleft

show how much walltime is left

wresleft

show how much walltime is left till next restart

tend <value>

update tend parameter with the <value>

nend <value>

update nend parameter with the <value>

dtres <value>

update dt_res parameter with the <value>

dthdf <value>

update dt_hdf parameter with the <value>

dtlog <value>

update dt_log parameter with the <value>

dttsl <value>

update dt_tsl parameter with the <value>

sleep <number>

wait <number> seconds

stop

finish the simulation in the current simulation step

help

print help info to stdout

Note

Note that only one line at a time is read from the user message file.

This interface is available to be used in automatic way and command from the system on the machine (e.g. related to UPS warnings) can be read during a simulation. To use this user has to prepare the software (UPS) on the computing machine to produce proper files in proper locations and set the path to possible file in problem.par file via system_message_file parameter. By default the path is set to /tmp/piernik_msg.

2.2.10. Restarting the code

During a simulation run checkpoint files (see more about restart files Types of output (+)) may be dumped. This should be scheduled by user via dtres parameter or be ordered via user or system message file (see: Interrupting a code run). Restart files are made up to contain all data necessary to restore the simulation from the file dump point. If a simulation run has been stopped for any reason a user may restart it specifying the restart file via parameters in RESTART_CONTROL namelist in problem.par file:

• nrestart - counter of a restart file to be used,
• restart - if 'last' then a restart file of the largest counter is used and nrestart parameter is ignored.

When restart parameters are set a simulation can be restarted in the same way as a new simulation (see: Running the code).