.. Piernik MHD code documentation master file, created by sphinx-quickstart on Mon Aug 4 13:00:52 2014. You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. First steps with Piernik code ============================================ .. _requirements: System & compilation requirements --------------------------------- To succesfully compile the Piernik code one may need to install the following packages: * **SCM system**: `git `_ to download the code, * **Fortran compiler**: decent Fortran 2003 compiler (such as `GNU ​>=gfortran-4.7 `_ or `Intel's ​>=ifort-13.1 `_ ) * **MPI implementation** with **ROM-IO**: any flavour, tested with `OpenMPI `_, `MPICH2 `_, `mvapich `_, * **HDF5 library**: `>=HDF5-1.8.8 `_ - compiled with high-level library, Fortran 2003 interface and MPI support (--enable-shared --enable-fortran **--enable-fortran2003** --enable-parallel); it allows you to use a wrapper 'h5pfc' as a compiler, * **python 2.7** Optionally you may also need: * **FFT library**: `>=FFTW3-3.0 `_ - optional (for selfgravity), * **Lapack library**: `Lapack `_ - optional (for selfgravity), * **visualization**: `yt `_ , `matplotlib `_ or `IDL `_. Code download ------------- To download PIERNIK directly from git repo type: ``git clone https://github.com/piernik-dev/piernik.git`` In the code main directory one can find: * **bin** a directory of scripts and tools * **compilers** a directory of compiler and library settings files * **doc** a directory of doxygen documentation (will vanish soon) * **docs** a directory of Sphinx documentation * **Makefile** a script * **make_test.py** a python script * **problems** a directory of simulation cases * **python** a direcotry of python scripts * **setup** a link to setup script * **src** the main Piernik source directory containing base, grid, fluids, IO and several extensions modules .. _code_configuration: Code configuration ---------------------------------------------- To prepare the code to use you will have to configure a compiler and libraries (see :ref:`requirements `) In **compilers** directory several configuration files exist. You can choose one of them or compose another that will be dedicated to your machine. A new configuration file has to be placed in the **compilers** directory and to have the **.in** extension. It is important to set the following flags: * **PROG** - the name of the compiled program which is *piernik*. Yet you can change it for instance to *myLOVLIESTandBESTcode* if you want * **F90** - path to (if it is not set in the environment) and filename of the Fortran90 compiler * **F90FLAGS** - default Fortran90 compiler flags * **LDFLAGS** - default linker flags In one configuration file you may specify 2 variants of settings: a normal case and a debug case. These variants may be governed by ``PIERNIK_DEBUG`` variable which is set during code compilation. As an example of configuration file: .. literalinclude:: ../../../compilers/gnu47.in If there were compiler flags (F90FLAGS) that are not default for your configuration and you want to use them only sometimes, it is possible to pass them through the command line in the next step (see :ref:`compilation` and :ref:`commandlineargs`):: > setup -c --f90flags=F90FLAGS where ```` is the name of a prepared compiler configuration file without the **.in** extension (e.g. ``compilers/.in``), and ``F90FLAGS`` is an additional compiler flag to pass. .. note:: To make things easier during compilation calls in the **compilers** directory ``default.in`` link should exist and be set to your configuration file. Simulation problem case ----------------------- In the directory **problems** simulation cases are stored. You can list existing within your code copy simulations problem cases typing the following command in the code main direcotry:: ./setup --problems That will cause listing of names and descriptions. In each problem directory several files must exist: * **info** - contains a name and a description to be listed by ``setup --problems`` command * **initproblem.F90** - the main problem module * **piernik.def** - a configure file for the problem; depanding on this file proper modules of the code are taken into account to be compiled * **problem.par** - a parameter list file; it is read during the execution of the code (see: :ref:`inputfile_problempar`) There might be placed also other additional files, usage of each is optional: * **.F90**, **.F90**,.. - extra modules, that are always compiled if the file extension is **.F90** * **problem.par.** - different versions of parameter list file (to use it instead of **problem.par** type: ``setup --param=problem.par. FILES``; see: :ref:`commandlineargs`) * **user_rules.h** - a macro for precompiler with problem-specific dependencies to be used across the code (it is switched on by :ref:`USER_RULES ` precompiler declaration * **.h** - files with the **.h** extension to be included into problem-specific modules. .. _compilation: Compilation ------------ To compile the configured code with a chosen problem case use the ``setup`` script in the code main directory:: ./setup It will cause choosing proper modules and files to be copied into ``obj`` directory and compile them using the settings chosen during :ref:`code_configuration` step. For non-default configuration you may use several options (see: :ref:`commandlineargs`):: ./setup --linkexe -c -o ala This will cause choosing code modules proper to the case from ``problems/`` directory, linking them (not copying) to the ``obj_ala`` directory and compiling them using a configuration from ``compilers/.in`` file. .. note:: You can use ``make`` command in the code main directory to recompile the code. See: :ref:`make_calls`. After the compilation an executable ``piernik`` and parameters file ``problem.par`` are copied to the automatically created directory ``run/``. You can verify the configuration of recently compiled code typing:: piernik --version (see: :ref:`piernik_inline`). .. _commandlineargs: Command line option arguments ------------------------------ SETUP script arguments ~~~~~~~~~~~~~~~~~~~~~~ Usage: ``setup [options] FILES`` Options: ``-h``, ``--help`` show this help message and exit ``-v``, ``--verbose`` try to confuse the user with some diagnostics ;-) ``-q``, ``--laconic`` compress stdout from make process ``-n``, ``--nocompile`` Create object directory, check for circular dependencies, but do not compile or prepare run directory. In combination with --copy will prepare portable object directory. ``--problems`` print available problems and exit ``-u``, ``--units`` print available units and exit ``--last`` call setup with last used options and arguments (require Pickles) ``--copy`` hard-copy source files instead of linking them ``-l``, ``--linkexe`` do not copy obj/piernik to runs/ but link it instead ``-p FILE``, ``--param=FILE`` use FILE instead problem.par ``-d CPPFLAGS``, ``--define=CPPFLAGS`` add precompiler directives, use comma-separated list ``--f90flags=F90FLAGS`` pass additional compiler flags to F90FLAGS ``-c FILE``, ``--compiler=FILE`` choose specified config from compilers directory ``-o POSTFIX``, ``--obj=POSTFIX`` use obj_POSTFIX directory instead of obj/ and runs/_POSTFIX rather than runs/ .. note:: Frequently used options (like ``--linkexe``, ``--laconic`` or ``-c`` ) can be stored in .setuprc and .setuprc.${HOSTNAME} files .. _make_calls: MAKE calls ~~~~~~~~~~ Makefile can be used to rebuild one or more object directories at a time. Usage: ``make`` rebuilds all obj*/piernik executables calling make for all obj* directories ``make obj_abc obj_xyz`` rebuilds just the two specified directories ``make resetup`` calls setup for all obj* directories, relying on obj*/.setup.call and current .setuprc file ``RS=1 make obj_abc`` calls setup only for obj_abc directory ``make obj_abc RS=1`` same as above ``make clean`` calls make clean for all obj* directories ``make obj_abc CL=1`` calls make clean for specified directories ``make allsetup`` creates object directories for all valid problems, but does not compile them ``make ctags`` recreate ctags for {src,problems} ``make dep [P=problem]`` create and show dependency graph ``$P`` defaults to mcrwind .. note:: Resetup will also call make for the object directories, unless you've specified ``--nocompile`` either in your .setuprc* files or it was stored in .setup.call file. .. _piernik_inline: Program arguments ~~~~~~~~~~~~~~~~~ usage: ``piernik [OPTIONS]`` Without further options, piernik prints the date and exits. piernik options: ``-v``, ``--version`` print version information and exit ``-n``, ``--namelist`` read namelist from command line ``-p``, ``--param`` path to the directory with problem.par and/or restarts (default: ".") ``-w``, ``--write`` path to the directory where output will be written (default: ".") ``-h``, ``--help`` print usage information and exit ``-t``, ``--time`` print time and exit .. note:: Program name **piernik** may be changed with **PROG** parameter during :ref:`code_configuration`. .. _inputfile_problempar: Input file problem.par ---------------------- After compiling the code using ``setup`` command (but not with ``make`` in *obj* directory) an executable program **piernik** is copied to **run/ directory (created or overwritten). Before running a simulation a user should check and, if necessary, update runtime parameters in the **problem.par** file. It is a file of parameters divided into namelists (sets). Namelists correspond to modules of the code or functionalities. .. code-block:: fortran $BASE_DOMAIN n_d = 100, 100, 40 nb = 4 xmin = -100.0 xmax = 100.0 ymin = -100.0 ymax = 100.0 zmin = -40.0 zmax = 40.0 / $AMR / .. note:: It is important to have all namelists, that are used in compiled modules, even empty, but present in **problem.par** file to be read and check during the simulation. To check parameters meaning look at its descriptions in :doc:`namelists` or use its :doc:`/tocsrc/dictsrc/idnamelist`. .. _runningthecode: Running the code ---------------- .. warning:: The **run/** directory with its contant should be moved (or copied) to an appropriate place on a machine (designed scratch) due to potentially big data produced during simulations. To run piernik code go to the directory you have already moved compiled source and runtime parameters file **problem.par**. To start the simulation as one process on 1 CPU type:: piernik or to start multiprocess simulation (see: :doc:`parallelruns`):: mpiexec piernik The name of the program (by default **piernik**) may be changed if you have editted the **PROG** variable in the used :ref:`configuration file `. .. note:: Commonly on large computing machines and computing clusters users are are obliged to use job schedulers. In such a case just use commands described above in a script for a job scheduler. Immediately, extracts to *stdout*/*stderr* will appear and you will be able to check whether all was intended. You also may look into the log file that will appear in the problem run directory. The name of the log file is given by two parameters from **problem.par** file from :ref:`OUTPUT_CONTROL ` namelist and a number of restart: **__.log**. .. _runinterrupt: Interrupting a code run ----------------------- During a simulation a user may control and govern it. The interface for this is a special file that should be placed in the simulation directory and be named as the :ref:`user_message_file ` parameter in **problem.par** file (default ``msg``). Command in this file should be from the following list: **res**, **dump** dump restart file (see: :doc:`outputtypes`) at the end of the current simulation step **hdf** dump regular IO file (see: :doc:`outputtypes`) at the end of the current simulation step **log** dump log file during the current simulation step **tsl** dump timeslice IO file (see: :doc:`outputtypes`) during current simulation step **wend ** update :ref:`wend ` parameter with the ** **wdtres ** update :ref:`wdt_res ` parameter with the ** **wleft** show how much walltime is left **wresleft** show how much walltime is left till next restart **tend ** update :ref:`tend ` parameter with the ** **nend ** update :ref:`nend ` parameter with the ** **dtres ** update :ref:`dt_res ` parameter with the ** **dthdf ** update :ref:`dt_hdf ` parameter with the ** **dtlog ** update :ref:`dt_log ` parameter with the ** **dttsl ** update :ref:`dt_tsl ` parameter with the ** **sleep ** wait ** seconds **stop** finish the simulation in the current simulation step **help** print help info to *stdout* .. note:: Note that only one line at a time is read from the user message file. This interface is available to be used in automatic way and command from the system on the machine (e.g. related to UPS warnings) can be read during a simulation. To use this user has to prepare the software (UPS) on the computing machine to produce proper files in proper locations and set the path to possible file in *problem.par* file via :ref:`system_message_file ` parameter. By default the path is set to ``/tmp/piernik_msg``. Restarting the code ---------------------- During a simulation run checkpoint files (see more about restart files :doc:`outputtypes`) may be dumped. This should be scheduled by user via :ref:`dtres ` parameter or be ordered via user or system message file (see: :ref:`runinterrupt`). Restart files are made up to contain all data necessary to restore the simulation from the file dump point. If a simulation run has been stopped for any reason a user may restart it specifying the restart file via parameters in :ref:`RESTART_CONTROL ` namelist in **problem.par** file: * :ref:`nrestart ` - counter of a restart file to be used, * :ref:`restart ` - if ``'last'`` then a restart file of the largest counter is used and *nrestart* parameter is ignored. When restart parameters are set a simulation can be restarted in the same way as a new simulation (see: :ref:`runningthecode`).