Running Waiwera benchmark tests

Benchmark tests

The /test/benchmark directory of the Waiwera source code contains a suite of benchmark test problems which can be used to verify that the code is working correctly. They can also serve as examples to help users become familiar with how Waiwera simulations are set up.

The Waiwera results for each benchmark test are compared with results generated using TOUGH2. For some tests, analytical or semi-analytical solutions are also available, or results using other simulators have previously been published, and these can also be used for comparison purposes.

The benchmark tests are organised into several sub-directories under the /test/benchmark directory, one for each category of test. These include:

  • MINC: benchmark tests with fractured rocks represented using the Multiple INTeracting Continua (MINC) approach (see Modelling fractured media using MINC)

  • Model Intercomparison Study: a selection of test problems from the 1980 “Geothermal Model Intercomparison Study”

  • NCG: tests for non-condensible gas equations of state (see Water / NCG EOS modules)

  • sources: tests for verifying correct behaviour of various source controls (see Source controls)

More details about some of these test problems can be found in various publications about Waiwera code development.

CREDO

The CREDO library is used to manage the Waiwera benchmark tests. CREDO is a Python toolkit for running, analysing and benchmarking computational models, originally developed as part of the Underworld geodynamics code project.

Waiwera uses a fork of the original CREDO library, modified to include modules specific to running and analysing Waiwera and TOUGH2 models.

CREDO can be installed from the Python Package Index (PyPI) using pip, e.g.:

pip install waiwera-credo

PyTOUGH

Running the Waiwera benchmark tests also requires PyTOUGH, a Python library for handling TOUGH2 simulations. This is needed for extracting the TOUGH2 results for each benchmark test, so they can be compared with the Waiwera results.

See the PyTOUGH website for instructions on how to install PyTOUGH.

Running the test suite

The whole suite of benchmark tests can be run using the Python script benchmark_tests.py, in the root directory of the Waiwera source code:

python benchmark_tests.py

By default, all tests will be run in serial. To run them in parallel, the script takes a -np parameter which can be used to specify the number of parallel processes, e.g.:

python benchmark_tests.py -np 4

would run the benchmark test suite using four processes.

Running using Docker

The above commands apply if running a native Linux executable (see Native Linux build). If you are running Waiwera via Docker (see Using Docker), you can use the benchmark_tests.py script using the --docker (or -d) parameter, e.g.:

python benchmark_tests.py --docker -np 2

This would run all the benchmark tests via Docker, using the waiwera-dkr script (see Running Waiwera using Docker), and using two parallel processes.

Note that if you are running Waiwera via Docker, using the waiwera-dkr script provided with PyWaiwera, you will probably not have a copy of the Waiwera source code on your machine. In that case, to run the benchmark tests you will first have to clone or download the Waiwera source code from its Github repository.

Test results

The result (pass or fail) for each benchmark test are written to the console, together with the overall result for the whole suite.

In addition, an HTML page with similar information is written to /test/benchmark/test_summary.html. This page can be viewed in a web browser, and contains links to detailed results for each benchmark test, including plots showing selected model results compared with results from other simulators (and/or analytical solutions).

Running individual tests

It is also possible to run individual benchmark tests, by navigating to the desired test directory and running the Python script for that test.

The test scripts all have filenames of the form test_*.py. Like the benchmark_tests.py script used for running the whole test suite, the individual test scripts also take the same -np and --docker (or -d) parameters for specifying the number of parallel processes and whether Docker is used to run the Waiwera models.

For example, in the /test/benchmark/minc/column directory, running:

python test_minc_column.py -np 2

runs the MINC column benchmark test on two processes.

The test results are summarised on the console. The HTML page with more detailed information about the test can be found in the test directory at /output/TESTNAME/TESTNAME-report.html, where TESTNAME is the name of the individual test.