Performance optimization with ScaLAPACK

[sere]

Dear all,

I am trying to use ScaLAPACK to improve the performance of the SCF procedure but the wall times with and without ScaLAPACK are nearly the same. To my understanding, the usage of the .SCALAPACK keyword should generally speed up matrix operations, provided LSDalton was linked to the appropriate MKL libraries. Is that correct or are there cases in your experience where you would advise against using .SCALAPACK?

I noticed that when using .SCALAPACK, the following message gets repeatedly printed to stdout during the SCF procedure:

Code: Select all

FALLBACK: mat_write_to_disk

I assume there is not enough memory, so the matrices are stored on disk which would hinder the performance. So now my question is: Is my interpretation of the above message correct and is this normal behavior when using .SCALAPACK or is there a way to minimize disk I/O and actually speed up the calculation?

Some technical details:

I compiled LSDalton 2018.0 with the Intel 2018 compilers and the following setup command:

Code: Select all

./setup --mpi --omp --csr --scalapack --extra-fc-flags="-O3 -xCORE-AVX2" --extra-cc-flags="-O3 -xCORE-AVX2" --extra-cxx-flags="-xHost -O3 -xCORE-AVX2"

The calculations ran on 28 cores with 4 MPI processes using 7 OMP threads each on a Haswell node cluster. There is about 2GB of memory per core, so the physically available memory should not be a limitation. Output files for sample HF/6-31G* calculations of a DNA base pair are attached.

I greatly appreciate any suggestions and I am happy to provide further information if needed.

Sebastian