Performance optimization with ScaLAPACK

Find answers or ask questions regarding LSDalton calculations
Post Reply
Posts: 1
Joined: 02 Feb 2019, 11:36
First name(s): Sebastian
Last name(s): Reiter
Affiliation: LMU Munich
Country: Germany

Performance optimization with ScaLAPACK

Post by sere » 04 Feb 2019, 11:44

Dear all,

I am trying to use ScaLAPACK to improve the performance of the SCF procedure but the wall times with and without ScaLAPACK are nearly the same. To my understanding, the usage of the .SCALAPACK keyword should generally speed up matrix operations, provided LSDalton was linked to the appropriate MKL libraries. Is that correct or are there cases in your experience where you would advise against using .SCALAPACK?

I noticed that when using .SCALAPACK, the following message gets repeatedly printed to stdout during the SCF procedure:

Code: Select all

FALLBACK: mat_write_to_disk
I assume there is not enough memory, so the matrices are stored on disk which would hinder the performance. So now my question is: Is my interpretation of the above message correct and is this normal behavior when using .SCALAPACK or is there a way to minimize disk I/O and actually speed up the calculation?

Some technical details:

I compiled LSDalton 2018.0 with the Intel 2018 compilers and the following setup command:

Code: Select all

./setup --mpi --omp --csr --scalapack --extra-fc-flags="-O3 -xCORE-AVX2" --extra-cc-flags="-O3 -xCORE-AVX2" --extra-cxx-flags="-xHost -O3 -xCORE-AVX2"
The calculations ran on 28 cores with 4 MPI processes using 7 OMP threads each on a Haswell node cluster. There is about 2GB of memory per core, so the physically available memory should not be a limitation. Output files for sample HF/6-31G* calculations of a DNA base pair are attached.

I greatly appreciate any suggestions and I am happy to provide further information if needed.

job output without scalapack
(74.46 KiB) Downloaded 20 times
stdout without scalapack
(6.35 KiB) Downloaded 20 times
job output with scalapack
(75.07 KiB) Downloaded 21 times
stdout with scalapack
(7.73 KiB) Downloaded 22 times

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest