Errors While Running CTest

[alan988]

Dear Dalton,

I did manage to compile the parallel Dalton software in Opensuse13.1 (kernel version 3.11) using the following compilers:

1. Intel Compilers + MKL version 11.0
2. OpenMPI version 1.7.2
3. Cmake version 2.8.12

and the command used was

Code: Select all

$ ./setup --fc=ifort --cc=icc --cxx=icpc --mkl=parallel 

Everything went accordingly and the MKL flag was passed successfully.

However, when i went on to test the program using the following command:

Code: Select all

$ ctest -j8 -R dft -O test.log 

I noticed that all tests were failed and some of the errors which I received are as follow:

FATAL ERROR: 8 MPI nodes were requested, however
FATAL ERROR: mpirun command has not been defined, does not exist or is not executable
FATAL ERROR: echo $MPIRUN gives this string: " "
evaluate output file ./dft_pbe.log:
THERE IS A PROBLEM: no output was produced -- 1 tests failed out of the 1 tests run so far

May I know how should I tackle with the mpi problem in order to get the program running properly please?

I have attached the log file along with the post hoping that anyone can lend me a hand to solve my problem.

Thank you very much.

Best regards,
Alan

[lyzhao]

try to recompile dalton with ./setup --fc=ifort --cc=icc --cxx=icpc --mkl=parallel --mpi

[tkjaer]

you do not compile with MPI and as such it will not work with MPI.

TK

[bast]

hi,
i think you (Alan) did everything right (assuming you wanted a non-MPI binary)
but i suspect that you have perhaps also set
export DALTON_NUM_MPI_PROCS=8
is this the case? if yes, then i can explain you what happened.
otherwise i don't understand where the mpirun comes from.
radovan

[bast]

try to recompile dalton with ./setup --fc=ifort --cc=icc --cxx=icpc --mkl=parallel --mpi

the correct way to get a MPI binary is either

Code: Select all

$ ./setup --mpi [--mkl=parallel|sequential]

or

Code: Select all

$ ./setup --fc=mpif90 --cc=mpicc --cxx=mpicxx [--mkl=parallel|sequential]

both do exactly the same thing.

[alan988]

Dear lyzhao, tkjaer and bast,

Thank you very much for your kind assistance and guidance. I managed to initiate the compilation of DALTON using the suggested commands, but with some complications as described in the following:

When I invoked the first command:

Code: Select all

$ ./setup --fc=ifort --cc=icc --cxx=icpc --mkl=parallel --mpi 

everything works fine except that I received a few warnings:

WARNING: mpi.mod compiled with different compiler, will use mpif.h instead
WARNING: Cannot determine whether MPI is built for 32bit integers
...
Performing Test MPI_COMPATIBLE - Failed
Performing Test MPI_COMPILER_MATCHES - Failed
Performing Test MPI_ITYPE_MATCHES - Failed

I did not proceed with the installation but instead tried to rebuild the software using:

Code: Select all

$ ./setup --mpi --mkl=parallel 

The computer compiled the program using GNU compiler instead of Intel and everything passed successfully this time round without any warning.
Then I proceeded with make command inside the build directory. However, I noticed that the time taken for the compilation has increased dramatically compared to the first time when it was compiled using Intel compiler.
A routine check on the system resources showed that only 1 processor core was actively involved in the compilation process.

I wish to ask whether it is normal to observe this kind of situation (where only 1 processor core is involved instead of all) with mpi compilation?

In the middle of the compilation process, an error occurred with the following message:

...
[ 69%] Building Fortran object CMakeFiles/dalton.x.dir/DALTON/abacus/dalton.F.o
Linking Fortran executable dalton.x
gfortran: error: unrecognized command line option ‘-mkl=parallel’
make[2]: *** [dalton.x] Error 1
make[1]: *** [CMakeFiles/dalton.x.dir/all] Error 2
make: *** [all] Error 2

The compilation stopped abruptly at this point and the installation is not successful.

Would you please point out my mistake and show me the correct way to compile the program if it is possible please? Thank you very much.


Best regards,
Alan

[taylor]

You are running foul of several combinations of problems related to OpenMPI and the compiler stack.

First, the GNU compiler mob assert that their binary module format and calling conventions are exactly consistent with the Intel conventions, so one can e.g. link gfortran-compiled binaries against the Intel libraries. This may be so: I have never tried it (and have no plans to...). It means therefore that in principle it should be possible to compile with gfortran and then link against Intel's MKL libraries. The -mkl=parallel flag, it is important to note, will try to link against the cluster-parallel versions of the Intel compiler suite libraries. This means (as far as I can see from our own setup) that you \emph{must} have Intel Cluster Studio installed, not just Intel Composer version XXX.

Second, although the OpenMPI mob brag that by laborious effort they have made the OpenMPI libraries completely agnostic (that is, you can build OpenMPI with either the GNU compilers or the Intel compilers, and either build will link correctly whether you compile your user code with GNU or Intel) this is unfortunately only true for C and C++. The problem is that the internal format of a .mod file (that is, the result of compiling a source routine that contains Fortran modules) is completely different and non-interoperable between GNU and Intel. If you or your systems people built OpenMPI using the GNU compilers, you can compile C or C++ source with either gcc or icc and it will link correctly against the OpenMPI libraries. But you cannot use ifort in that case, you \emph{must} use gfortran, as you have observed. Contrariwise, if OpenMPI was built using icc, it will accommodate Fortran compilations with ifort, but not gfortran. (Fortran 90 compilations, I mean, if you don't use modules at all there is no problem, but Dalton does.)

At home and at VLSCI (my centre) we have separate builds of OpenMPI called (say) openmpi and openmpi-intel to deal with these cases, and we switch between them by loading and unloading "modules", by which I mean the module system within Linux --- the terminology is confusing here!

My own thinking is to make a decision about which path (no pun intended...) to walk down and stick to it. There is no doubt in my mind, thanks to extensive testing I've done, that if the Intel compiler/libraries are available the performance will be on a different page from the GNU stack (assuming here that the latter uses only the GNU-supplied compilers and libraries, although ATLAS is a possibility for optimized linear algebra routines). But either way, the starting point is to have a \emph{complete} software stack built one way or the other, and not to mix-and-match, or (what is happening here with your options) letting the build mix-and-match.

Finally, if you \emph{do} have Intel Cluster Studio, it should come with its own optimized MPI libraries, which means you don't need OpenMPI at all.

Best regards
Pete

[bast]

hi,

just a quick comment: the -mkl flag works to my knowledge only with Intel compilers.
you can use sequential or threaded MKL with GNU but then configure like this:

Code: Select all

$ export MATH_ROOT=/opt/intel/mkl   # adapt to your path if it deviates
$ ./setup --mpi

then in the output you get from setup/cmake look for BLAS and LAPACK. it should then point to MKL
(provided you have MKL and setup found it).

please also read this:
http://daltonprogram.org/installation/2013/math.html

the other thing is that the reason why setup --mpi suddenly took GNU is that setup --mpi
is an alias for setup --fc=mpif90 --cc=mpicc --cxx=mpicxx
and in your case mpif90 resolved to a GNU compiler:

Code: Select all

$ which mpif90

good luck!
radovan

[bast]

you also asked about slow compilation.
"make" compiles using 1 simultaneous job. does not matter whether
you compile an MPI binary or sequential one.
"make -jN" compiles using N simultaneous jobs. again does not matter
whether this is MPI or sequential.

[magnus]

YThe -mkl=parallel flag, it is important to note, will try to link against the cluster-parallel versions of the Intel compiler suite libraries. This means (as far as I can see from our own setup) that you \emph{must} have Intel Cluster Studio installed, not just Intel Composer version XXX.

The "-mkl=parallel" uses the threaded MKL while "-mkl=cluster" will use the cluster-parallel version.

[taylor]

Sorry about that, yes, confused it on my part. All that said, if Intel Cluster Studio is available, then the Intel MPI stack will be, and this should be used in preference to OpenMPI, I think.

Best regards
Pete