 Re: memory increasing as MD steps increase and the version issue on it ( No.1 ) |
- Date: 2021/07/20 11:02
- Name: T. Ozaki
- Hi,
Thank you for your detailed report.
I will try to see what's happening, and post my report once the problem is figured out.
Regards,
TO
|
| Re: memory increasing as MD steps increase and the version issue on it ( No.2 ) |
- Date: 2021/07/24 17:51
- Name: T. Ozaki
- Hi,
I have checked the memory usage of OpenMX in both v3.8.5 and v3.9.2 with the input file you provided
by monitoring RSS as shown at
https://t-ozaki.issp.u-tokyo.ac.jp/calcs/monitoring_memory.PNG
Both the calculations were performed using 20 MPI cores, and the sum of the memory usage required by 20 MPI cores
is shown in the figure. As we can see in the figure, the memory usage of v3.9.2 seems to be comparable to that of v3.8.5.
So, from the comparison it seems to be difficult to say that any specific memory leak exists in v3.9.2.
Howvever, I have noticed from a series of benchmarks that an unexpected memory leak seems to occur when pdgemm or pzgemm is called in some environment.
Note that only v3.9.2 calls pdgemm or pzgemm in the eigenvalue solver.
I am not sure whether what I found is related to the issue mentioned at
https://software.intel.com/content/www/us/en/develop/documentation/onemkl-linux-developer-guide/top/managing-performance-and-memory/using-memory-functions/avoiding-memory-leaks-in-onemkl.html
If so, the issue may depend on the version of MKL.
I will keep the issue in mind.
Anyway, thank you very much for your detailed resport.
Regards,
TO
 |
| Re: memory increasing as MD steps increase and the version issue on it ( No.3 ) |
- Date: 2021/07/27 15:11
- Name: Kunihiro Yananose <ykunihiro@snu.ac.kr>
- Dear prof. Ozaki,
Thank you for the test and a kind reply.
I compiled openmx with the intel oneAPI MKL. So the MKL version might be related to this issue as you remarked.
I tried to apply the solutions in the link and did some tests. However, I couldn't see any visible improvement.
Specifically,
(1) variable setting at the job submit bash script by
export MKL_DISABLE_FAST_MM=1
(2) Recompile with the code modification of openmx.c by adding
at line 73,
#include <mkl.h>
void mkl_free_buffers(void);
void mkl_thread_free_buffers(void);
and at line 677,
mkl_free_buffers();
mkl_thread_free_buffers();
I don't know whether it is a correct way or not.
Because this is not an urgent issue to me, I'll also put off this issue for the present.
Thank you so much,
K. Yananose
|
| Re: memory increasing as MD steps increase and the version issue on it ( No.4 ) |
- Date: 2021/12/10 18:28
- Name: Kunihiro Yananose <ykunihiro@snu.ac.kr>
- Dear all,
Recently, I revisited this issue and made some progress. So I'd like to share what I observed. I found that this is not an issue in the openmx code, but the packages and mpi wrapper are related.
The same tests of the memory usage but with different compilations have been done. In the new tests, I used only the 3.9.9 version.
First, let's take a reference from my first post, ver 3.9.2 with RF, as the case (1).
(1) intel mpi + intel mkl : 13.5 % to 46.85 %
- In this case, both mpi and mkl are from the intel oneAPI 2021 version.
The following two cases are results of new compilations.
(2) intel mpi + netlib scalapack compiled with intel mpi wrapper : 13.1 % to 26.85 %
- The memory usage was reduced. But it was still increasing at every md steps. Computation speed is almost the same as (1).
(3) gcc compiler + openmpi + netlib scalapack : 10.4 % to 12.25 %
- Much less memory was used. The memory usage was saturated nearly at the 14th md step. But the speed is slow: openmx runtest takes almost twice as intel mpi case.
These results strongly suggest that the intel-mpi causes memory leaking.
In addition, I found that the SIESTA community recently has discussed a similar issue (https://gitlab.com/siesta-project/siesta/-/issues/29).
They suspect that the intel-mpi 2019 or later versions cause memory leaking. But I couldn't check different intel-mpi versions with openmx.
To me, using (2) for a usual case and (3) for heavy memory consuming case will be a solution. I guess it will be the same in the SCF-only calculations.
If you encounter a similar issue, I suggest trying to compile openmx with openmpi or intel-mpi 2018 version.
Sincerely,
K. Yananose
----------------------------------------------------------------------------------
Appendix
- my compile options for (2)
FFTW = -I/(my path to lib)/fftw/include
CC = mpiicc -qopenmp -DscaLAPACK -O3 -xHOST -ip -no-prec-div $(FFTW)
FC = mpiifort -qopenmp -DscaLAPACK -O3 -xHOST -ip -no-prec-div $(FFTW)
LIB= -L/(my path to lib)/fftw/lib -lfftw3 \
-L/(my path to lib)/scalapack-2.1.0 -lscalapack \
-L/(my path to lib)/lapack-3.10.0 -llapack -lrefblas \
-L/(my path to intel oneAPI)/mpi/latest/lib -lmpi -lifcore -liomp5 -lpthread -lm -ldl \
- my compile options for (3)
FFTW = -I/(my path to gcc_lib)/fftw/include
CC = mpicc -DscaLAPACK -O3 -ffast-math -fopenmp $(FFTW)
FC = mpifort -DscaLAPACK -O3 -ffast-math -fopenmp $(FFTW)
LIB= -L/(my path to gcc_lib)/fftw/lib -lfftw3 \
-L/(my path to gcc_lib)/scalapack-2.1.0 -lscalapack \
-L/(my path to gcc_lib)/lapack-3.10.0 -llapack -lrefblas -lgfortran \
-L/(my path to gcc_lib)/openmpi/lib -lmpi -lmpi_mpifh -lgomp -lpthread -lm -ldl \
|
| Re: memory increasing as MD steps increase and the version issue on it ( No.5 ) |
- Date: 2022/01/11 16:19
- Name: Kunihiro Yananose <ykunihiro@snu.ac.kr>
- additional information:
I tested a new case,
(4) icc compiler (from oneAPI 2021) + openmpi + netlib scalapack : 9.15 % to 13.25 %
- the final memory consuming is a little bit larger than (3), but almost ignorable difference. It was increasing even after the 40th MD step, but not at every MD steps.
Its speed is much better than (3). Just a few seconds were took more than intel-mpi&mkl case in the openmx runtest.
compile options are:
FFTW = -I/(my path to icc_lib)/fftw/include
CC = mpicc -qopenmp -DscaLAPACK -O3 -xHOST -ip -no-prec-div $(FFTW)
FC = mpifort -qopenmp -DscaLAPACK -O3 -xHOST -ip -no-prec-div $(FFTW)
LIB= -L/(my path to icc_lib)/fftw/lib -lfftw3 \
-L/(my path to icc_lib)/scalapack-2.1.0 -lscalapack \
-L/(my path to icc_lib)/lapack-3.10.0 -llapack -lrefblas \
-L/(my path to icc_lib)/openmpi/lib -lmpi -lmpi_mpifh -lmpi_usempif08 -lifcore -liomp5 -lpthread -lm -ldl
|
|
This thread is locked.Only browsing is available.
memory increasing as MD steps increase and the version issue on it