Those build recipe prototypes are siple bash scripts but they could be easily converted into Jenkins jobs:
Showing posts with label GROMACS. Show all posts
Showing posts with label GROMACS. Show all posts
Compiling GROMACS 2019.5 with Intel Compilers and CUDA on CentOS 7 and 8
All tests reported bellow are performed by using Intel Parallel Studio XE 2019 Update 4. I presume you may achieve the same positive results using older versions of the Intel Compilers package.
Notes on CentOS 7: You have to install the SCL repository and EPEL (that are MANDATORY requirements you cannot skip):
$ sudo yum install centos-release-scl centos-release-scl-rh epel-release
Then, using the new repositories, install dvetoolset-8 and cmake3:
$ sudo yum install devtoolset-8 cmake3
In the bash session, where the compilation process will take place, execute:
$ scl enable devtoolset-8 bash
To install the CUDA latest compiler and libraries, you might use the procedure described in my previous posts. The latest version of CUDA is 10.2. Once all the pre-requisites are satisfied, download the GROMACS 2019.5 source code tarball, unpack it, create a folder named "build" inside the source tree, enter it, load the Intel Compilers variables, then the SCL environment and run the compilation and installation process:
$ mkdir ~/build $ cd build $ wget ftp://ftp.gromacs.org/pub/gromacs/gromacs-2019.5.tar.gz $ tar xvf gromacs-2019.5.tar.gz $ cd gromacs-2019.5 $ mkdir build $ cd build $ source /opt/intel/compilers_and_libraries/linux/bin/compilervars.sh intel64 $ CC=icc CXX=icpc CFLAGS="-xHost" CXXFLAGS="-xHost" cmake3 .. -DCMAKE_INSTALL_PREFIX=/usr/local/appstack/gromacs-2019.5-icc -DGMX_MPI=OFF -DGMX_BUILD_OWN_FFTW=OFF -DGMX_GPU=ON -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda -DGMX_FFT_LIBRARY=mkl $ make -j6 install
The product of the compilation will be installed in /usr/local/appstack/gromacs-2019.5-icc. You might change that destination by set another value to -DCMAKE_INSTALL_PREFIX.
Compiling xdrfile library by using PGI C compiler (fast and dirty way)
Content:
2. Downloading the code of xdrfile library.
3. Compiling and installing the code.
1. Introduction.
The xdrfile library is a side-project of GROMACS and provides an easy to use universal interface for reading XTC and TRR trajectories from within python, C/C++, and Fortran code based applications.
The library code could be easily compiled by using GNU C and Intel C compilers, but the attempt to use PGI compilers collection through the configure script will fail. One of the reasons for that failure is that PGI compilers do not support invocation with GNU style parameters. Of course, that might be solved by changing the configure script and thus matching the PGI C compiler set of input parameter, but unless you wanna pack the compilation as RPM, DEB, or other package, it is not worth to do it. The code might be compiled directly from command line without any automation in seconds, and the goal of this document is to show how to.
2. Downloading the code of xdrfile library.
The source code tarball of xdrfile library could be downloaded from the download page of GROMACS project (scroll down the page - the xdrfile links are in the bottom of the page):
http://www.gromacs.org/Downloads
The current version is 1.1.4. Create a separate folder, download and unzip the tarball into it, and enter the folder:
$ mkdir ~/tmp/xdrfile
$ cd ~/tmp/xdrfile
$ wget ftp://ftp.gromacs.org/contrib/xdrfile-1.1.4.tar.gz
$ tar zxvf xdrfile-1.1.4.tar.gz
$ cd xdrfile-1.1.4/src
3. Compiling libxdrfile library and installing it locally.
Before start compiling the code one need to realize that both shared or static library might be needed build because some of the applications which will use the library might follow different model of compilation - some of them might need the shared version of the library, another ones might need the static version. So the example bellow shows how to compile both of them:
3.1. Compiling and installing locally the shared library libxdrfile.so.
The compilation of shared library (using position independent code (PIC) model):
$ cd ~/tmp/xdrfile/xdrfile-1.1.4/src
$ pgcc -fastsse -fPIC -c xdrfile_xtc.c xdrfile_trr.c xdrfile.c -I../include
$ pgcc -fastsse -fPIC -shared -o libxdrfile.so xdrfile.o xdrfile_trr.o xdrfile_xtc.o
$ sudo cp ~/tmp/xdrfile/xdrfile-1.1.4/src/libxdrfile.so /usr/local/lib
3.2. Creating and installing locally the static library libxdrfile.a.
To compile the static version of the library, create first the object files (as shown above) and use the GNU ar tool to pack them:
$ cd ~/tmp/xdrfile/xdrfile-1.1.4/src
$ pgcc -fastsse -fPIC -c xdrfile_xtc.c xdrfile_trr.c xdrfile.c -I../include
$ ar rcs libxdrfile.a xdrfile_xtc.o xdrfile_trr.o xdrfile.o
$ sudo cp ~/tmp/xdrfile/xdrfile-1.1.4/src/libxdrfile.a /usr/local/lib
Note that using the static version of the library is not recommended, unless you are very certain about what you want to achieve by compiling your code statically.
4. Testing the compilation.
The xdrfile library is supplied with two test tools - one written in C and another one - in Python. Because the test requires some TRR and XTC trajectories to exist, the C tool should be run first to generate them, after its successful execution, of course.
4.1. Testing the shared library libxdrfile.so and generating sample TRR and XTC trajectories by invoking the tool xdrfile_c_test.
To execute xdrfile_c_test we need to have it compiled from the C-source. The compilation with respect the shared library libxdrfile.so (for the example it is installed locally in /usr/local/bin) follows the recipe:
$ cd ~/tmp/xdrfile/xdrfile-1.1.4/src
$ pgcc -fastsse -o xdrfile_c_test xdrfile_c_test.c -L/usr/local/lib -lxdrfile -I../include
To check the if the shared library compiled before works as expected and generate sample TRR and XTC trajectories to test the tools on, execute xdrfile_c_test (presume the libxdrfile.so is in /usr/local/bin):
$ cd ~/tmp/xdrfile/xdrfile-1.1.4/src
$ export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
$ ./xdrfile_c_test
If the library libxdrfile.so is successfully compiled, loaded, and works as expected the following output will appear on the display:
Testing basic xdrfile library: PASSED
Testing xtc functionality: PASSED
Testing trr functionality: PASSED
and these new trajectory files will be created:
$ ~/tmp/xdrfile/xdrfile-1.1.4/src/test.trr
$ ~/tmp/xdrfile/xdrfile-1.1.4/src/test.xtc
4.2. Testing reading the generated sample TRR and XTC trajectories by using the Python tool xdrfile_test.py.
Execute the tool xdrfile_test.py (have Python 2.7, presume the libxdrfile.so is in /usr/local/bin):
$ cd ~/tmp/xdrfile/xdrfile-1.1.4/src/python
$ export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
$ ./xdrfile_test.py
If the execution is successful the displayed result will look like:
../test.trr OK
../test.xtc OK
4.3. Converting TRR trajectory into XTC by using trr2xtc tool.
Compile trr2xtc from its C-source (for the example the shared library libxdrfile.so is installed locally in /usr/local/bin):
$ cd ~/tmp/xdrfile/xdrfile-1.1.4/src/python
$ pgcc -fastsse -o trr2xtc trr2xtc.c -L/usr/local/lib -lxdrfile -I../include
Execute it to convert the sample TRR trajectory into XTC one (if libxdrfile.so is in /usr/local/bin):
$ cd ~/tmp/xdrfile/xdrfile-1.1.4/src/python
$ export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
$ ./trr2xtc -i test.trr -o converted.xtc
To be sure that the conversion is successful, compare the SHA256 checksums of the produced converted.xtc and the sample test.xtc (that will work in case test.xtc has not been modified after its creation):
$ cd ~/tmp/xdrfile/xdrfile-1.1.4/src/python
$ sha256sum converted.xtc
$ sha256sum test.xtc
They must match!
Compiling and installing GROMACS 2016 by using Intel C/C++ and Fortran compiler, and adding CUDA support
Content:
2. Setting the building environment.
3. Short notes on AVX-capable CPUs support available in GROMACS.
4. Downloading and installing CUDA.
5. Compiling and installing OpenMPI.
6. Compiling and installing GROMACS.
1. Introduction
GROMACS is an open source software for performing molecular dynamics sumulations. It also provides an excellent set of tools which can be used to analyze the results of the sumulations. GROMACS is fast and robust and its code supports a long range of run-time and compile-time optimizations. This document explains how to compile GROMACS 2016 on CentOS 7 and Scientific Linux 7 with CUDA support, by means of using Intel C/C++ and Fortran compiler.
Before starting with the compilation, be sure that you are awared of the following:
Do not use the latest version of GROMACS for production right after its official release, unless you are developer or just want to see what is new. Every software product based on such a huge amount of source code might contain some critical bugs at the beginning. Wait up for 1-2 weeks after the release date and then check carefully the GROMACS user-support forums. If you see no critical bugs reported (or minor bugs which might affect your simulations in particular) there you can compile the latest release of GROMACS. Even then test the build agains some known simulation results of yours. If you see no big differences (or see some expected ones) you can proceed with the implementation of the latest GROMACS release on your system for production.
If you administer HPC facility where the compute nodes are equipped with different processores, you most probably need to compile GROMACS code separately to match each CPU type features. To do so create build hosts for each CPU type by using nodes that matches that type. Compile there GROMACS and then clone the installation to the rest of the nodes of the same CPU type.
Always use the latest CUDA compatible to the particular GROMACS release (carefully check the GROMACS GPU documentation).
During the compilation of GROMACS always build its own FFTW library. That really boosts the productivity of GROMACS.
Compiling OpenMPI with Intel Compiler is not of critical importance (the system OpenMPI libraries provided by the Linux distributions could be employed instead), but it might rise up the productivity of the simulations. Also Intel C/C++ and Fortran compiler provides its native MPI support that could be used later, but having freshly compiled OpenMPI always helps you to be always up to date to the resent MPI development. Before starting the compilation be absolutely sure what libraries and compiler options do you need to successfully compile your custom OpenMPI and GROMACS!
Use the latest Intel C/C++ and Fortran Compiler if it is possible. That largerly guarantees that the specific processor flags of the GPU and CPU will be taken into account by the C/C++ and Fortran compilers during the compilation process.
2. Setting the building environment.
Before starting be sure you have your build folder created. You might need an unprivileged user to perform the compilation. Open this document to see how to do that:
https://vessokolev.blogspot.com/2016/08/speeding-up-your-scientific-python-code.html
See paragraphs 2, 3, and 4 there.
3. Short notes on AVX-capable CPUs support available in GROMACS.
If the result of the execution of cat /proc/cpuinfo shows avx2 CPU flag (coloured red bellow):
$ cat /proc/cpuinfo
...
processor : 23
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz
stepping : 2
microcode : 0x37
cpu MHz : 1221.156
cache size : 30720 KB
physical id : 0
siblings : 24
core id : 13
cpu cores : 12
apicid : 27
initial apicid : 27
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc
bogomips : 4589.54
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
then your CPU supports Intel® Advanced Vector Extensions 2 (Intel® AVX2). GROMACS supports AVX2 and that feature busts significantly the performance of the sumulations when compute bonded interactions. More on that CPU architecture features here:
https://software.intel.com/en-us/articles/how-intel-avx2-improves-performance-on-server-applications
4. Downloading and installing CUDA.
You need to install CUDA rpm packages on both build host and compute nodes. The easiest and most efficient way to do so and get updates later (when there are any available) is through yum. To install the NVidia CUDA yum repository file visit:
https://developer.nvidia.com/cuda-downloads
and download the repository rpm file, as illustrated in the picture shown bellow:
Alternative way to get the repository rpm file is to browse the NVidia CUDA repository directory at:
http://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64
scroll down there, and find and download the rpm file named "cuda-repo-rhel7-*" (select the recent one). Then install it locally by using yum localinstall
# yum localinstall /path/to/cuda-repo-rhel7-*.rpm
Once ready with the CUDA respository installation do become a root or super user and install the CUDA Toolkit rpm packages:
# yum install cuda
Note that the process of installation takes time, which mainly depends on both network connectivity and productivity of the local system. Also note that yum automatically installs (through dependencies in the rpm packages) DKMS to support the rebuilding of the NVidia kernel modules when booting a new kernel.
If you do not expect to use your compute nodes for compiling any code with CUDA support and only execute compiled binary code there, you might no need to install all rpm packages through the meta package "cuda" (as shown above). You could specify which of the packages you really need to install there (see the repository). To preview all packages provided by the NVidia CUDA repository execute:
# yum --disablerepo="*" --enablerepo="cuda" list available
Alternative way to preview all packages available in the repository "cuda" is to use the cached locally sqlite3 database of that repository:
# yum makecache
# HASH=`ls -p /var/cache/yum/x86_64/7/cuda/ | grep -v '/$' | grep primary.sqlite.bz2 | awk -F "-" '{print $1}'`
# cp /var/cache/yum/x86_64/7/cuda/${HASH} ~/tmp
# bunzip ${HASH}-primary.sqlite.bz2
# sqlite3 ${HASH}-primary.sqlite
sqlite> select name,version,arch,summary from packages;
5. Compiling and installing OpenMPI
Be sure you have the building environment set as explained before. The install the packages hwloc-devel and valgrind-devel:
# yum install hwloc-devel valgrind-devel
and finally proceed with the configuration, compilation, and installation:
$ cd /home/builder/compile
$ . ~/.intel_env
$ . /usr/local/appstack/.appstack_env
$ wget https://www.open-mpi.org/software/ompi/v2.0/downloads/openmpi-2.0.0.tar.bz2
$ tar jxvf openmpi-2.0.0.tar.bz2
$ cd openmpi-2.0.0
$ ./configure --prefix=/usr/local/appstack/openmpi-2.0.0 --enable-ipv6 --enable-mpi-fortran --enable-mpi-cxx --with-cuda --with-hwloc
$ gmake
$ gmake install
$ ln -s /usr/local/appstack/openmpi-2.0.0 /usr/local/appstack/openmpi
$ export PATH=/usr/local/appstack/openmpi/bin:$PATH
$ export LD_LIBRARY_PATH=/usr/local/appstack/openmpi/lib:$LD_LIBRARY_PATH
Do not forged to update the variables PATH and LD_LIBRARY_PATH by editting their values in the file /usr/local/appstack/.appstack_env. The OpenMPI installation thus compiled and installed provides your applications with more actual MPI tools and libraries than might be provided by the recent Intel C/C++/Fortran Compiler package.
6. Compiling and installing GROMACS
Be sure you have the building environment set as explained before and OpenMPI installed as shown above. Then proceed with GROMACS compilation and installation:
$ cd /home/builder/compile
$ wget ftp://ftp.gromacs.org/pub/gromacs/gromacs-2016.tar.gz
$ tar zxvf gromacs-2016.tar.gz
$ cd gromacs-2016
$ . ~/.intel_env
$ . /usr/local/appstack/.appstack_env
$ cmake . -DCMAKE_INSTALL_PREFIX=/usr/local/appstack/gromacs-2016 -DGMX_MPI=ON -DGMX_BUILD_OWN_FFTW=ON -DGMX_GPU=ON -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda -DMPI_C_LIBRARIES=/usr/local/appstack/openmpi/lib/libmpi.so -DMPI_C_INCLUDE_PATH=/usr/local/appstack/openmpi/include -DMPI_CXX_LIBRARIES=/usr/local/appstack/openmpi/lib/libmpi.so -DMPI_CXX_INCLUDE_PATH=/usr/local/appstack/openmpi/include
$ gmake
$ gmake install
$ export PATH=/usr/local/appstack/gromacs-2016/bin:$PATH
$ export LD_LIBRARY_PATH=/usr/local/appstack/gromacs-2016/lib64:$LD_LIBRARY_PATH
Do not forged to update the variables PATH and LD_LIBRARY_PATH by editting their values in the file /usr/local/appstack/.appstack_env.
7. Invoking GROMACS
To invoke GROMACS compiler and installed by following the instruction in this document you need to have the executable gmx_mpi (not gmx!!!) in your PATH environmental variable, as well as the path to libgromacs_mpi.so. You may set those paths by appending to your .bashrc the line:
. /usr/local/appstack/.appstack_env
You may execute as well the above as a command line only when you need to invoke gmx_mpi if you do not want to write this down to .bashrc.
