Multi-threading Extension of MEAM Force Field in Molecular Dynamic Simulations
Bowman, A., Mun, S., Horstemeyer, M., & Baskes, M. I. (2019). Multi-threading Extension of MEAM Force Field in Molecular Dynamic Simulations. High-Performance Computing Modernization Program (HPCMP) User Group Meeting (UGM). Vicksburg, MS: U.S. Army Engineer Research and Development Center (ERDC).
Threading optimization is added to Modified Embedded Atom Method (MEAM) code in popular LAMMPS molecular dynamics simulation package where the main parallelism is done by distributed-memory Message Passing Interface (MPI). The multithreading technique that we considered in this work is OpenMP. Strong Scaling performance for a polyethylene system containing 90 million total atoms and 8 million atoms (carbon and hydrogen) for various combinations of MPI ranks and OpenMP threads. Simulations were performed on the Argonne Leadership Computing Facilities Mira compute cluster. OpenMP+MPI hybrid system with 32 MPI ranks per node using 2 MPI threads per processor shows the best performance at high node counts such as 16384 nodes for large scale system and 8192 nodes for small scale system. Molecular Dynamic (MD) simulations are widely used to provide an atomic level of the physical and chemical properties of complex systems including metal, polymer, ceramic and the mixture. At its core, interatomic forcefields including MEAM plays an important role to provide a variety of useful microscopic information such as phase diagram, and structural properties that can be extracted from the potential energy as a function of atomic position. Specifically, MD simulations based on forcefields considerably reduce the computational complexity by employing a simpler model, i.e., considering the nuclear core together with its orbital electrons as a single entity, enabling the size of the simulation up to the order of billions of atoms in microseconds of simulation time. In spite of its implicit electron interaction, the forcefields have shown impressive prediction capability because they are based on experimental data and first principle (or quantum chemistry) calculations. Among many forcefields such as EAM, REBO, COMB, and ReaxFF, MEAM is considered for this work because of three reasons; 1) its extensive database on single and multi-element materials, 2) its reactive character allowing the ability to observe the changes in the chemical reaction of materials, and 3) its well-balanced approach to complexity/accuracy. The parallel molecular dynamics code, LAMMPS, was utilized to perform all simulations in this study. LAMMPS was designed to run on parallel processors employing the spatial-decomposition technique. Physical domain (simulation box) is divided into multiple 3-dimensional sub-domains, and one subdomain is handled by one processor. Each processor computes forces on all atoms in the subdomain using the information from nearby processors via MPI. Even though LAMMPS is well known for its MPI scalability, there still is a room for further improvement in MEAM code in a case where local density for each subdomain is different so that some processors finish the calculations faster than the others such as polymeric system. In such a case, multithreading can improve parallel performance. In MEAM code, we added OpenMP directive on the main computation intensive loops in order to add parallelization over neighbor interaction loops. The screening function and the force calculation, together responsible for almost 80% of the CPU time, where parallelized with OpenMP and showed considerable speedup. The addition of OpenMP to the source code provides the possibility of outperforming MPI only program, especially when the system size is increased into the millions of atoms and MPI performance begins to decrease due to the communication burden.