Loci

Loci: Logic Programming for Parallel Computational Field Simulations

The Loci Framework

The Loci framework was originally developed in the late 1990s with support from the NSF ERC for Computational Field Simulations to simplify the development of complex numerical models that can take advantage of massively parallel high end computing systems. The framework provides a rule-based programming model whereby an application is described in terms of a collection of simple computational kernels. A key feature of the Loci framework are that these computational kernels are documented such that the data-flow between kernels can be extracted using relational annotations that are provided by the rules. In this system the given data is provided as a set of facts (relations). The application is then given in the form of a collection of transformation rules. Finally, the user makes a query (e.g. what is the heat flux at this surface?) From this the scheduler provided by the Loci framework provide a composition of the provided transformations that will answer the query. The effect of the programming model is to place numerical programming in a sphere similar to one of information management using relation databases where problems are solved using powerful relational queries.

The advantage of the Loci framework is that it views the data-structures of a scientific computation abstractly as a relation. Computational kernels then read and write data using relational queries. Relations provide a mechanism for abstracting the data-structure from the computations. In consideration of performance most data-structures in Loci are implemented using distributed arrays, but the model does not preclude alternative implementations such as hashed containers. This separation of concerns gives the Loci programming model an ability to morph an application onto a wide variety of architectural configurations.

To facilitate flexible scheduling of computations, the majority of computational transformations described by Loci rules have single-assignment semantics and are referentially transparent. Loci takes advantage of these features in order to optimize the scheduling of computations in order to factor in non-uniform cost structures presented by modern high end computing systems. For example, Loci provides a work replication optimization that replicates computations on remote processors in order to reduce inter-processor bandwidth requirements. These optimization have been shown to improve application scalability on clusters that use low-cost and subsequently low-bandwidth interconnect. Optimizations that change the order of computations in order to enhance locality have also been implemented and shown to improve application performance.

Why Is The System Called Loci?

The name of this system, Loci, is the plural of locus. The meaning of locus that is implied here is the set of points that satisfy some given set of rules. For example, a circle is the the locus of points that are equidistant from a given center point. The Loci system derives the control structure by finding the set of "entities" that satisfy the rules given to the system. Thus, in a discrete sense, the program control structure in the Loci system is derived by computing a set of locus. From this the process the Loci system borrows its name.

Simulation Tools

CHEM - Extension, Consulting and Customized Development
CFD Research has more than 40 years of combined experience in developing customized capabilities for the Loci/CHEM flow solver. The solver has been applied to large-scale CFD simulations of reacting flows, including high-speed and real-fluids-based rocket engine and gas-turbine simulations. The solver capabilities have been extended to allow production-level coupled fluid-structure, continuum-rarefied, and gas-granular flows of interest to NASA, DoD, DoE and Aerospace and Defense industry partners.

GGFS - Gas-Granular Flow Solver
CFD Research has developed a Eulerian-Eulerian gas-granular flow solver for modeling granular material efficiently and accurately. The solver has been applied to large scale simulations of plume-soil interactions, upwards of 100M cells, and 8-10K processors. The results include cratering characteristics, debris and regolith characteristics post impact. The solver can also be applied in process modeling industry and fluidized bed modeling. Constitutive relationships for polydisperse spherical particle systems, and for specific applications DEM-informed non-spherical particle systems can be incorporated into the models.

Boltzmann - Rarefied Gas-Dynamics Flow Solver
CFD Research has developed a Eulerian rarefied gas-dynamics flow solver which can provide more than an order-of-magnitude superior performance compared to the Monte Carlo based solvers. The solver can also be coupled to Loci/CHEM to allow combined continuum-rarefied flow simulations accurately. Full-collision integrals, as well as BGK based approaches for 3D and axisymmetric configurations is available. Monoatomic and diatomics species mixtures can be simulated. The tool can be applied for various in-space applications, including space thrusters, plume evolution, and other aspects.

THRUST and Acoustics Solver
CFD Research has developed capability to allow high-order acoustics propagation in conjunction with CFD Solver. The tools can be used for acoustic analysis and for combustion stability analysis.

VOF - Gas-Liquid Flow Solver Module
A fully unstructured volume-of-fluid (VOF) solver module that links with Loci/CHEM and Loci/STREAM is available. The module has been applied for slosh-dynamics modeling, surface tension driven flows in micro-gravity, ignition-over pressure mitigation and other applications. It has also been used for injector design and analysis. The module allows capability to deduce vof structures, and allows coupling with Lagrangian particle transport framework for smaller (under-resolved) liquid structures. Simulations with ~100M cells have been conducted on NASA and DoD supercomputers.

RTE - Generalized Radiative Transfer Equation Solver for Combustion Systems
A generalized radiative transfer equation solver has been developed for analyzing combustion systems. The solvers and radiative property models allow close to line-by-line accuracy for radiation transport modeling in combustion systems. Multiple RTE solvers including spherical harmonics, finite-volume discrete ordinates, and photon Monte Carlo methods are available. High-accuracy radiative property databases for key combustion gases including CO2, CO, H2O, and C2H4 based on the full-spectrum k-distribution approach are available, as well as line-by-line random-number relationships from 0.1 bar to 80 bar are available. The solver can also couple with any existing CFD solver with a well-defined API. The RTE Solver also provides a dual-mesh capability, to allow a coarser radiation grid compared to the flow grid.

Loci-Stream - Parallel-Computing and Streamlining
The Loci-Stream code is a product of the coupling of a CFD flow solver (Stream) to an open-source parallel-computing framework (Loci). The Loci framework is a highly scalable framework that powers the Stream solver to run on large-scale machines with thousands of processors. Loci-Stream is designed to take advantage of existing commercially available front-end technology (standard grid generators) and back-end technology (post-processing software such as Tecplot, FieldView, EnSight etc.) which is relatively inexpensive compared to proprietary pre-processing and post-processing software provided by the major CFD vendors.

Representative Papers

  • "A Rule-Based Specification System for Computational Fluid Dynamics", PhD Dissertation, Mississippi State University, December 1999
  • "Loci: A Deductive Framework for Graph-Based Algorithms", ISCOPE'99 Copyright Springer-Verlag pp 142-153
  • E. Luke and T. George, "Loci: A Rule-Based Framework for Parallel Multidisciplinary Simulation Synthesis," Journal of Functional Programming, Special Issue on Functional Approaches to High-Performance Parallel Programming, Volume 15, Issue 03, pp. 477-502, Cambridge University Press
  • Y. Zhang and E. Luke, "Dynamic Memory Management in the Loci Framework," Parallel and Distributed Computing Practices , Volume 7, Number 3, September, 2006
  • P. Adhikari, E. Luke, and E. Allen, "Verification of a Loop Scheduling Protocol using Finite State Verification," ISCA 22nd International Conference on parallel and Distributed Computing and Communication Systems (PDCCS-2009), September 24-26, Louisville, Kentucky, 2009
  • Y. Zhang and E. Luke, "The Design and Implementation of Dynamic Irregular Parallel Computations using the Key-Value-Reference Model," ISCA 22nd International Conference on parallel and Distributed Computing and Communication Systems (PDCCS-2009), September 24-26, Louisville, Kentucky, 2009
  • K. Soni, N. Cain, and E. Luke, "Work Replication: A Communication Optimization in Loci," ISCA 21st International Conference on Parallel and Distributed Computing and Communication Systems (PDCCS-2008), September 24-26, 2008, New Orleans, LA
  • Y. Zhang and E. Luke, "Concurrent Composition Using Loci," Computing in Science and Engineering, Volume 11, Issue 3 (May 2009), pp. 27-35
  • Y. Zhang and E. Luke, "Using the Loci Framework for Automated Program and Component Generation," Workshop on Automated Program Generation for Computational Science, ICCS May 31-Jun 2, 2010, Amsterdam, The Netherlands, Procedia Computer Science, Volume 1, Issue 1, May 2010, pp. 1855-1861
  • E. Luke, "Loci: Automated Synthesis for Terascale Computing Systems," 27th Army Science Conference, Orlando, Fl, November 29th - December 2nd, 2010, CP-08
  • S. Medders, E. Allen and E. Luke, "Using Rule Structure to Evaluate the Completeness of Rule -Based System Testing," Journal of Software Engineering and Knowledge Engineering, Volume 20, Issue 7, pp. 975-986, 2010

Example Applications

The Open Source FlowPsi CFD Solver

The flowPsi solver is an open source implementation of a fluid flow solver using computational fluid dynamics model to solve the Navier Stokes equations for compressible ideal gas flows. It can be found on source forge at flowPsi.

Chem a Finite-Rate Chemistry Solver

The Loci/CHEM code, originally developed as a technology demonstrator for the Loci framework, has become a well developed and mature simulation code for complex multiphysics simulations. The solver that forms the central core of the code is based on high-resolution Godonov methods implemented for multicomponent flows using an implicit time integration scheme that makes the code effective for high Reynolds number flows at high speeds. The core algorithims have been extended to accurately model flows at low speeds through the use of PDE preconditioning techniques.

Since the original development in 1999, the Loci/CHEM code has been significantly extended under support from NASA Marshall Space Flight Center in order to simulate cryogenics, high pressure combustion, multiphase flows, coupled non-gray radiation effects. More recently support from the Air Force, Department of Homeland Security and US Army are extending the capabilities of this mature production code into areas of hypersonic flows and blast simulations. Although the first version of the code was released in 1999, by 2006 the Loci/CHEM code was among the top 30 in CPU hours requested on the DoD MSRC centers. Today it is used widely by NASA and industry predominately in the simulation of aerospace related problems. Notably it has been used extensively as the reference model in external aerodynamics simulations for the ARES I. More information about the CHEM code can be found here.