Implementing flexible and scalable Particle-in-Cell methods for massively parallel computations

Particle-in-cell methods have a long history in modeling of mantle convection, lithospheric deformation and crustal dynamics. They are primarily used to track material information, the strain a material has undergone, the pressure-temperature history of a certain material, or the amount of volatiles or partial melt present in a region. However, their efficient parallel implementation - in particular combined with adaptive meshes - is complicated due to the complex communication and frequent reassignment of particles to cells. Consequently, many scientific software packages accomplish this efficiency by designing particle methods for a single purpose, like the advection of scalar properties that do not evolve over time (e.g., chemical heterogeneities). Design choices for particle advection, data storage, and parallel communication are then optimized for this single purpose, making the code rigid to changing requirements.

Here, we present algorithms for a flexible, scalable and efficient particle-in-cell method for massively parallel finite-element codes with adaptively changing meshes. Using a modular plugin structure, we allow maximum flexibility of the generation of particles, the carried tracer properties, the advection and output algorithms, and the projection of properties to the finite-element mesh. We discuss the complexity of the these algorithms and present scaling tests ranging up to tens of thousands of cores and tens of billions of particles. We also discuss load-balancing strategies such as balanced repartitioning for particles in adaptive meshes, quantify sources of errors for the advection of particles, as well as how a proposed velocity correction can address the divergence of the velocity within a cell, and how higher-order finite elements can reduce the need for such a correction. Finally, we present whole mantle convection models as application cases, and compare our implementation to a modern advection-field approach.. We have implemented these algorithms in ASPECT (http://aspect.dealii.org), an open-source community code for mantle convection simulations.