EPI-LBC

The Lattice Boltzmann kernel included simulates a 3D vortex shedding around a cylindrical geometry at a low Reynolds number in double floating-point precision. As a particle method, the kernel discretizes space into a 3D structured grid (a lattice), time according to the CFL condition, and velocity into a number (here, 19) of fixed flow directions. This results in each lattice holding a particle distribution function (PDF) F(velocity, space) that describes the flow and allows one to recover variables such as velocity and momentum using simple integrals over these distribution functions and known weights. This results in the well-known D3Q19 model, where the grid is traversed, and the new distribution function values are then calculated only depend on old grid values.

D3Q19 accesses for lattice sites

The three main components of the kernel are firstly the streaming part, where the distribution functions from 19 neighboring lattice sites are pulled, the calculation of macroscopic variables, and lastly the collision step, wherein an operator is executed, calculating the new distribution function and storing the new grid.

A 2D example of a lattice update

he distribution function is stored as a 4D array in one of two forms: F(q,i,j,k) or F(i,j,k,q), where ‘q’ denotes velocity and the other three coordinates denote a lexicographic numbering of the grid. Since Fortran is column major the first layout presents more of a ‘structure of array’ type pattern where each lattice point stores the velocity vectors contiguously, as opposed to the second layout which includes the spatial coordinates in the most rapidly changing dimension.

Compilation and Configuration

First you must ascertain that your MPI library was built targeting the fortran compiler you intend to use.

Next you may set your mpifort compiler path on line 623 of the Makefile. Then simple run make.

$ cd src
$ make

This will generate a binary called lbc.

Problem parameters (grid size, processor orientation, iterations, etc) may be set with the lbc.params file. The grid size in a given direction must be a multiple of both, the number of processors in that given direction, as well as 32, which is the tile block length in every loop.

The Makefile allows you to set different MPI options and performance settings. Two interesting performance options are listed:

1. To change the data layout mentioned above, comment out either of the layout options on line 136 of the Makefile and enable the other.
2. To split the stream and collide routines, comment out the “LBMODEL += $(FDEF)-DCOMBINE_STREAM_COLLIDE” line in the Makefile, and enable the “-DSPLITLOOPS” flag.

The performance.res file will be generated after the first run to save performance metrics.

Execution

You can execute this kernel by running

$ mpirun -np <number of ranks> ./lbc

The number of ranks should match the number set in the lbc.params file.