Electromagnetic Forward Modeling
Controlled-source electromagnetic method has played an important role on marine oil exploration in these years. This project parallelizes a finite-elment time-domain algorithm for this method. The key step is solving sparse linear system. We design and implement a parallel iterative solver and a customized Incomplete Cholesky preconditioner to achieve both convergence and scalability. The research is supported by National Natural Science Foundation of China. Now, the project is in the charge of Fang Jiarui, a PHD candidate in our group.
Finite-difference time-domain (FDTD) methods have long been the most popular solution for transient electromagnetic (TEM) modeling. Compared with FDTD methods, finite-element time-domain (FETD) methods have the potential to reduce the number of unknowns and the number of time steps. However, with a requirement to solve an irregular and large sparse matrix equation at every time step, it is generally difficult for FETD methods to take advantage of massively parallel computers. In our work, we propose a parallel FETD solution for TEM diffusion phenomena. We design and implement a parallel iterative solver, which includes a preconditioner that is customized for the FETD method to achieve both fast convergence and good scalability.
Furthermore, we have presented two speedup approaches to gain more performance benefits in parallel environment. One is to overlap PCG and iChol calculation. The other is to reuse preconditioners. Based on these two schemes, the performance and efficiency of the parallel FETD method are further improved when the core is increased. Using a seabed model, our experiments in show that our parallel FETD solution provides the same accuracy as the FDTD methods. As indicated in Fig 1 on the performance side, using 128 CPU cores, our parallel FETD method runs 4 times faster than the serial verison when simulating a seabed model with around 100,000 unknowns.
Fig 1 scalability of FETD method based on seabed model
The diffusion equations in FETD method is as follow:
After spatial discretization by finite-element and temporal discretization by backward Eular, the problem can be transformed to solve a linear equations. The basic workflow is indicated in Fig 2.
Fig 2 workflow of FETD
Fig 3 matrix construction phase in our design
During the matrix construction phase, we use a master-slave pattern to read parameters and generate matrix on different processors as indicated in Fig 3. During the matrix solving phase, we first redistribute the matrix to different matrix according to ParMETIS reordering to reduce the rows dependency between different processors. Then we utilize our customized incomplete Cholesky factorization PCG method to solve the linear equations.
Our customized iChol preconditioner is based on Guassian elimination, we propose a communication model based on elimination tree to reduce the overhead of message passing between processors. The CG solver is also parallelized with BLAS operation based on MPI. To dig deep parallelism under the FETD workflow, as indicated in Fig 5 we overlap the preconditioner and CG solver on half processors, respectively. In addition, we also reuse the preconditioner for different CG solver to reduce the calculation of preconditioner.
Fig 5 overlap scheme for future optimization of FETD
We intend to future our study on the FETD method on other parallel platforms, such as GPU and Intel Xeon Phi. A customized preconditioner designed for PCG on these platform is also Instructive to solve large-scale sparse matrix in other fields.