Introduction to PETSc
Matrix
Jeremy Foulon
Institut du Calcul et de la Simulation - UPMC
14 mai 2013
J
er
emy Foulon
Introduction to PETSc
About Matrices for PETSc
Matrix in PETSc are called Mat.
PETSc provides a large variety of matrix implementation because no single
formats is appropriate for all problems.
Currently PETSc support dense storage and compressed sparse row
storage in sequential and parallel versions, as well as several specialized
formats. Additional formats can be added.
We present here the basic use of PETSc matrices involves in the following
actions :
1
create a matrix with a particular type
insert/add values in the matrix
process the matrix
use the matrix
destroy the matrix
Documentation : all matrix routines : http://www.mcs.anl.gov/
petsc/petsc-current/docs/manualpages/Mat/index.html
J
er
emy Foulon
Introduction to PETSc
Creating matrix
Simplest routines for forming a PETSc Matrix :
MatCreate(MPI comm comm, Mat* A)
comm : PETSC COMM SELF (sequential application) or
PETSC COMM WORLD (parallel application)
A : a pointer on the matrix
MatSetSizes(Mat* A, int m, int n, int M, int N)
m : local number of rows
n : local number of columns
M : global number of rows
N : global number of columns
J
er
emy Foulon
Introduction to PETSc
Creating matrix
Remarks :
1 the user specifies either the global or local matrix dimensions.
Useless parameter can be replaced by the keyword PETSC DECIDE
2
PETSc manage memory allocation
by default MatCreate use sparse AIJ format
J
er
emy Foulon
Introduction to PETSc
Adding or inserting values in the matrix
To insert or add values entries to a matrix, one calls a variant of
MatSetValues, either :
MatSetValues(Mat A, int m, const int idxm[],int n, const
int idxn[], const PetscScalar values[], INSERT VALUES)
or
MatSetValues(Mat A, int m, const int idxm[],int n, const
int idxn[], const PetscScalar values[], ADD VALUES)
This routine inserts a mxn block of values in the matrix.
m : number of rows
idxm : global indexes of rows
n : number of columns
idxn : global indexes of columns
values : array containing values to be inserted.
The value to be put in row idxm[i] and column idxn[j] is located in
values[i*n+j].
J
er
emy Foulon
Introduction to PETSc
Adding or inserting values in the matrix
Remarks :
1
row and column indices begin with zero (use C convention)
to insert values in column major order use the option (not supported
by all sparse implementation of matrix) :
MatSetOption(Mat A, MAT COLUMN ORIENTED, PETSC TRUE)
3
with block compressed sparse row format (MatSeqBAIJ or
MatMPIBAIJ), we can use for more efficiency the routine :
MatSetValuesBlocked(...)
J
er
emy Foulon
Introduction to PETSc
Assembling matrix
The routines for matrix processing are :
MatAssemblyBegin(Mat A, MAT FINAL ASSEMBLY)
MatAssemblyEnd(Mat A, MAT FINAL ASSEMBLY)
Remarks
1
by placing code between these two calls, the user can perform
computation while messages are in transit
we cannot mixed INSERT VALUES ADD VALUES operations
for such intermediate assembly we can use
MAT FLUSH ASSEMBLY
J
er
emy Foulon
Introduction to PETSc
Sparse Matrix
The default matrix representation within PETSc is the general sparse AIJ
format (CSR : Compressed Sparse Row format). An alternative is the
Block Compressed Row and Block diagonal storage much more efficiency
for problems with multiple degrees of freedom per node.
CSR format : use two arrays of integer (row, col) and one array of
double (val).
With i the i-th indice of row
row[i+1] - row[i] = number of non zeros values on the i-th row
from col[row[i]] to col[row[i+1] -1] : list of non zeros column indices
for the i-th row
from val[row[i]] to val[row[i+1] -1] : non zeros values in the same
order as list of non zeros indices of columns for i-th row
J
er
emy Foulon
Introduction to PETSc
Sparse Matrix
Example : CSR format with arrays row, col and val.
1 2 0
0 0
4 1 2 0 0
0
2
5
0 2
0
0
1 3 3
8
0
0
2 1
We obtain :
row
col
val
= {0, 2, 5, 8, 11, 14}
= {0, 1, 0, 1, 2, 1, 2, 4, 2, 3, 4, 0, 3, 4}
= {1, 2, 4, 1, 2, 2, 5, 2, 1, 3, 3, 8, 2, 1}
J
er
emy Foulon
Introduction to PETSc
Sequential AIJ Sparse Matrices
MatCreateSeqAIJ(PETSC COMM SELF, int m, int n, int nz, int*
nnz, Mat* A)
m : number of rows
n : number of columns
nz : can be used to preallocate matrix memory. Constant number of
non-zero values by rows. Can be set nz=0
nnz : can be used to preallocate matrix memory. nnz[i] represents
the number of non-zeros in the i-th rows. Can be set
nnz=PETSC NULL
J
er
emy Foulon
Introduction to PETSc
Parallel AIJ Sparse Matrices
MatCreateMPIAIJ(PETSC COMM WORLD, int m, int n, int M, int
N, int d nz, int* d nnz, int o nz, int* o nnz, Mat* A)
m : local number of rows, can be set PETSC DECIDE
n : local number of columns, can be set PETSC DECIDE
M : global number of rows, can be set PETSC DECIDE
N : global number of columns, can be set PETSC DECIDE
d nz : analogous to nz for the diagonal portion of the local rows.
Can be set d nz=0
d nnz : analogous to nnz for the diagonal portion of the local rows.
Can be set d nnz=PETSC NULL
o nz : analogous to nz for the off-diagonal portion of the local rows.
Can be set o nz=0
o nnz : analogous to nnz for the off-diagonal portion of the local
rows. Can be set o nnz=PETSC NULL
J
er
emy Foulon
Introduction to PETSc
Preallocation Memory for Parallel AIJ Sparse Matrices
Example :
1 2
0 5
9 0
13 0
0 18
0 0
25 26
30 0
The diagonal sub-matrix
0
0
3
0
6
7
0
0
10 11 0
0
14 15 16 17
0 19 20 21
0 22 23 0
27 0
0 28
0 31 32 33
on the first process
1 2 0
0 5 6
9 0 10
While the off-diagonal sub-matrix is
0 3 0
7 0 0
11 0 0
J
er
emy Foulon
0
4
8
0
12 0
0
0
0
0
24 0
29 0
0 34
is given by :
given by :
0 4
8 0
12 0
Introduction to PETSc
Preallocation Memory for Parallel AIJ Sparse Matrices
Processor 1 : d nz = 2 or alternatively d nnz = {2, 2, 2} and o nz = 2
or alternatively o nnz = {2, 2, 2}.
Processor 2 : d nz = 3 (maximum of non-zero) or alternatively
d nnz = {3, 3, 2} and o nz = 2 or alternatively o nnz = {2, 1, 1}.
Processor 3 : d nz = 1 or alternatively d nnz = {1, 1} and o nz = 4 or
alternatively o nnz = {4, 4}.
J
er
emy Foulon
Introduction to PETSc
Preallocation Memory for AIJ Sparse Matrices
Remarks
1
preallocation of memory of matrix is critical for achieving good
performance during matrix assembling, as this reduces the number
of allocations and copies required
use the option -info during execution will print information about
the success of preallocation during matrix assembly
J
er
emy Foulon
Introduction to PETSc
Dense Matrix
Sequential :
MatCreateSeqDense(PETSC COMM SELF, int m, int n,
PetscScalar* data, Mat* M)
Parallel :
MatCreateMPIDense(PETSC COMM WORLD, int m, int n, int M,
int N, PetscScalar* data, Mat* A)
m : number global of rows, can be replace by PETSC DECIDE
n : number global of colums, can be replace by PETSC DECIDE
M : number local of rows, can be replace by PETSC DECIDE
N : number local of columns, can be replace by PETSC DECIDE
data : optional argument, indicate location of data for matrix
storage, can be replace by PETSC NULL
J
er
emy Foulon
Introduction to PETSc
Matrix operations
Matrix-Vector product :
MatMult(Mat A, Vec x, Vec y)
By default if the user lets PETSc decide the number of components to be
stored locally (by using PETSC DECIDE), vectors and matrices of the
same dimension are automatically compatible parallel matrix-vector
operations.
J
er
emy Foulon
Introduction to PETSc
Matrix operations
other matrix operations :
MatAXPY
Y =Y +aX
MatMult
y =Ax
MatMultAdd
z =y +Ax
MatMultTranspose
y = AT x
MatNorm
r = kAktype
MatDiagonalScale A = diag (l) A diag (r )
MatScale
A=aA
MatConvert
B=A
MatCopy
B=A
MatGetDiagonal
x = diag (A)
MatTranspose
B = AT
MatZeroEntries
A=0
MatShift
Y =Y +aI
J
er
emy Foulon
Introduction to PETSc
Matrix operations
Print a matrix
MatView(Mat A, PetscViewer viewer)
viewer : use in general PETSC STDOUT VIEWER WORLD or
PETSC STDOUT VIEWER SELF . There is additional viewers like :
PETSC STDOUT DRAW WORLD which draws non-zero structure
of the matrix in X-default window
Destroy a matrix : frees space taken by a matrix
MatDestroy(Mat* A)
J
er
emy Foulon
Introduction to PETSc
Exercice : creation dune matrice, affichage, destruction, ...
Create a vector with value : u[i] = (i + 1) 10
Create the identity matrix
Scale the matrix with a double value
Multiply the matrix and the vector
J
er
emy Foulon
Introduction to PETSc
Other matrix types
Some matrix types available in PETSc :
MATSEQBAIJ
MATMPIBAIJ
MATSEQSBAIJ
MATMPISBAIJ
MATSHELL
....
Set type of the matrix :
MatSetType(Mat mat, const MatType matype)
J
er
emy Foulon
Introduction to PETSc
Exercice 1
Soit lequation
2u
(x, t) = f (x, t) x [0, L]
x 2
Pour resoudre numeriquement le probl`eme de Poisson par la methode des
differences finies, on discretise en espace. Soit N est un entier positif,
1
et xi = ih avec i = 0, 1, 2, ...N + 1. Soit ui une
nous posons h = N+1
approximation de u(x) au point x = xi . Nous noterons
ui ' u(xi ), i = 1, 2, ..., N. Do`
u:
1
(ui1 + 2ui ui+1 )
h2
J
er
emy Foulon
= fi
i = 1, .., N
Introduction to PETSc
(1)
Exercice 1 (suite)
Soit A la matrice N N tridiagonale definie par :
2 1 0
0 ...
1 2 1 0 . . .
1
0 1 2 1 . . .
A= 2 .
..
..
..
..
h ..
.
.
.
.
0
0 . . . 1 2
0
0 . . . 0 1
0
0
0
..
.
1
2
On resout le syst`eme
Au = f
.
1
Assemble matrix from a finite difference method on Poisson equation
Use the preallocation method to improve efficiency on assembling.
Increase the size of the discretization and use the command time to
compare efficiency.
J
er
emy Foulon
Introduction to PETSc
Exercice 2
Implement gradient conjugate method to solve a system Ax = b :
initialisation;
r0 := b Ax0 , p0 := r0 ;
while (j <it max && non convergence) do
j := (rj , rj )/Apj , pj );
xj+1 := xj + j pj ;
rj+1 := rj j Apj ;
j := (rj+1 , rj+1 )/(rj , rj );
pj+1 := rj+1 + j pj ;
end
J
er
emy Foulon
Introduction to PETSc
Exercice 3
1
Solve the problem :
u = f on = [0, ]
avec f (x) = sin(x). We define u = 0 on x = 0 et x = .
Print the solution with gnuplot : store the your solution and an
exact solution in file.
File description for gnuplot :
x0
x1
..
.
u0
u1
..
.
sol0
sol1
..
.
xN
uN
solN
command to plot with gnuplot : plot filename.txt u 1 :2 w l,
filename.txt u 1 :3 wl
J
er
emy Foulon
Introduction to PETSc
Get information about the matrix
Returns the numbers of rows and columns in a matrix
MatGetSize(Mat mat,PetscInt *m,PetscInt* n)
or
Returns the number of rows and columns in a matrix stored locally
MatGetLocalSize(Mat mat,PetscInt *m,PetscInt* n)
m : number global or local of rows
n : number global of local columns
Remarks : not collective function.
J
er
emy Foulon
Introduction to PETSc
Get information about the matrix
Returns the range of matrix rows owned by this processor
MatGetOwnershipRange(Mat mat,PetscInt *m,PetscInt* n)
m : global index of the first local row
n : global index + 1 of the last local row
Returns the range of matrix rows owned by each process
MatGetOwnershipRanges(Mat mat,const PetscInt **ranges)
ranges : returns the range of matrix rows owned by each process
J
er
emy Foulon
Introduction to PETSc
Get information about the matrix
MatGetInfo(Mat mat,MatInfoType flag,MatInfo *info)
flag : flag indicating the type of parameters to be returned
(MAT LOCAL - local matrix, MAT GLOBAL MAX - maximum over
all processors, MAT GLOBAL SUM - sum over all processors)
info : matrix information context
typedef struct {
PetscLogDouble block_size; //block size
//number of nonzeros
PetscLogDouble nz_allocated,nz_used,nz_unneeded;
PetscLogDouble memory;
//memory allocated
PetscLogDouble assemblies; //nb of matrix assemblies called
PetscLogDouble mallocs; //nb of mallocs during MatSetValues()
// fill ratio for LU/ILU
PetscLogDouble fill_ratio_given,fill_ratio_needed;
// nb of mallocs during factorization
PetscLogDouble factor_mallocs;
} MatInfo;
J
er
emy Foulon
Introduction to PETSc
Get information about the matrix
Remarks :
to get information about the matrix, we can also use the options :
-info or -mat view info.
to print skeleton of your matrix use the viewer
PETSC VIEWER DRAW WORLD or the option -mat view draw
(with a sleep time : -draw pause 5 ).
J
er
emy Foulon
Introduction to PETSc
Get data store in the matrix
Gets a block of values from a matrix
MatGetValues(Mat mat,PetscInt m,const PetscInt idxm[]
,PetscInt n,const PetscInt idxn[],PetscScalar v[])
m : number of rows
idxm : global indices of rows
n : number of columns
idxm : global indices of columns
v : a logically two-dimensional array for storing the values
Remark :
not collective function, returns only local values
J
er
emy Foulon
Introduction to PETSc
Get data store in the matrix
Gets a row of a matrix
MatGetRow(Mat mat,PetscInt row,PetscInt *ncols,const
PetscInt *cols[],const PetscScalar *vals[])
row : indice of the row
ncols : if not NULL, nb of non-zeros
cols : if not NULL, indices column number
vals : if not NULL, values
Remark :
not collective function, return only local row
J
er
emy Foulon
Introduction to PETSc
Get data store in the matrix
Gets the diagonal of a matrix
MatGetDiagonal(Mat mat,Vec v)
v : vector with diagonal values
Remark :
collective function
J
er
emy Foulon
Introduction to PETSc
Get data store in the matrix
Extracts several submatrices from a matrix. If submat points to an array
of valid matrices, they may be reused to store the new submatrices
MatGetSubMatrices(Mat mat,PetscInt n,const IS irow[],const
IS icol[],MatReuse scall,Mat *submat[])
or
Gets a single submatrix on the same number of processors as the original
matrix
MatGetSubMatrix(Mat mat,IS isrow,IS iscol,MatReuse cll,Mat
*newmat)
Remark :
collective functions
J
er
emy Foulon
Introduction to PETSc
Collective functions to modify the matrix
Zeros all entries of a matrix. For sparse matrices this routine retains the
old nonzero structure
MatZeroEntries(Mat mat)
Zeros all entries (except possibly the main diagonal) of a set of rows of a
matrix
MatZeroRows(Mat mat,PetscInt numRows,const PetscInt
rows[],PetscScalar diag,Vec x,Vec b)
numRows : number of rows
rows : indices of rows
diag : values put in diagonal position
x : optional vector of solutions for zeroed rows (other entries in
vector are not used)
b : optional vector of right hand side, that will be adjusted by
provided solution
J
er
emy Foulon
Introduction to PETSc
Collective functions to modify the matrix
Remarks :
MatZeroRowsLocal : equivalent of MatZeroRows with local indices
MatZeroRowsColumns : zeros all entries (except possibly the main
diagonal) of a set of rows and columns of a matrix
J
er
emy Foulon
Introduction to PETSc
Collective functions to modify the matrix
Calculates various norms of a matrix
MatNorm(Mat mat,NormType type,PetscReal *nrm)
type : the type of norm, NORM 1, NORM FROBENIUS,
NORM INFINITY
nm : the resulting norm
Gets the norms of each column of a sparse or dense matrix
MatGetColumnNorms(Mat A,NormType type,PetscReal *norms)
type : the type of norm, NORM 1, NORM 2, NORM INFINITY
norms : an array as large as the total number of columns in the
matrix
J
er
emy Foulon
Introduction to PETSc
CSRPreallocation
Allocates memory for a sparse parallel matrix in AIJ format
MatMPIAIJSetPreallocationCSR(Mat B,const PetscInt
i[],const PetscInt j[], const PetscScalar v[])
i : the indices into j for the start of each local row
j : the column indices for each local row
v : optional values in the matrix
J
er
emy Foulon
Introduction to PETSc
Exercice 4 : Solve Poisson finite element problem
Use and complete the file main.c in the repository EF to solve a finite
element problem.
1
line 223 : define start and end integers which define range of the
assembly loop. (Use the example with point in line 259).
assemble the finite element matrix
solve the system with conjugate gradient method develop previously
change the method to apply boundary conditions and use
MatZeroRow(...) method.
J
er
emy Foulon
Introduction to PETSc
