Abstract image of a woman sitting on books and studying on a laptop

Matrices ii

Download as PPT, PDF
Solo Hermelin, profile picture
Solo Hermelin

The document is a comprehensive overview of matrices, focusing on singular values, properties, and the Moore-Penrose pseudoinverse. It discusses the geometric interpretation of singular values, projection methods, and the decomposition of matrices, providing detailed definitions and theorems related to hermitian matrices and orthogonal matrices. Various lemmas are presented to establish the relationships between eigenvalues, unitary matrices, and their properties.

Science
1 of 112
Downloaded 109 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
1 Matrices II SOLO HERMELIN Updated: 20.07.07http://www.solohermelin.com
2 SOLO Matrices Table of Content Singular Values Definitions Domain and Codomain of a Matrix A Properties of Square Orthogonal Matrices Definition of the Singular Values Geometric Interpretation of Singular Values Properties of Singular Values Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Householder Transformation Projection of a vector on a vector .b  a  111 min&min 1 nxmxnxmxn x xbxAd nx   −= Computation of Moore-Penrose Pseudoinverse Matrix, A † Properties of Moore-Penrose Pseudoinverse Matrix, A † Description of Projections Related to Moore-Penrose Pseudoinverse Particular case (1) r = n ≤ m: Particular case (2) r = m ≤ n:
3 SOLO Matrices Table of Content (continue – 1) General Solution of Amxn Xnxp = Bmxp Particular case (1) r = m ≤ n Particular case (2) r = n ≤ n General Solution of YpxmAmxn = Cpxn Particular case (1) r = m ≤ n Particular case (2) r = n ≤ n Inverse of Partitioned Matrices Matrix Inverse Lemmas Identities Matrix Schwarz Inequality Trace of a Square Matrix References
4 SOLO Matrices Singular Values Definitions: Any complex matrix A with n rows (r1, r2,…,rn) and m columns (c1,c2,…,cm) [ ]m n nxm ccc r r r A ,,, 21 2 1   =               = can be considered as a linear function (or mapping or transformation) for a m-dimensional domain to a n-dimensional codomain. ( ) ( ){ }AcodomyAdomxxAyA nxmxnxm ∈⇒∈= 11;: In the same way its conjugate transpose: [ ]H n HH H m H H H mxn rrr c c c A ,,, 21 2 1   =               = is a linear function (or mapping or transformation) for an-dimensional codomain to a m-dimensional domain. ( ) ( ){ }AcdomxAcodomyyAxA mxnx HH mxn ∈⇒∈= 111111 ;: Table of Contents
5 SOLO Matrices Domain and Codomain of a Matrix A The domain of A can be decomposed into orthogonal subspaces: ( ) ( ) ( )ANARAdom H ⊥ ⊕= ( )H AR ( )AN ( )H AN ( )AR xAy = 11 yAx H = ( )Adomxmx ∈1 11mxx ( )Acodomy nx ∈11 1nxyR (AH ) – is the row space of AH (dimension r) N (A) – is the null-space of A (x∈ N (A) ⇔ A x = 0) or the kernel of A (ker (A)) (dimension m-r) The codomain of A (domain of AH ) can be decomposed into orthogonal subspaces: ( ) ( ) ( )H ANARAcodom ⊥ ⊕= R (A) – is the column space of A (dimension r) N (AH ) – is the null-space of AH (dimension n-r) Singular Values Table of Contents
6 SOLO Hermitian = Symmetric if A has real components Hermitian Matrix: AH = A, Symmetric Matrix: AT = A Matrices Properties of Square Orthogonal Matrices Use Pease, “Methods of Matrix Algebra”, Mathematics in Science and Engineering Vol.16, Academic Press 1965 Definitions: Adjoint Operation (H): AH = (A*)T (* is complex conjugate and T is transpose of the matrix) Skew-Hermitian = Anti-Symmetric if A has real components. Skew-Hermitian: AH = -A, Anti-Symmetric Matrix: AT =-A Unitary Matrix: UH = U-1, Orthonormal Matix: OT = O-1 Unitary = Orthonormal if A has real components. Charles Hermite 1822 - 1901
7 SOLO Matrices Properties of Square Orthogonal Matrices (continue – 1) Lemma1: All the eigenvalues of a hermitian matrix H are real and the eigenvectors are orthogonal. Proof of Lemma1: Pre-multiply by :iii xHx λ= H ix i H iii H i xxHxx λ= and take the conjugate transpose: ( ) i H iii HH i H i H i xxxHxHxx * λ== This proves that the eigenvalues of H are real. Subtract those two equations: ( ) 00 ** ≠=→=− i H iiii H iii xxsincexx λλλλ From ( ) H ij H jj H j HH j H j xxHxHxHx λλ ==== * Pre-multiply by and post-multiply by and subtractiii xHx λ= H jx H ij H j xHx λ= ix ( ) 0=−→     = = i H jji i H jji H j i H jii H j xx xxHxx xxHxx λλ λ λ 0=→≠ i H jji xxλλIf If we can use the Gram-Schmidt procedure to obtain an eigenvector orthogonal to . ji λλ = jx~ ix ( ) ( ) i i H i j H i jj x xx xx xx −=~
8 SOLO Matrices Properties of Square Orthogonal Matrices (continue – 2) Lemma1: All the eigenvalues of a hermitian matrix H are real and the eigenvectors are orthogonal. Proof of Lemma1 (continue – 1): ( ) 0=−→     = = i H jji i H jji H j i H jii H j xx xxHxx xxHxx λλ λ λ 0=→≠ i H jji xxλλIf If we can use the Gram-Schmidt procedure to obtain an eigenvector orthogonal to . ji λλ = jx~ ix ( ) ( ) i i H i j H i jj x xx xx xx −=~ we can see that ( ) ( ) 0~ =−= i H i i H i j H i j H ij H i xx xx xx xxxx ( ) ( ) ( ) ( ) ( ) ( ) jii i H i j H i jiii i H i j H i jji i H i j H i jj xx xx xx xx xx xx xxH xx xx xHxH ~~ λλλλ =         −=−=−= q.e.d.
9 SOLO Matrices Properties of Square Orthogonal Matrices (continue – 3) Lemma2: Any hermitian matrix H can be factored in H = U Λ UH where Λ=diag (λ1,λ2,…,λn) and U is unitary i.e. U UH = UH U = In. Proof of Lemma2: Let normalize the orthogonal eigenvectors of H ;i.e. iii xxu /:= or H U = U Λ where U = [u1,u2,…,un] Because U is a square matrix having orthonormal columns, and is a square matrix, U is also a unitary matrix satisfying UH U=U UH =In. q.e.d. [ ] [ ]             = 000 00 00 ,,,,,, 2 1 2121      λ λ nn uuuuuuH
10 SOLO Matrices Properties of Square Orthogonal Matrices (continue – 4) Lemma3: A AH and AH A are hermitian matrices that have the same nonzero real positive eigenvalues. Proof of Lemma3: q.e.d. ( ) ( ) HHHHHH AAAAAA == i.e. A AH is hermitian, therefore the eigenvalues λi (A AH ) are real and positive according to Lemma 1. Suppose ui is a normalized eigenvector of λi (A AH ) ≠0 ( ) i H ii H uAAuAA λ= Pre-multiply by AH and define ( ) i H H i i uA AA v λ 1 := ( ) ( ) ( ) ( ) ( ) ( ) ( ) i H ii H H i i H H i H i i H H i HH ii HH vAAvAA AA uA AA AA uA AAuAAAuAAA λ λ λ λ λ =→ →=→= we get We can see that νi is the eigenvector of AH A and λi (A AH ) is the corresponding eigenvalue, meaning that both AH A and A AH have the same nonzero eigenvalues. From ( ) ( ) ( ) ( ) ( ) ( ) 02 2 >==→=→= i i i H i i H iH ii H i H ii HH ii H ii H v vA vv vAvA AAvvAAvAAvvAAvAA λλλ Therefore we can define ( ) 0: >= H ii AAλσ
11 SOLO Matrices Properties of Square Orthogonal Matrices (continue – 5) Lemma4: If U is a unitary matrix then all its eigenvalues have unit modulus. . Proof of Lemma4: form the inner product IUUUU HH == Consider the set of eigenvalues x1, x2, …, xn which we know to be complete and iii xUx λ= nixxIxxUxUxxx iiii H ii H ii HH ii H iii ,11 ** =∀==→=== λλλλλ q.e.d.
12 SOLO Matrices Properties of Square Orthogonal Matrices (continue – 6) Proof of Lemma5: Lemma5: Every unitary matrix U can be expressed as an exponential matrix: where H is hermitian (jH is skew-hermitian) jH eU = Since the eigenvalues of U have unit modulus; i.e. we can writenii ,11 =∀=λ niej Hij HiHii ,1sincos =∀=+= λ λλλ ( ){ } jHjjj eSeeediagSU HnHH == −1 ,,, 21 λλλ  ( ){ } 1 21 ,,, − = SdiagSH HnHH λλλ where: q.e.d.
13 SOLO Matrices Properties of Square Orthogonal Matrices (continue – 7) Table of Contents Decomposition of Square Matrices: ( ) ( ) ( ) ( )    − − ++=−++= HHHH AA j jAAAAAAA 22 1 2 1 2 1 ( ) ( )H H H AAAA +=    + 2 1 2 1 ( ) ( ) ( )HH H H AAAAAA −−=−=      − 2 1 2 1 2 1 here: Hermitian Skew-Hermitian ( ) ( ) ( )HH H H AA j AA j AA j − − =−=    − − 222 Hermitian ( ) ( )      − − ++= HH AA j jAAA 22 1 the matrix generalization of the decomposition of a complex number in the real and imaginary part.
14 SOLO Matrices Lemma6: (6.1) Every complex nxm matrix of rank can be factored into: Definition of the Singular Values ( ) [ ] ( ) ( ) ( ) ( ) ( )                Σ =Σ= − − −−− − H H rmxrnxrrn rmrxH mxmnxmnxnnxm xmrm rxmrxr rnnxnxr V V UUVUA 2 11 21 00 0    where ( ) 0,,, 21211 >≥≥≥=Σ rrdiagrxr σσσσσσ  Unxn and Vmxm are unitary matrices, i.e.: [ ] [ ] ( ) ( )       =         ==         = −− rnxrn rxr H H H H H H I I UU U U UU U U UUUU 0 0 21 2 1 2 1 21 [ ] [ ] ( ) ( )       =         ==         = −− rmxrm rxr H H H H H H I I VV V V VV V V VVVV 0 0 21 2 1 2 1 21 (6.2) σi i=1,…,r are the positive square roots of the nonzero eigenvalues of AH A or A AH and are called the singular values of A. (6.3) The dyadic expansion of A is: where ui and vi are the columns of U1 and V1 respectively. ∑= = r i H iii vuA 1 σ Singular Values
15 SOLO Matrices Lemma6 (continue – 1): Definition of the Singular Values (6.5) The columns of V are orthonormal eigenvectors of AH A: [ ] [ ] ( ) ( ) ( ) ( )        Σ = −−− − rmxrmxrrm rmrxH rxr VVVVAA 00 0 2 1 2121    i.e. the columns of V1 are the eigenvectors of the nonzero eigenvalues, and the columns of V2 are the eigenvectors of the zero eigenvalues of AH A. (6.6) The following relations exist between U1 and V1 1 111 1 111 − − Σ= Σ= rxrnxrmxr rxrmxrnxr UAV VAU H nxm nxm i.e. the columns of U1 are the eigenvectors of the nonzero eigenvalues, and the columns of U2 are the eigenvectors of the zero eigenvalues of A AH . (6.4) The columns of U are orthonormal eigenvectors of A AH : [ ] [ ] ( ) ( ) ( ) ( )        Σ = −−− − rmxrnxrrn rmrxH rxr UUUUAA 00 0 2 1 2121    Singular Values
16 SOLO Matrices Lemma6 (continue – 2): Definition of the Singular Values ( )H AR ( )AN ( )H AN ( )AR xAy = 11 yAx H = ( )Adomxmx ∈1 11mx x ( )Acodomy nx ∈11 1nxy (6.7) The columns U1 of form an orthonormal basis for the column space of A: ( ) ( )ARUR =1 The columns of U2 form an orthonormal basis for the nullspace of AH : ( ) ( ) ( )HH AANUR ker2 == The columns of V1 form an orthonormal basis for the column space of AH : ( ) ( )H ARVR =1 The columns of V2 form an orthonormal basis for the nullspace of A: ( ) ( ) ( )AANVR ker2 == Singular Values
17 SOLO Matrices Lemma6 (continue – 3): Proof of Lemma 6: Definition of the Singular Values and From Lemma 3 we have ( ) [ ] ( ) ( ) ( ) ( ) ( )                 Σ =Λ= − − −−− − H H rnxrnxrrn rnrxH nxnnxn H mxnnxm xnrn rxnrxr rnnxnxrnxn U U UUUUAA 2 1 2 1 211 00 0 ( ) [ ] ( ) ( ) ( ) ( ) ( )                 Σ =Λ= − − −−− − H H rmxrmxrrm rmrxH mxmmxmnxm H mxn xmrm rxmrxr rmmxmxrmxm V V VVVVAA 2 1 2 1 212 00 0 where ( ) 0,,, 21211 >≥≥≥=Σ rrdiagrxr σσσσσσ  and ( ) ( ) riAAAA H i H ii ,,2,10: =>== λλσ Those equations can be rewritten as: [ ] [ ] [ ] [ ]0 0 1121 1121 Σ= Σ= VVVAA UUUAA H H or 0 0 2 2 111 111 = = Σ= Σ= VAA UAA VVAA UUAA H H H H H U2 H V2 ( ) ( ) 00 00 22222 22222 =→== =→== VAVAVAVAAV UAUAUAUAAU HHH HHHHHH Singular Values
18 SOLO Matrices Lemma6 (continue – 4): Proof of Lemma 6 (continue – 1): Definition of the Singular Values 02 =UAH The columns of U2 form an orthonormal basis for the nullspace of AH : ( ) ( ) ( )HH AANUR ker2 == 02 =VA The columns of V2 form an orthonormal basis for the nullspace of A: ( ) ( ) ( )AANVR ker2 == 111 Σ= UUAA H The columns U1 of form an orthonormal basis for the column space of A: ( ) ( )ARUR =1 111 Σ= VVAAH The columns of V1 form an orthonormal basis for the column space of AH : ( ) ( )H ARVR =1 ( )H AR ( )AN ( )H AN ( )AR xAy = 11 yAx H = ( )Adomxmx ∈1 11mx x ( )Acodomy nx ∈11 1nx y Singular Values
19 SOLO Matrices Lemma6 (continue – 5): Proof of Lemma 6 (continue – 2): Definition of the Singular Values ( )H AR ( )AN ( )H AN ( )AR xAy = 11 yAx H = ( )Adomxmx ∈1 11mx x ( )Acodomy nx ∈11 1nx y From ( ) riuAuA AA v i H i i H H i i ,,2,1 11 : === σλ we have [ ] [ ]               = r r H r uuuAvvv σ σ σ /100 0/10 00/1 2 1 2121      or 1 111 − Σ= UAV H 11 1 1 2 11 1 111 2 111 Σ=ΣΣ=Σ= − Σ= − UUUAAVA UUAA H H riuuuAAvA iiii i i H i i ,,2,1 11 2 ==== σσ σσ from which 1 111 − Σ= VAU rivAu i i i ,,2,1 1 == σ 111 Σ=VAU H Singular Values
20 SOLO Matrices Lemma6 (continue – 6): Proof of Lemma 6 (continue – 3): Definition of the Singular Values ( )H AR ( )AN ( )H AN ( )AR xAy = 11 yAx H = ( )Adomxmx ∈1 11mx x ( )Acodomy nx ∈11 1nx y Using and let compute AH A V111 Σ=VAU H 02 =VA ( ) ( ) [ ] ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )         Σ =         =         = −−− − −−− − − − rmxrnxrrn rmrx nxm H nxm H nxm H nxm H nxmH H mxmnxm H nxn rxr rmmxxrrnmxrxrrn rmmxrxnmxrrxn rmmxmxr xnrn rxn VAUVAU VAUVAU VVA U U VAU 00 01 2212 2111 21 2 1 From this equation we obtain: ( ) [ ] ( ) ( ) ( ) ( ) ( ) ∑= −−− − =Σ=                 Σ = − − r i H iii H H H rmxrnxrrn rmrx nxm vuVU V V UUA rxmrxrnxr xmrm rxmrxr rnnxnxr 1 111 2 11 21 00 0 σ Singular Values Table of Contents
21 SOLO Matrices Let perform the following transformations in the domain and codomain : Geometric Interpretation of Singular Values Suppose, first, that A is square and r = n = m, and consider the spherical hypersurface in the domain of A for which: 1v 1x 2 x 2 v 1=x 111 vAv =σ 111 xAx =λ 222 xAx =λ 222 vAv =σ The Indicator Ellipsoid of a 2 x 2 Matrix11 11 nxnxnnx mxmxmmx Uy Vx η ς = = Because y = A x: ( ) ςςςη Σ=Σ=== UVVUVAUy H From which: 11 mxnxmnx ςη Σ= 1 1 22 2 2 2 ===== ∑= r i i HHH VVxxx ςςςς From we have and the mapping of the spherical hypersurface, in the codomain of A is the hypersurface of an ellipsoid: 11 mxnxmnx ςη Σ= iii σης /= 1 1 2 =        ∑= r i i i σ η This ellipsoid is called the indicator ellipsoid of A and the singular values are the lengths of the principal axes of this ellipsoid. Singular Values
22 SOLO Matrices If the square matrix A is singular, i.e., r < n = m, the indicator ellipsoid shrinks to zero in the directions of the principal axes vi for which σi = 0. In this case: Geometric Interpretation of Singular Values (continue – 1) 01 1 1 2 ===<        + = ∑ nr r i i i ηη σ η  If the general case of nonsquare matrices with r < n ≠ m, if we choose the cylindrical hypersurface that has a circular hypersurface projection in R (AH ): 01 1 1 2 ==== + = ∑ mr r i i ςςς  then its mapping will be the surface of the ellipsoid in R (A). 01 1 1 2 ====        + = ∑ nr r i i i ηη σ η  ( )H AR ( )AN ( )H AN ( )AR Singular Values Table of Contents
23 SOLO Matrices Properties of Singular Values (1) The maximum singular value of Anxm is: [ ] [ ] 2 2 02121 maxmaxmaxmax: 22 x xA xAxAAA xxx i i ≠=≤ ==== σσ (2) The minimum singular value of Anxm is: [ ] [ ] 2 2 02121 minminminmin: 22 x xA xAxAAA xxx i i ≠=≤ ==== σσ Proof of (1) and (2) Using x = V ζ we can write: ( ) ∑= = =Σ=Σ=== m i ii H xV HHHHH xVVxxAAxxAxAxA 1 22222 2 ςσςς ς 1 1 22 2 ≤==== ∑= = m i i H xV HHH xVVxxxx ςςς ς To obtain the maximum/minimum of that satisfy the condition we construct the Hamiltonian by adjoin the constraint to the extremum problem: 2 2 xA 1 2 2 ≤x ( )       −+±= ∑∑ == 1:, 1 2 1 22 m i i m i ii H ςλςσλς + for maximum - for minimum Singular Values
24 SOLO Matrices Properties of Singular Values Proof of maximum/minimum singular value of Anxm (continue – 1) The necessary conditions for extremum are: ( )       −+±= ∑∑ == 1:, 1 2 1 22 m i i m i ii H ςλςσλς + for maximum - for minimum ( ) mi H ii i ,,2,102 2 ==+±= ∂ ∂ ςλσ ς Kuhn-Tucker Condition ( ) mi minimumfor maximumforH i i ,,2,1 0 0 2 2 2 2 =    > < +±= ∂ ∂ λσ ς Maximization problem solution: ( ) [ ]AH 22 1,max σσλς == with 0 10,1 2 1 1 2 21 <−= =⇒==== ∑= σλ ςςςς m i im  Kuhn-Tucker Condition Minimization problem solution: ( ) [ ]AH m 22 ,min σσλς == with 0 10,1 2 1 2 11 ≥= =⇒==== ∑= − m m i imm σλ ςςςς  Kuhn-Tucker Condition Singular Values
25 SOLO Matrices Properties of Singular Values Proof of maximum/minimum singular value of Anxm (continue – 2) For any x ≠ 0 we have: [ ] [ ] ( ) 0max max 2 2 2 2 2 ≤ − =−=− xx xIAAx AA xx xAAx A x xA H HH H H HH λ λσ The inequality holds because (AH A-I λmax [AH A]) is non-positive definite. We can see that the equality is satisfied for x = eigenvector (AH A) that corresponds to λmax [AH A], therefore: [ ] 0max 2 2 0 =         − ≠ A x xA x σ In the same way, or any x ≠ 0 we have: [ ] [ ] ( ) 0min min 2 2 2 2 2 ≥ − =−=− xx xIAAx AA xx xAAx A x xA H HH H H HH λ λσ We can see that the equality is satisfied for x = eigenvector (AH A) that corresponds to λmin [AH A], therefore: [ ] 0min 2 2 0 =         − ≠ A x xA x σ [ ]A x xA x σ= ≠ 2 2 0 max [ ]A x xA x σ= ≠ 2 2 0 min Singular Values
26 SOLO Matrices Properties of Singular Values (3) is a norm of Anxm, because it satisfies the norm properties:[ ]Aσ (3.1) is non-negative and if and only if A = 0. Proof of (3.1): From Lemma 3: [ ]Aσ [ ] 0=Aσ [ ] 00 =⇔= AAσ (3.2) Multiplication by a complex constant α: [ ] [ ]AA σαασ = (3.3) Triangle Inequalities: (3.4) Schwarz Inequality: [ ] [ ] [ ] [ ] [ ]BABABA σσσσσ +≤+≤− [ ] [ ] [ ]BABA σσσ ≤ [ ] 0≥Aσ [ ] 000 =Σ=⇔=Σ⇔= H VUAAσ Proof of (3.2): [ ] ( ) [ ]AAAA xx σαααασ === ≤≤ 2121 22 maxmax Singular Values
27 SOLO Matrices Properties of Singular Values (3) is a norm of Anxm, because it satisfies the norm properties:[ ]Aσ (3.1) is non-negative and if and only if A = 0.[ ]Aσ [ ] 0=Aσ [ ] 00 =⇔= AAσ (3.2) Multiplication by a complex constant α: [ ] [ ]AA σαασ = (3.3) Triangle Inequalities: (3.4) Schwarz Inequality: [ ] [ ] [ ] [ ] [ ]BABABA σσσσσ +≤+≤− [ ] [ ] [ ]BABA σσσ ≤ Proof of (3.3): [ ] ( ) ( ) [ ] [ ]BAxBxA xBxAxBABA xx xx σσ σ +=+≤ +≤+=+ ≤≤ ≤≤ 2121 22121 22 22 maxmax maxmax [ ] ( )[ ] [ ] [ ]BBABBAA σσσσ ++≤−+=From which [ ] [ ] [ ]BABA +≤− σσσ In the same way [ ] [ ] [ ]BAAB +≤− σσσ [ ] [ ] [ ]BABA +≤− σσσ Proof of (3.4): [ ] ( ) ( ) [ ] [ ]BA x xB y yA x xB xB xBA x xBA BA xy xx σσ σ =≤ == ≠≠ ≠≠ 2 2 0 2 2 0 2 2 2 2 0 2 2 0 maxmax maxmax Hermann Amandus Schwarz 1843 - 1921 Singular Values
28 SOLO Matrices Properties of Singular Values (4) The absolute value of the eigenvalues of a square matrix Anxn are bounded between the minimum and the maximum singular values: [ ] [ ] [ ] niAAA i ,,2,1 =≤≤ σλσ Proof of (4): We have: [ ] [ ] 0 2 2 ≠∀≤≤ xA x xA A σσ If xi is any normalized eigenvector: A xi = λi xi, then ni x x x x x xA i i ii i ii i i ,,2,1 2 2 2 2 2 2 ==== λ λλ Therefore: [ ] [ ] [ ] niAAA i ,,2,1 =≤≤ σλσ Singular Values
29 SOLO Matrices Properties of Singular Values (5) A square matrix Anxn is singular iff its minimal singular value is zero. [ ] 0=⇔ ASingularA σ Proof of (5): ( ) [ ] [ ]AAVdiagUA n H n σσσσσσσσ =≥≥≥==  2121 ,,, Therefore: [ ] [ ] [ ] 00det ==⇔=⇔ AAASingularA nσσ ( )[ ] [ ] ∏= == n i i H n VdiagUA 11 21 1 det,,,detdetdet σσσσ   (6) For a nonsingular square matrix Anxn we have [ ] [ ] [ ] [ ]11 1 & 1 −− ==⇔ A A A ArNonsingulaA σ σ σ σ Proof of (6): ( ) ( ) H n H n UdiagVAVdiagUA σσσσσσ /1,,/1,/1,,, 21 1 21  =⇒= − [ ] [ ] [ ] [ ]1 1 1 21 /1/10 −− =≥≥=⇒>=≥≥≥= AAAA nn σσσσσσσσσ  Hence: [ ] [ ] [ ] [ ]11 1 & 1 −− == A A A A σ σ σ σ Singular Values
30 SOLO Matrices Properties of Singular Values (7) If the square matrix (A+B) is singular then the maximum singular values of A and of B are greater or equal than the minimum singular value of B and A, respectively. The opposite is not true. ( ) [ ] [ ] [ ] [ ]ABBASingularBA σσσσ ≥≥⇒+ & Proof of (7): If (A+B) is singular, there exists a normalized eigenvector u (║u║2=1), s.t.: ( ) 22 0 uBuAuBuAuBA =⇒−=⇒=+ From this equation we obtain: [ ] [ ]BxBuBuAxAA xx σσ =≥=≥= ≤≤ 212221 22 minmax [ ] [ ]AxAuAuBxBB xx σσ =≥=≥= ≤≤ 212221 22 minmax To prove that the opposite is not true, consider a counterexample: [ ] [ ] [ ] [ ] 15&34 30 05 10 04 =>==>=     − =      = ABBABA σσσσ The right side is satisfied, but is nonsingular.( )      − =+ 40 01 BA Singular Values
31 SOLO Matrices Properties of Singular Values ( ) [ ] [ ] [ ] [ ]ABBASingularBA σσσσ ≥≥⇒+ & Proof of (8): (8) A sufficient condition that the square matrix (A+B) is nonsingular is: We just proved: ( ) [ ] [ ] [ ] [ ]ABorBAingularonsNBA σσσσ <<⇒+ The proof follows directly from property (7). If (A+B) is singular then [ ] [ ] [ ] [ ]ABBA σσσσ ≥≥ & ; hence if then (A+B) is nonsingular. [ ] [ ] [ ] [ ]ABorBA σσσσ << (7) If the square matrix (A+B) is singular then the maximum singular values of A and of B are greater or equal than the minimum singular value of B and A, respectively. The opposite is not true. Singular Values
32 SOLO To prove this we will consider the following three cases: - (A+B) singular, - (A+B) nonsingular but A and B are singular, - (A+B) nonsingular but A or B, or both are nonsingular. Matrices Properties of Singular Values (9) The minimum singular value of a square matrix (A+B) satisfies the inequalities: [ ] [ ] [ ] [ ]( ) [ ] [ ] [ ] [ ] [ ]( )ABBABAABBA σσσσσσσσσ ++≤+≤−− ,min,max Proof of (9): (9.1) - (A+B) singular According to property (5) [ ] 0=+ BAσ Since (A+B) is singular use property (7) [ ] [ ] [ ] [ ] [ ]BABAAB +=≤−⇒≥ σσσσσ 0 [ ] [ ] [ ] [ ] [ ]BAABBA +=≤−⇒≥ σσσσσ 0 This completes the proof when (A+B) is singular. Singular Values
33 SOLO (9.2) - (A+B) nonsingular but A and B are singular, Matrices Properties of Singular Values (9) The minimum singular value of a square matrix (A+B) satisfies the inequalities: [ ] [ ] [ ] [ ]( ) [ ] [ ] [ ] [ ] [ ]( )ABBABAABBA σσσσσσσσσ ++≤+≤−− ,min,max Proof of (9) (continue – 1): ( )  ( ) 22 0 uBuBAuBuBuAuBA =+⇒=+=+ If A is singular, ,there exists a normalized eigenvector u (║u║2=1), s.t. A u=0:[ ] 0=Aσ [ ] ( ) ( ) [ ] [ ] [ ]ABBxBuBuBAxBABA xx σσσσ +==≤=+≤+=+ ≤≤ 212221 22 maxmin and [ ] [ ] [ ] [ ] [ ]BABABA σσσσσ +≤+<− In the same way for (A+B) nonsingular and B singular: [ ] [ ] [ ] [ ] [ ]ABBAAB σσσσσ +≤+<− This completes the proof when (A+B) is nonsingular but A and B are singular. Singular Values
34 SOLO (9.3) - (A+B) nonsingular but A or B, or both are nonsingular. Matrices Properties of Singular Values (9) The minimum singular value of a square matrix (A+B) satisfies the inequalities: [ ] [ ] [ ] [ ]( ) [ ] [ ] [ ] [ ] [ ]( )ABBABAABBA σσσσσσσσσ ++≤+≤−− ,min,max Proof of (9) (continue – 2): BAC +=:Suppose that (A+B) and A are nonsingular, and define: Pre-multiply by C-1 and post-multiply by A-1 : 1111 −−−− += ABCCA Let take any norm of this equation and write triangle an Schwarz inequalities: BACABC ABCCAABCC 1111 1111111 −−−− −−−−−−− ≤ −≤≤− BACCABACC 1111111 −−−−−−− −≤≤− ( ) B ABA B A +≤ + ≤− −−− 111 111 Using property (3), we can define , and because property (6) the previous equation is equivalent to: [ ]** σ= [ ] [ ] [ ] [ ] [ ]BABABA σσσσσ +≤+<− If B is nonsingular in the same way we can prove that: [ ] [ ] [ ] [ ] [ ]ABBAAB σσσσσ +≤+<− This completes the proof when (A+B) is nonsingular but A or B, or both are nonsingular. Singular Values
35 SOLO Using this and property (3.3): Matrices Properties of Singular Values (10) If the square matrix A is a big matrix relative to the square matrix B, then (A+B) can be approximated by A: [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]ABAABABABAIf σσσσσσσσ ≈+≈+≤+⇒>> & Proof of (10): We have: [ ] [ ] [ ] [ ]BBAA σσσσ ≥>>≥ [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]ABAABABABAA σσσσσσσσσ ≈+⇒≈+≤+≤−≈ [ ] [ ] [ ] [ ] [ ]BABABA σσσσσ +≤+≤− Using: and property (9):[ ] [ ] [ ] [ ]BBAA σσσσ ≥>>≥ [ ] [ ] [ ] [ ]( ) [ ] [ ] [ ] [ ] [ ]( )ABBABAABBA σσσσσσσσσ ++≤+≤−− ,min,max we have: [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]ABAABABABAA σσσσσσσσσ ≈+⇒≈+≤+≤−≈ Singular Values
36 SOLO Matrices Properties of Singular Values (11) Multiplicative Inequalities for square matrices: [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]BABABABABABA σσσσσσσσσσ ≤≤≤≤ & Proof of (11): The proof is given in the following steps: (11.1) is the Schwarz inequality of property (3)[ ] [ ] [ ]BABA σσσ ≤ (11.2) prove that :[ ] [ ] [ ]BABA σσσ ≤ If A or B is singular ( or is zero) then A B is singular ( det [A B] = det [A]. det [B]=0 and ) and we have equality. [ ]Aσ [ ]Bσ [ ] 0=BAσ If A or B is nonsingular then A B is nonsingular ( det [A B] = det [A]. det [B]≠0 ) and: ( ) ( ) 11111111 −−−−−−−− ≤=== ABABBAABBA We define , and use the property (6):[ ]** σ= [ ] [ ] [ ] [ ]11 1 & 1 −− == A A A A σ σ σ σ to obtain: [ ] [ ] ( ) [ ]BA BABA BA σσσ ≤≤= −−− 111 111 This result is opposite to Schwarz inequality, proving that is not a norm.[ ]Aσ Singular Values
37 SOLO Matrices Properties of Singular Values (11) Multiplicative Inequalities for square matrices: [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]BABABABABABA σσσσσσσσσσ ≤≤≤≤ & Proof of (11) (continue – 1): (11.3) prove that : If A is singular then: [ ] [ ] [ ] [ ] [ ]BABAorBA σσσσσ ≤ [ ] [ ] [ ]BABA σσσ ≤= 0 If A is nonsingular then: [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]BAB A BA A BAABAAB σσ σ σ σ σσσσ ≤⇒=≤= −− 1111 If B is singular then: [ ] [ ] [ ]BABA σσσ ≤= 0 If B is nonsingular then: [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]BABA B BA BBABBAA σσσ σ σ σσσσ ≤⇒=≤= −− 11 Singular Values
38 SOLO Matrices Properties of Singular Values (11) Multiplicative Inequalities for square matrices: [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]BABABABABABA σσσσσσσσσσ ≤≤≤≤ & Proof of (11) (continue – 2): (11.4) prove that : If A or B are singular then A B is singular, and: [ ] [ ] [ ] [ ] [ ]BABAorBA σσσσσ ≤ [ ] [ ] [ ] [ ] [ ]( )BAorBABA σσσσσ ≤= 0 If B is nonsingular then: [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]BABA B BA BBABBAA σσσ σ σ σσσσ ≤⇒=≥= −− 11 We also have: [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]BABA B BA BBABBABBAA σσσ σ σ σσσσσσ ≤⇒ =≥≥= −−− 111 If A is nonsingular then: [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]BABA A BA BAABAAB σσσ σ σ σσσσ ≤⇒=≥= −− 11 We also have: [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]BABA A BA BAABAABAAB σσσ σ σ σσσσσσ ≤⇒ =≥≥= −−− 111 q.e.d. Singular Values
39 SOLO Matrices Properties of Singular Values (12) Any unitary matrix U (U UH = UH U = I) has all the singular values equal to 1. Proof of (12): [ ] [ ] [ ] iIUUU i H ii ∀=== 12/12/1 λλσ (13) If U is a unitary matrix (U UH = UH U = I) then: Proof of (13): [ ] [ ] [ ] iAUAAU iii ∀== σσσ [ ] ( ) ( )[ ] [ ] [ ] [ ] iAAAAUUAAUAUAU i H i HH i H ii ∀==== σλλλσ 2/12/12/1 [ ] ( ) ( )[ ] [ ] [ ] [ ] [ ] iAAAAAUUUAAUUAUAUA i H i HH i HH i H ii ∀===== σλλλλσ 2/12/12/12/1 q.e.d. Singular Values q.e.d.
40 SOLO Matrices Properties of Singular Values - Summary (4) The absolute value of the eigenvalues of a square matrix Anxn are bounded between the minimum and the maximum singular values: [ ] [ ] [ ] niAAA i ,,2,1 =≤≤ σλσ (3) is a norm of Anxm, because it satisfies the norm properties:[ ]Aσ (3.1) is non-negative and if and only if A = 0.[ ]Aσ [ ] 0=Aσ [ ] 00 =⇔= AAσ (3.2) Multiplication by a complex constant α: [ ] [ ]AA σαασ = (3.3) Triangle Inequalities: (3.4) Schwarz Inequality: [ ] [ ] [ ] [ ] [ ]BABABA σσσσσ +≤+≤− [ ] [ ] [ ]BABA σσσ ≤ (1) The maximum singular value of Anxm is: [ ] [ ] 2 2 02121 maxmaxmaxmax: 22 x xA xAxAAA xxx i i ≠=≤ ==== σσ (2) The minimum singular value of Anxm is: [ ] [ ] 2 2 02121 minminminmin: 22 x xA xAxAAA xxx i i ≠=≤ ==== σσ Singular Values
41 SOLO Matrices Properties of Singular Values – Summary (continue – 1) (5) A square matrix Anxn is singular iff its minimal singular value is zero. [ ] 0=⇔ ASingularA σ (6) For a nonsingular square matrix Anxn we have [ ] [ ] [ ] [ ]11 1 & 1 −− ==⇔ A A A ArNonsingulaA σ σ σ σ (9) The minimum singular value of a square matrix (A+B) satisfies the inequalities: [ ] [ ] [ ] [ ]( ) [ ] [ ] [ ] [ ] [ ]( )ABBABAABBA σσσσσσσσσ ++≤+≤−− ,min,max (8) A sufficient condition that the square matrix (A+B) is nonsingular is: ( ) [ ] [ ] [ ] [ ]ABorBAingularonsNBA σσσσ <<⇒+ ( ) [ ] [ ] [ ] [ ]ABBASingularBA σσσσ ≥≥⇒+ & (7) If the square matrix (A+B) is singular then the maximum singular values of A and of B are greater or equal than the minimum singular value of B and A, respectively. The opposite is not true. Singular Values
42 SOLO Matrices Properties of Singular Values – Summary (continue – 2) (12) Any unitary matrix U (U UH = UH U = I) has all the singular values equal to 1. (13) If U is a unitary matrix (U UH = UH U = I) then: [ ] [ ] [ ] iAUAAU iii ∀== σσσ (11) Multiplicative Inequalities for square matrices: [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]BABABABABABA σσσσσσσσσσ ≤≤≤≤ & (10) If the square matrix A is a big matrix relative to the square matrix B, then (A+B) can be approximated by A: [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]ABAABABABAIf σσσσσσσσ ≈+≈+≤+⇒>> & Singular Values Table of Contents
43 SOLO Matrices Householder Transformation nˆ ( )xnn T  ˆˆ ( )xnn T  ˆˆ x  'x  O A We want to compute the reflection of over a plane defined by the normal ( )1ˆˆˆ =nnn T x  From the Figure we can see that: ( ) ( ) xHxnnIxnnxx TT  =−=−= ˆˆ2ˆˆ2' 1ˆˆˆˆ2: =−= nnnnIH TT We can see that H is symmetric: ( ) HnnInnIH TTTT =−=−= ˆˆ2ˆˆ2 In fact H is also a rotation of around OA so it must be orthogonal, i.e. HT H=H HT =I. x  ( ) ( )  InnnnnnInnInnIHHHH TTTTTT =+−=−−== ˆˆˆˆ4ˆˆ4ˆˆ2ˆˆ2 1 Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Table of Contents Alston Scott Householder 1904 - 1993
44 SOLO Matrices The same result is obtained if we compute α that minimizes: a  b  P a  α ab  α−p  We want to find such that ( )pba  −⊥ap  α= Projection of a vector on a vector .b  a  or: ( ) ( ) ( ) baaaabapba TTTT  1 0 − =⇒−=−= αα and: ( ) ( )[ ] bPbaaaabaaaaap TTTT  ==== −− 11 α ( )[ ]TT aaaaP  1 : − = Projection Matrix ( ) ( ) ( )aababbabababd TTT T  2 2 2 2minminminmin ααααα αααα +−=−−=−= ( ) ( ) ( ) baaa aa d aaba d TT T TT    1 min 2 22 2 0 022 − =⇒        >= ∂ ∂ =+−= ∂ ∂ α α α α Properties of Projection Matrix (1) P is idempotent P2 = P (2) P is symmetric PT = P ( ) cbcPIcPcbP  ,∀−=−⊥ Proof: ( ) ( ) ( ) cbcPIPbcPIbP TT T  ,0 ∀−=−= ( ) 0=− PIPT Hence: PPP TT = ( ) PPPPP TTT == 2 PPPPP TT === b  bP  cP  cP c   − c  Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Table of Contents
45 SOLO Note: If A and b were real, instead of H (transpose & complex conjugate) we have only T (transpose). Matrices Given: Amxn of rank (Amxn) = r ≤ min (m,n) and 1mx b  Find: such that is minimal1nx x  11 mxnxmxn bxAd  −= If the solution is not unique choose such that is minimal1nx x  1nx x  Solution: The minimum is obtained when ( ) ( ) ( )        >= ∂ ∂ =−=−=−= ∂ ∂ 0 0 2 22 2 11 2 AA x d bAxAAbxAAbxA x d H HHH mxnxmxn    ( ) bAAAx HH  1− = A unique solution exists if AH A is positive definite, or rank (AH A) = n, or det|AH A| ≠ 0 ( ) ( )1111 2 11 2 mxnxmxn H mxnxmxnmxnxmxn bxAbxAbxAd  −−=−=Analytic: Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . 111 min&min 1 nxmxnxmxn x xbxAd nx   −=
46 SOLO Matrices Given: Amxn of rank (Amxn) = r ≤ min (m,n) and 1mx b  Find: such that is minimal1nx x  11 mxnxmxn bxAd  −= If the solution is not unique choose such that is minimal1nx x  1nx x  Solution (continue – 1): Geometric: We have A x∈ R (A) for all x ∈ domain (A). We want to find x0 ∈ domain (A), such that is normal to A x.0 xAbpb  −=− ( ) ( ) ( )AdomainxxAbpbxA ∈∀−=−⊥  0 ( ) ( ) ( ) ( )AdomainxxAAbAxxAbxA HHHH ∈∀−=−=  00 0 Hence: 00 =− bAxAA HH  ( )H AR ( )AN ( )H AN ( )AR xA  ( )Adomx ∈0  ( )AcodomY ∈ Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 b  R x0  Nx0  0 xAp  = pb  − xA  ( )Adomx ∈  0 xA  Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . 111 min&min 1 nxmxnxmxn x xbxAd nx   −=
47 SOLO Matrices Let decompose as0 x  ( )H AR ( )AN ( )H AN ( )AR xA  ( )Adomx ∈0  ( )AcodomY ∈ Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 b  R x0  N x0  0xAp  = pb  − xA  ( )Adomx ∈  0xA  ( ) ( ) NR N H R NR xx AofspaceNullANx AofspaceRowARx xxx 00 0 0 000     ⊥    ∈ ∈ += Therefore: RNR xAxAxAxAp 0 0 000    =+== Hence if: (a) N (A) = 0 or (b) The rows of A are linearly dependent or (c) rank (A) = r < m (d) AH A is singular there are a infinity of solutions NRNR xxxxx 00000  ⊥+= The norm of is:0x  NR xx NR xxxxx NR 00000 00   +=+= ⊥ Hence: 0&min 000 == NR xxx  Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . 111 min&min 1 nxmxnxmxn x xbxAd nx   −=
48 SOLO Matrices Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . ( )T AR ( )AN ( )H AN ( )AR xA  ( )Adomx ∈0  ( )AcodomY ∈ Null Space of A Ker (A) span by VA2 T Row Space of A span by VA1 T Column Space of A span by UA1 Left Null Space of A span by UA2 b  R x0  N x0  0xAp  = pb  − xA  ( )Adomx ∈  0xA  bAx R  + =0 0&min 000 == NR xxx  Define the Linear Transformation (Matrix), that gives from , as the Pseudoinverse of A. (A is the direct transformation that gives from : Rx0  b  p  x  xAp  = bAx † R  =0 A† is called Moore-Penrose Pseudoinverse Matrix, because was defined independently by E.H.Moore in 1920 and Roger Penrose in 1955. Eliakim Hastings Moore 1862 - 1932 Roger Penrose 1931 - 111 min&min 1 nxmxnxmxn x xbxAd nx   −= Table of Contents
49 SOLO Matrices Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn .a  ( )H AR ( )AN ( )H AN ( )AR xA  ( )Adomx ∈0  ( )AcodomY ∈ Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 b  R x0  N x0  0xAp  = pb  − xA  ( )Adomx ∈  0xA  bAx † R  =0 bAx † R  =0 Computation of Moore-Penrose Pseudoinverse Matrix, A † Perform Singular Value Decomposition (S.V.D.) of Amxn: where ( ) 0,,, 21211 >≥≥≥=Σ rrA diagrxr σσσσσσ  UAmxm and VAnxn are unitary matrices, i.e.: ( ) [ ] ( ) ( ) ( ) ( ) ( )                Σ =Σ= − − −−− − H A H A rnxrmxrrm rnrxA AA H AAAmxn xnrn rxnrxr rmmxmxrnxnmxnmxm V V UUVUA 2 11 21 00 0  [ ] ( ) ( ) [ ] H AA H AA H AAH A H A AA rmxrm rxr AAH A H A A H A UUUUUU U U UU I I UU U U UU =+=      =      =      = −− 2211 2 1 2121 2 1 0 0 [ ] ( ) ( ) [ ] H AA H AA H AAH A H A AA rnxrn rxr AAH A H A A H A VVVVVV V V VV I I VV V V VV =+=      =      =         = −− 2211 2 1 2121 2 1 0 0
50 SOLO Matrices ( )H AR ( )AN ( )H AN ( )AR xA  ( )Adomx ∈0  ( )AcodomY ∈ Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 b  R x0  N x0  0xAp  = pb  − xA  ( )Adomx ∈  0xA  bAx † R  =0 Since the norm is invariant to the product of orthogonal matrices ( ) bUxVbxVUUbxVUbxA HHHHH  −Σ=−Σ=−Σ=− Introduce the new unknown: R H N H R HH xVxVxVxVy  =+==: But  ( ) 0 0 =⇒Σ=+Σ=+= N H R H N H R H NR xVxVUxVxVUxAxAxA  RR I HH RR HH xxVVxxVxVy    =         === 2/1 ( ) ( ) ( ) ( ) bUybUybxA H rnxrmxrrm r rnrx y H yx         −                 =−Σ=− −−− − 00 0 0 0 minminmin 1 σ σ Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Computation of Moore-Penrose Pseudoinverse Matrix, A †
51 SOLO Matrices ( )H AR ( )AN ( )H AN ( )AR xA  ( )Adomx ∈0  ( )AcodomY ∈ Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 b  Rx0  Nx0  0xAp  = pb  − xA  ( )Adomx ∈  0xA  bAx † R  =0 Therefore: R HH xVxVy  ==: RR I HT RR HH xxVVxxVxVy    =         === 2/1 ( ) ( ) ( ) ( ) bUybUybxA H rnxrmxrrm r rnrx y H yx         −                 =−Σ=− −−− − 00 0 0 0 minminmin 1 σ σ ( ) ( ) ( ) ( ) ( )                     +                 = − −−− −         any xrm rx H rmxrnxrrn r rmrx XbUy 1 1 1 0 00 /10 0 0/1 σ σ R xyx 0 minmin  == ( ) ( ) ( ) ( ) R HH rmxrnxrrn r rmrx xVbUy 0 1 00 /10 0 0/1           =                 = + Σ −−− − σ σ bAbUVx †H† R  =Σ=0 H†† UVA Σ= Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Computation of Moore-Penrose Pseudoinverse Matrix, A †
52 SOLO Matrices ( )H AR ( )AN ( )H AN ( )AR xA  ( )Adomx ∈0  ( )AcodomY ∈ Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 b  R x0  N x0  0 xAp  = pb  − xA  ( )Adomx ∈  0xA  bAx † R  =0 Where: ( ) ( ) ( ) ( )                   =Σ −−− − − = rmxrnxrrn r rmrx † nxm 00 0 0 0 : 1 1 1      σ σ H mxm † nxmnxn † nxm UVA Σ= Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Computation of Moore-Penrose Pseudoinverse Matrix, A † ( ) [ ] ( ) ( ) ( ) ( ) ( ) H AAAH A H A rmxrnxrrn rmrxA AnA † nxm rxmrxrnxr xmrm rxmrxr rnnxmxr UV U U VVA 1 1 11 2 1 1 1 21 00 0 : − −−− − − Σ=                Σ = − −            =Σ − = 1 1 1 1 0 0 : r 1- A rxr σ σ    Table of Contents
53 SOLO Matrices Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Properties of Moore-Penrose Pseudoinverse Matrix, A † ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )         =                                   =ΣΣ −−− − −−− − −−− − − = rnxrnxrrn rnrxrxr rnxrmxrrm r rnrx rmxrnxrrn r rmrx mxn † nxm I 00 0 00 0 0 0 00 0 0 0 1 1 1 1           σ σ σ σ ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )         =                                   =ΣΣ −−− − −−− − − = −−− − rmxrmxrrm rmrxrxr rmxrnxrrn r rmrx rnxrmxrrm r rnrx † nxmmxn I 00 0 00 0 0 0 00 0 0 0 1 1 11           σ σ σ σ ( ) ( ) ( ) ( ) ( ) † nxmmxn rmxrmxrrm rmrxrxr†† nxmmxn I ΣΣ=         =ΣΣ −−− − 00 0 ( ) ( ) ( ) ( ) ( ) mxn † nxm rnxrnxrrn rnrxrxr† mxn † nxm I ΣΣ=         =ΣΣ −−− − 00 0 Using the definition of the Pseudoinverse we can see that
54 SOLO Matrices Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Properties of Moore-Penrose Pseudoinverse Matrix, A † ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) † nxm rmxrnxrrn r rmrx rmxrnxrrn r rmrx rnxrnxrrn rnrxrxr† nxmmxn † nxm I Σ=                   =                           =ΣΣΣ −−− − − = −−− − − = −−− − 00 0 0 0 00 0 0 0 00 0 1 1 1 1 1 1           σ σ σ σ ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) mxn rnxrmxrrm r rnrx rnxrmxrrm r rnrx rmxrmxrrm rmrxrxr mxn † nxmmxn I Σ=                 =                         =ΣΣΣ −−− − −−− − −−− − 00 0 0 0 00 0 0 0 00 0 11           σ σ σ σ ( ) ( ) ( ) ( ) † nxmmxn H†H†† Def† †H††H†H†† nxmmxn AAUUUUUUUVVUAA =ΣΣ=ΣΣ=ΣΣ=ΣΣ= ( ) ( ) ( ) ( ) mxn † nxm H†H†† Def† †H††HH†† mxn † nxm AAVVVVVVVUUVAA =ΣΣ=ΣΣ=ΣΣ=ΣΣ= Also: Let check the same operations for Matrix A † ( ) ( ) ( ) ( ) mxn HH†HH†H mxn † nxmmxn AVUVUVUUVVUAAA =Σ=ΣΣΣ=ΣΣΣ= ( ) ( ) ( ) ( ) † nxm H†H††H†HH†† nxmmxn † nxm AUVUVUVVUUVAAA =Σ=ΣΣΣ=ΣΣΣ=
55 SOLO Matrices Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Properties of Moore-Penrose Pseudoinverse Matrix, A † - Summary) ( ) ††† nxmmxnnxmmxn AAAA = ( ) mxn † nxm † mxn † nxm AAAA = mxnmxn † nxmmxn AAAA = † nxm † nxmmxn † nxm AAAA = 1 2 3 4 Table of Contents
56 SOLO Matrices Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Description of Projections Related to Moore-Penrose Pseudoinverse bPbAAxAp † R  1 ===1 ( )H AR ( )AN ( )H AN ( )AR Rx  ( )Adomx∈ ( )Acodomb ∈  Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 b  bAx † R  = ( ) R † xA bAAp   = = R xAp  = P1 is a projection matrix because ( ) ( ) ( )    =ΣΣ=ΣΣ== === 11 1 2 1 PUUUVVUAAP PAAAAAAP HH†HH†HH†H ††† P1=A A† projects into column space of A, R (A)b  H†H†H† UUUVVUAAP ΣΣ=ΣΣ==:1
57 SOLO Matrices Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Description of Projections Related to Moore-Penrose Pseudoinverse 2 ( )H AR ( )AN ( )H AN ( )AR R x  ( )Adomx ∈  ( )Acodomb ∈  Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 b  bAx † R  = ( ) R † xA bAAp   = = RxAp  = ( )bAAIpb †  −=− ( ) 0=− pbA†  pAx † R  = P2=(I - A A † ) is a projection matrix because ( ) ( ) ( ) ( )     =−=−=−=−= =+−−=−−= 21112 2 2 2 PPIPIPIAAIP PAAAAAAAAIAAIAAIP HHH†H † A †††††  Because , is the projection of into . ( ) ( )H ANAR ⊥ pb  − ( )H ANb  We can see, also, that: ( ) H†H† † UIUUUI AAIPIP ΣΣ−=ΣΣ−= −=−= 12 : ( ) ( ) 00    ==         −=−=− bbAAAAbAAIApbA † A †††††† pAbAx †† R  == ( ) ( ) bPbAAIpb †  2 =−=−
58 SOLO Matrices Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Description of Projections Related to Moore-Penrose Pseudoinverse 3 ( ) ( ) NR N H R NR xx AofspaceNullANx AofspaceRowARx xxx     ⊥    ∈ ∈ += ( ) ( ) xPxAAxAApAbAx †††† R  3 ===== P3=A † A is a projection matrix of in R (AH )x  ( ) ( ) ( ) ( ) ( )     =ΣΣ=ΣΣ== ==== 33 3 2 3 PVVVUUVAAP PAAAAAAAAAAP HH†HHH†H†H † A †††† †  4 ( ) ( ) xPxAAIxAAxxxx †† RN  4 =−=−=−= P4=I-A † A is a projection matrix of in N (A)x  ( )  ( ) ( )     =−=−=−= =+−=−= 43333 4 2 33 2 3 2 4 3 2 PPIPIPIP PPPIPIP HHHH P ( )H AR ( )AN ( )H AN ( )AR ( )Adomx ∈  ( )Acodomb ∈  Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 b  bAx † R  = ( ) R † xA bAAp   = = R xAp  = ( )bAAIpb †  −=− ( ) 0=− pbA†  pAx † R  = ( ) xAAx † R  = ( )H AR ( )AN ( )H AN ( )AR ( )Adomx ∈  ( )Acodomb ∈  Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 b  bAx † R  = ( ) R † xA bAAp   = = R xAp  = ( )bAAIpb †  −=− ( ) 0=− pbA†  pAx † R  = ( ) xAAx † R  = ( ) xAAIx † N  −= 0=NxA 
59 SOLO Matrices Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Description of Projections Related to Moore-Penrose Pseudoinverse (Summary) 3 ( ) ( ) ( )H†††† R ARxPxAAxAApAbAx ∈=====  3 4 ( )H AR ( )AN ( )H AN ( )AR ( )Adomx ∈  ( )Acodomb ∈  Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 b  bAx † R  = ( ) R † xA bAAp   = = R xAp  = ( )bAAIpb †  −=− ( ) 0=− pbA†  pAx † R  = ( ) xAAx † R  = ( ) xAAIx † N  −= 0=NxA  ( ) ( ) ( )ANxPxAAIxAAxxxx †† RN ∈=−=−=−=  4 ( ) ( ) ( )H† ANbPbAAIpb ∈=−=−  2 2 ( )ARbPbAAxAp † R ∈===  1 1 H†† UUAAP ΣΣ==:1 † AAIPIP −=−= 12 : H†† VVAAP ΣΣ==:3 AAIPIP † −=−= 34 : Table of Contents
60 SOLO Matrices Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Particular case (1) r = n ≤ m: ( ) [ ] ( ) H A xnnm A AA H AAAmxn nxn nxn nmmxmxnnxnmxnmxm VUUVUA        Σ =Σ= − − 0 1 21  (a) rank (Amxn) = n or (b) columns of Amxn are linear independent or (c) N (Amxn) = 0 or (d) Anxm H Amxn is nonsingular This is equivalent to: where ( ) 0,,, 21211 >≥≥≥=Σ nnA diagnxn σσσσσσ  [ ] ( ) ( ) [ ] H AA H AA H AAH A H A AA nmxnm nxn AAH A H A A H A UUUUUU U U UU I I UU U U UU =+=      =      =      = −− 2211 2 1 2121 2 1 0 0 ( )[ ] ( ) H AAAH A H A nmnxAA † nxm nxmnxnnxn xmnm nxm nxnnxn UV U U VA 1 1 1 2 11 1 0: − − − Σ=         Σ= − ( )H AR ( ) 0=AN ( )H AN ( )AR x  b  Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 ( )bAAxAp †  == pAx †  = ( )bAAIpb  + −=− xAp  = bAx †  = ( ) 0=− pbA† 
61 SOLO Matrices Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Particular case (1) r = n ≤ n: (continue – 1) ( )[ ] ( ) H AAAH A H A nmnxAA † nxm nxmnxnnxn xmnm nxm nxnnxn UV U U VA 1 1 1 2 11 1 0: − − − Σ=         Σ= − ( ) [ ] ( ) H A xnnm A AA H AAAmxn nxn nxn nmmxmxnnxnmxnmxm VUUVUA        Σ =Σ= − − 0 1 21  [ ] [ ] H AAA H A A AAH A H A AA H VVVUU U U VAA 2 1 1 21 2 1 1 0 0 Σ=     Σ       Σ=  ( ) H AAA H VVAA 2 1 1 −− Σ= ( ) †H AAA H AAA H AAA HH AUVUVVVAAA =Σ=ΣΣ= −−− 1 1 111 2 1 1 or ( ) H nxmmxn H nxm † nxm AAAA 1− = ( )H AR ( ) 0=AN ( )H AN ( )AR x  b  Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 ( )bAAxAp †  == pAx †  = ( )bAAIpb  + −=− xAp  = bAx †  = ( ) 0=− pbA†  We have only one solution that minimize 11 mxnxmxn bxAd  −= x  and is given by: ( ) 1 1 11 mx H nxmmxn H nxmmx † nxmnx bAAAbAx  − == Table of Contents
62 SOLO Matrices Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Particular case (2) r = m ≤ n: (a) rank (Amxn) = m or (b) rows of Amxn are linear independent or (c) N (Anxm H ) = 0 or (d) AmxnAnxm H is nonsingular This is equivalent to: where ( ) 0,,, 21211 >≥≥≥=Σ mmA diagmxm σσσσσσ  ( ) [ ] ( ) H AAA H A xmmn A AnA † nxm mxmmxmnxmmxm mxm mnnxmxr UVUVVA 1 11 1 1 21 0 : − − − Σ=        Σ = −  ( )[ ] ( ) H AAAH A H A mnmxAA H AAAmxn mxnmxmmxm xnmn mxn mxmmxmnxnmxnmxm VU V V UVUA 11 2 1 1 0 Σ=         Σ=Σ= − − [ ] ( ) ( ) [ ] H AA H AA H AAH A H A AA mnxmn mxm AAH A H A A H A VVVVVV V V VV I I VV V V VV =+=      =      =         = −− 2211 2 1 2121 2 1 0 0 ( )H AR ( )AN ( ) ( )ARBAN H ∈≡ &0 ( )AR xAb  = ( ) N † nxn xxAAI  =− ( ) R † xxAA  = Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 bAx † R  = R xAb  = b  0=N xA 
63 SOLO Matrices Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Particular case (2) r = m ≤ n: (continue – 1) ( ) [ ] ( ) H AAA H A xmmn A AnA † nxm mxmmxmnxmmxm mxm mnnxmxr UVUVVA 1 11 1 1 21 0 : − − − Σ=        Σ = −  ( )[ ] ( ) H AAAH A H A mnmxAA H AAAmxn mxnmxmmxm xnmn mxn mxmmxmnxnmxnmxm VU V V UVUA 11 2 1 1 0 Σ=         Σ=Σ= − − H AAA H AA I A H AAA H UUUVVUAA m 2 11111 Σ=ΣΣ=  ( ) H AAA H UUAA 2 1 1 −− Σ= ( ) †H AAA H AAA H AAA HH AUVUUUVAAA =Σ=ΣΣ= −−− 1 11 2 111 1 or ( ) 1− = HH† AAAA ( )H AR ( )AN ( ) ( )ARBAN H ∈≡ &0 ( )AR xAb  = ( ) N † nxn xxAAI  =− ( ) R † xxAA  = Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 bAx † R  = R xAb  = b  0=N xA  We have an infinite number of solutions that minimize 11 mxnxmxn bxAd  −= ( ) bAAAbAx HH† R  1− == The solution that minimizes the norm is given by: Rx  Rx  Table of Contents
64 SOLO Matrices General Solution of Amxn Xnxp = Bmxp X - nxp unknowns with mxp equations mxpnxpmxn BXA = Perform Singular Value Decomposition (S.V.D.) of Amxn: where ( ) 0,,, 21211 >≥≥≥=Σ rrA diagrxr σσσσσσ  UAmxm and VAnxn are unitary matrices, i.e.: ( )H AR ( )AN ( )H AN ( )ARBXA = 11 yAx H = ( )AdomX ∈ ( )AcodomY ∈ 1nx y Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 B ( ) [ ] ( ) ( ) ( ) ( ) ( )                Σ =Σ= − − −−− − H A H A rnxrmxrrm rnrxA AA H AAAmxn xnrn rxnrxr rmmxmxrnxnmxnmxm V V UUVUA 2 11 21 00 0  [ ] ( ) ( ) [ ] H AA H AA H AAH A H A AA rmxrm rxr AAH A H A A H A UUUUUU U U UU I I UU U U UU =+=      =      =      = −− 2211 2 1 2121 2 1 0 0 [ ] ( ) ( ) [ ] H AA H AA H AAH A H A AA rnxrn rxr AAH A H A A H A VVVVVV V V VV I I VV V V VV =+=      =      =         = −− 2211 2 1 2121 2 1 0 0
65 SOLO Matrices General Solution of Amxn Xnxp = Bmxp Let multiply by and using: mxpnxpmxn BXA =         H H U U 2 1 we obtain: [ ]         =              Σ         BU BU X V V UU U U H A H A H A H AA I AAH A H A m 2 1 2 11 21 2 1 00 0     or:         =              Σ BU BU X V V H A H A H A H AA 2 1 2 11 00 0 or: ( ) ( )xprmmxp H A BU xmrm −=− 02 (m-r)xp - constraints equivalent to condition Bmxp∈ℜ (Amxn) mxp H Anxp H AA BUXV rxmrxnrxr 111 =Σ rxp - independent equations nxp – unknowns since r ≤ n → # Eq. ≤ # Unknown ( )H AR ( )AN ( )H AN ( )ARBXA = 11 yAx H = ( )AdomX ∈ ( )AcodomY ∈ Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 B ( ) [ ] ( ) ( ) ( ) ( ) ( )                Σ =Σ= − − −−− − H A H A rnxrmxrrm rnrxA AA H AAAmxn xnrn rxnrxr rmmxmxrnxnmxnmxm V V UUVUA 2 11 21 00 0 
66 ( ) ( )xprmmxp H A BU xmrm − =− 02 (m-r)xp - constraints equivalent to condition Bmxp∈ℜ (Amxn) SOLO Matrices General Solution of Amxn Xnxp = Bmxp mxp H Anxp H AA BUXV rxmrxnrxr 111 =Σ rxp - independent equations nxp – unknowns since r ≤ n → # Eq. ≤ # Unknown This equation is a Necessary and Sufficient Condition for any solutions of equation Amxn Xnxp = Bmxp. It is equivalent to Bmxp∈ℜ (Amxn) or Bmxp ∩ N (AT ) = ∅. If this condition is fulfilled, then from we have nxp unknowns ≥ rxp independent equations, that means (n-r)xp degrees of freedom. mxp H Anxp H AA BUXV rxmrxnrxr 111 =Σ mxp H AAnxp H A BUXV rxmrxrrxn 1 1 11 − Σ= Since VA1 T VA1=Ir & VA1 T VA2 = 0 the General Solution of Amxn Xnxp = Bmxp is: ( ) ( ) ( ) ( )    AN xprnA AR mxp H AAAnxp YVBUVX rnnx T rxmrxrnxr ∈ − ∈ − − +Σ= 21 1 11 where Y(n-r)xp is any (n-r)xp matrix, i.e. we used all (n-r)xp degrees of freedom. ( )H AR ( )AN ( )H AN ( )ARBXA = ( )AdomX ∈ ( )AcodomY ∈ Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 BB has to be in the column space of A ( ) ( ) =∩∈ ANBorARB
67 SOLO Matrices General Solution of Amxn Xnxp = Bmxp Check: ( ) [ ] ( ) ( ) ( ) ( ) ( ) ( ) ( )( ) ( ) [ ] ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) [ ] ( ) ( ) ( ) ( ) ( ) ( ) [ ] ( ) ( ) [ ] ( ) ( ) [ ] ( ) mxpmxp xmrm H A H A AA mxpxmrm H A mxp H A AA xprm mxp H A AA xprn mxp H AA rnxrmxrrm rnrxA AA xprn I A H Amxp H AAA H A xprnA H Amxp H AA I A H A rnxrmxrrm rnrxA AA xprnAmxp H AAAH A H A rnxrmxrrm rnrxA AAnxp H AAAnxpmxn BB U U UUBU BU UU BU UU Y BU UU YVVBUVV YVVBUVV UU YVBUV V V UUXVUXA rxm rmmxmxr rxm rmmxmxr rxm rmmxmxr rxmrxrrxr rmmxmxr rnnxxnrnrxmrxrnxrxnrn rnnxrxnrxmrxrnxrrxnrxr rmmxmxr rnnxrxmrxrnxr xnrn rxnrxr rmmxmxrnxnmxnmxm =         =           =         =        Σ        Σ =             +Σ +Σ        Σ = +Σ                Σ =Σ= − − −− − −−− − − − − − −−− − − − −−− − −− −− −−− − − − − − 2 1 21 0 2 1 21 1 21 1 1 11 21 221 1 1 0 12 0 211 1 1111 21 21 1 11 2 11 21 000 0 00 0 00 0        
68 SOLO Matrices where r is such that: General Solution of Amxn Xnxp = Bmxp Algorithm to solve Amxn Xnxp = Bmxp: (1) Compute s.v.d. of Amxn and partition according to: ( ) 0,,, 21211 >≥≥≥=Σ rrA diagrxr σσσσσσ  (2) Check if: ( ) ( )xprmmxp H A BU xmrm − =− 02 (3) If (2) is not true → no solution for (1) ( ) ( )  any xprnAmxp H AAAnxp YVBUVX rnnxrxmrxrnxr − − − +Σ= 21 1 11 ( ) [ ] ( ) ( ) ( ) ( ) ( )                Σ =Σ= − − −−− − H A H A rnxrmxrrm rnrxA AA H AAAmxn xnrn rxnrxr rmmxmxrnxnmxnmxm V V UUVUA 2 11 21 00 0  If (2) is true → (n-r)xp solutions:
69 SOLO Matrices General Solution of Amxn Xnxp = Bmxp Moore-Penrose Pseudoinverse of A: ( ) [ ] ( ) ( ) ( ) ( ) ( ) H AAAH A H A rmxrnxrrn rmrxA AnA † nxm rxmrxrnxr xmrm rxmrxr rnnxmxr UV U U VVA 1 1 11 2 1 1 1 21 00 0 : − −−− − − Σ=                Σ = − −  then ( ) [ ] ( ) ( ) ( ) ( ) ( ) ( ) [ ] ( ) ( ) ( ) ( ) ( ) H AAH A H A rmxrnxrrn rmrxA I AAH A H A rnxrmxrrm rnrxA AA † mxn rxnnxr xmrm rxmrxr nxn rnnxnxr xnrn rxnrxr rmmxmxrnxm UU U U VV V V UUAA 11 2 1 1 1 21 2 11 21 00 0 00 0 =                Σ                Σ = − − − − −−− − − −−− −     ( ) ( ) ( ) ( ) ( ) H AA H AA H AA H AA † nxmmxnmxm xmrmrmmxrxmmxrxmrmrmmxrxmmxr UUUUUUUUAAI −−−− =−+=− 22112211 also ( ) [ ] ( ) ( ) ( ) ( ) ( ) ( ) [ ] ( ) ( ) ( ) ( ) ( ) H AAH A H A rnxrmxrrm rnrxA I AAH A H A rmxrnxrrn rmrxA AAnxm † mxn rxnnxr xnrn rxnrxr mxm rmmxmxr xmrm rxmrxr rnnxnxr VV V V UU U U VVAA 11 2 11 21 2 1 1 1 21 00 0 00 0 : =                Σ                Σ = − − − − −−− − −−− − −     ( ) ( ) ( ) ( ) ( ) H AA H AA H AA H AAmxn † nxmnxn xnrnrmnxrxnnxrxnrnrmnxrxnnxr VVVVVVVVAAI −−−− =−+=− 22112211
70 SOLO Matrices General Solution of Amxn Xnxp = Bmxp Moore-Penrose Pseudoinverse of A (continue - ): Define also Znxp such that: ( ) ( ) nxp H Axprn ZVY xrrn− =− 2 : ( ) ( )  any xprnAmxp H AAAnxp YVBUVX rnnxrxmrxrnxr − − − +Σ= 21 1 11 Since, if ( ) ( )xprmmxp H A BU xmrm − =− 02 The solution of Amxn Xnxp = Bmxp is ( ) ( ) ( ) ( ) ( ) H AA H AA H AA H AAmxn † nxmnxn xnrnrmnxrxnnxrxnrnrmnxrxnnxr VVVVVVVVAAI −−−− =−+=− 22112211 ( ) [ ] ( ) ( ) ( ) ( ) ( ) H AAAH A H A rmxrnxrrn rmrxA AnA † nxm rxmrxrnxr xmrm rxmrxr rnnxmxr UV U U VVA 1 1 11 2 1 1 1 21 00 0 : − −−− − − Σ=                Σ = − −  Therefore: ( )  any nxpmxn † nxmnxnmxp † nxmnxp ZAAIBAX −+= Note: By writing the solution this way we lose the fact that we have only (n-r)xp different solutions as we have seen. Check: ( ) ( ) ( )  xprn xnrnrnnxrxmrxrnxr Y nxp H AAmxp H AAAnxp ZVVBUVX − −− +Σ= − 221 1 11
71 SOLO Matrices General Solution of Amxn Xnxp = Bmxp Moore-Penrose Pseudoinverse of A (continue - ): ( ) ( ) ( )  any nxpmxn † nxmnxnmxp † nxm any xprnAmxp H AAAnxp ZAAIBAYVBUVX rnnxrxmrxrnxr −+=+Σ= − − − 21 1 11 ( ) ( ) ( ) ( ) ( ) ( ) ( )xprmmxp H AA ANonBofprojection mxp H BUU BAAI orANB orARB xmrmrmmx T − + == − Ο/=∩ ∈ −− 0 0 22     Solutions exists iff: ( )H AR ( )AN ( )H AN ( )AR BXA = ( ) YVZAAI A † nxn 2=− BA† Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 B BA†Z i.e. the projection (Imxm – Amxn Anxm † ) of B on N (AH ) is zero. ( ) ( ) ( ) mxpmxp H AAmxp † nxmmxnmxm BUUBAAI mnrmrmmx 0 0 22 ==− −−  Table of Contents
72 SOLO Matrices General Solution of Amxn Xnxp = Bmxp Particular case (1) r = m ≤ n: solutions always exist ( )H AR ( )AN ( ) ( )ARBAN H ∈≡ &0 ( )AR BXA = ( ) YVZAAI A † nxn 2=− BA† Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 B BA†Z Since ( )[ ] ( )         Σ=Σ= − − H A H A mnmxAA H AAAmxn xnmn rxn mxmmxmnxnmxnmxm V V UVUA 2 1 1 0 ( ) ( ) ( ) ( )xprmmxp H ArmmxA BUU xmrmrmmx −− ≡⇒≡ −− 00 22 nxp unknowns ≥ mxp equations, meaning (n-m)xp degrees of freedom ( ) ( ) ( )  any nxpmxn † nxmnxnmxp † nxm any xprnAmxp H AAAnxp ZAAIBA YVBUVX rnnxrxmrxrnxr −+= +Σ= − − − 21 1 11 ( ) [ ] ( ) ( ) 11 11 1 1 21 0 : −− − − =Σ=        Σ = − H nxmmxn H nxm H AAA H A xmmn A AnA † nxm AAAUVUVVA mxmrxrnxrmxm mxm mnnxnxm  ( )[ ] ( ) ( ) [ ] ( ) mxm H A xmmn A AAH A H A mnmxAA † nxmmxn IUVV V V UAA mxm mxm mnnxnxm xnmn mxn mxmmxm =        Σ         Σ= − − − − − 0 0 1 1 21 2 1  Table of Contents
73 SOLO Matrices General Solution of Amxn Xnxp = Bmxp Particular case (2) r = n ≤ n: mxp equations ≥ nxp unknowns, meaning (n-m)xp constraints Only if solutions exist. ( ) ( )xprmmxp H A BU xmrm − =− 02 In this case we have nxp unknowns and mxp equations - (m-p)xp constraints = nxp independent equations, i.e. a unique solution: mxpnxmmxp H AAAnxp BABUVX nxmnxnnxn +− =Σ= 1 1 11 ( ) [ ] ( ) H A xnnm A AA H AAAmxn nxn nxn nmmxmxnnxnmxnmxm VUUVUA        Σ =Σ= − − 0 1 21  ( )[ ] ( ) H AAAH A H A nmnxAA † nxm nxmnxnnxn xmnm nxm nxnnxn UV U U VA 1 1 2 11 0: − − − Σ=         Σ= − ( )[ ] ( ) ( ) [ ] ( ) nxn H A xnnm A AAH A H A nmnxAAmxn † nxm IVUU U U VAA nxn nxn nmmxmxn xmnm nxm nxnnxn =        Σ         Σ= − − − − − 0 0 1 21 2 11  ( )H AR ( ) 0=AN ( )H AN ( )ARBXA = BA† 1nx y Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 BB has to be in the column space of A ( ) ( ) =∩∈ ANBorARB Table of Contents
74 SOLO Matrices General Solution of YpxmAmxn = Cpxn Y - pxm unknowns with pxn equations pxnmxnpxm CAY = Perform Singular Value Decomposition (S.V.D.) of Amxn: where ( ) 0,,, 21211 >≥≥≥=Σ rrA diagrxr σσσσσσ  UAmxm and VAnxn are unitary matrices, i.e.: [ ] ( ) ( ) [ ] H AA H AA H AAH A H A AA rmxrm rxr AAH A H A A H A UUUUUU U U UU I I UU U U UU =+=      =      =      = −− 2211 2 1 2121 2 1 0 0 [ ] ( ) ( ) [ ] H AA H AA H AAH A H A AA rnxrn rxr AAH A H A A H A VVVVVV V V VV I I VV V V VV =+=      =      =         = −− 2211 2 1 2121 2 1 0 0 ( )H AR ( )AN ( )H AN ( )ARCAY = 11 yAx H = C Y Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 ( ) [ ] ( ) ( ) ( ) ( ) ( )                Σ =Σ= − − −−− − H A H A rnxrmxrrm rnrxA AA H AAAmxn xnrn rxnrxr rmmxmxrnxnmxnmxm V V UUVUA 2 11 21 00 0 
75 SOLO Matrices we obtain: [ ] [ ] [ ]2121 2 11 21 00 0 AAAAH A H AA AA VCVCVV V V UUY  =              Σ or: ( ) ( )rnpxApxn rnnx VC −=− 02 px(n-r) - constraints equivalent to condition Cpxn∈ℜ (Amxn H ) nxrrxrmxr ApxnAApxm VCUY 111 =Σ pxr - independent equations pxm – unknowns since r ≤ m → # Eq. ≤ # Unknown ( ) [ ] ( ) ( ) ( ) ( ) ( )                Σ =Σ= − − −−− − H A H A rnxrmxrrm rnrxA AA H AAAmxn xnrn rxnrxr rmmxmxrnxnmxnmxm V V UUVUA 2 11 21 00 0  General Solution of YpxmAmxn = Cpxn ( )H AR ( )AN ( )H AN ( )ARCAY = C Y Null Space of A Ker (A) span by VA2 T Row Space of A span by VA1 T Column Space of A span by UA1 Left Null Space of A span by UA2 C has to be in the Row Space of A H AC ( ) ( ) Ο/=∩ ∈ ANC orARC HLet post-multiply by and using: [ ]21 VVpxnmxnpxm CAY = or: [ ] [ ]21 1 21 00 0 AA A AA VCVCUUY  =     Σ
76 SOLO Matrices Since UA1 T UA1=Ir & UA1 T UA2 = 0 the General Solution of YpxmAmxn = Cpxn is: where Xpx(m-r) is any px(m-r) matrix, i.e. we used all px(m-r) degrees of freedom. General Solution of YpxmAmxn = Cpxn nxrrxrmxr ApxnAApxm VCUY 111 =Σ pxr - independent equations pxm – unknowns since r ≤ m → # Eq. ≤ # Unknown This equation is a Necessary and Sufficient Condition for any solutions of equation YpxmAmxn = Cpxn. It is equivalent to Cpxn∈ℜ (Amxn) or Cpxn ∩ N (AT ) = ∅. If this condition is fulfilled, then from we have nxp unknowns ≥ rxp independent equations, that means (n-r)xp degrees of freedom. H ApxnAApxm nxrrxrmxr VCUY 111 =Σ 1 111 − Σ= rxrnxrmxr AApxnApxm VCUY px(n-r) - constraints equivalent to condition Cpxn∈ℜ (Amxn H )( ) ( )rnpxApxn rnnx VC −=− 02 ( ) ( ) H A any rmpx H AAApxnpxm xmrmrxmrxrnxr UXUVCY −− − +Σ= 21 1 11  ( )H AR ( )AN ( )H AN ( )ARCAY = C Y Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 C has to be in the Row Space of A H AC ( ) ( ) Ο/=∩ ∈ ANC orARC H H AC
77 SOLO Matrices Check: ( ) ( ) ( ) ( ) [ ] ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )[ ] ( ) ( ) ( ) ( ) ( ) ( )[ ] ( ) ( ) ( ) ( ) ( ) [ ] ( ) pxnH A H A AApxnH A H A ApxnApxn H A H A rnpxApxnH A H A rnxrmxrrm rnrxA rmpxAApxn H A H A rnxrmxrrm rnrxA I A H ArmpxA H AAApxnA H Armpx I A H AAApxn H A H A rnxrmxrrm rnrxA AA H Armpx H AAApxnmxnpxm C V V VVC V V VCVC V V VC V V XVC V V UUXUUVCUUXUUVC V V UUUXUVCAY xnrn rxn rnpxnxr xnrn rxn rnpx rnpxnxr xnrn rxn nxr xnrn rxnrxr rxrnxr xnrn rxnrxr rmmxxmrmrmmxrxmrxrnxrmxrxmrmmxrrxmrxrnxr xnrn rxnrxr rmmxmxrxmrmrxmrxrnxr =         =                   =         =                Σ Σ=                Σ         +Σ+Σ=                Σ +Σ= − − − − − −− − −−−− − −− − −−− − − − −−− − − − − − −−− − − − 2 1 21 2 1 0 21 2 1 1 2 111 11 2 11 22 0 21 1 11 0 1211 1 11 2 11 2121 1 11 0 00 0 00 0 00 0           General Solution of YpxmAmxn = Cpxn
78 SOLO Matrices where r is such that: Algorithm to solve YpxmAmxn = Cpxn: (1) Compute s.v.d. of Amxn and partition according to: ( ) 0,,, 21211 >≥≥≥=Σ rrA diagrxr σσσσσσ  (2) Check if: (3) If (2) is not true → no solution for (1) ( ) [ ] ( ) ( ) ( ) ( ) ( )                Σ =Σ= − − −−− − H A H A rnxrmxrrm rnrxA AA H AAAmxn xnrn rxnrxr rmmxmxrnxnmxnmxm V V UUVUA 2 11 21 00 0  General Solution of YpxmAmxn = Cpxn ( ) ( )rnpxApxn rnnx VC −=− 02 ( ) ( ) H A any rmpx H AAApxnpxm xmrmrxmrxrnxr UXUVCY −− − +Σ= 21 1 11  If (2) is true → px(m-r) solutions:
79 SOLO Matrices Moore-Penrose Pseudoinverse of A (continue - ): Define also Wpxm such that: ( ) ( )rmmxApxmrmpx UWX − =− 2: ( ) ( ) ( ) ( ) ( ) H AA H AA H AA H AAmxn † nxmnxn xnrnrmnxrxnnxrxnrnrmnxrxnnxr VVVVVVVVAAI −−−− =−+=− 22112211 ( ) [ ] ( ) ( ) ( ) ( ) ( ) H AAAH A H A rmxrnxrrn rmrxA AnA † nxm rxmrxrnxr xmrm rxmrxr rnnxmxr UV U U VVA 1 1 11 2 1 1 1 21 00 0 : − −−− − − Σ=                Σ = − −  Therefore:  ( )† nxmmxnmxm any pxm † nxmpxnpxm AAIWACY −+= Note: By writing the solution this way we lose the fact that we have only px(m-r) different solutions as we have seen. If ( ) ( )rnpxApxn rnnx VC − =− 02 General Solution of YpxmAmxn = Cpxn the solution of YpxmAmxn = Cpxn is ( ) ( ) H A any rmpx H AAApxnpxm xmrmrxmrxrnxr UXUVCY −− − +Σ= 21 1 11  Check: ( ) ( ) ( ) H A X Apxm H AAApxnpxm xmrm rmpx rmmxrxmrxrnxr UUWUVCY − − − +Σ= − 221 1 11 
80 SOLO Matrices Moore-Penrose Pseudoinverse of A (continue - ): Solutions exists iff: i.e. the projection (Inxn – Anxm † Amxn) of C on N (A) is zero. General Solution of YpxmAmxn = Cpxn ( ) ( )  ( )† nxmmxnmxm any pxm † nxmpxn H A any rmpx H AAApxnpxm AAIWAC UXUVCY xmrmrxmrxrnxr −+= +Σ= −− − 21 1 11  ( ) ( ) ( ) ( ) pxn H AApxnmxn † nxmnxnpxn xnrn rnpx rnnx VVCAAIC 02 0 2 ==− − − −  ( )H AR ( )AN ( )H AN ( )ARCAY = C pxm Y Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 C has to be in the Row Space of A H AC ( ) ( ) orANC orARC H Ο/=∩ ∈ ( ) ( ) ( ) ( ) ( ) pxn H AApxn ANonprojection mxnnxmnxnpxn xnrn rnpx rnnx VVCAAIC 02 0 2 ==− − − − +    ( ) ( ) ( ) ( ) H Armpx ANonWanyofprojection nxmmxnmxmpxm xrrm H pxm UXAAIW −− + =− 2    pxmW H AC ( ) ( ) ( ) ( ) ( ) ( ) ( ) pxn H AApxn ANonprojection mxn † nxmnxnpxn H xnrn rnpx rnnx VVC AAIC orANC orARC 02 0 2 == − Ο/=∩ ∈ − − −     ††† Table of Contents
81 SOLO Matrices Particular case (1) r = m ≤ n: ( )[ ] ( )         Σ=Σ= − − H A H A mnmxAA H AAAmxn xnmn rxn mxmmxmnxnmxnmxm V V UVUA 2 1 1 0 ( ) [ ] ( ) ( ) 11 11 1 1 21 0 : −− − − =Σ=        Σ = − H nxmmxn H nxm H AAA H A xmmn A AnA † nxm AAAUVUVVA mxmrxrnxrmxm mxm mnnxnxm  ( )[ ] ( ) ( ) [ ] ( ) mxm H A xmmn A AAH A H A mnmxAA † nxmmxn IUVV V V UAA mxm mxm mnnxnxm xnmn mxn mxmmxm =        Σ         Σ= − − − − − 0 0 1 1 21 2 1  General Solution of YpxmAmxn = Cpxn Only if solutions exist.( ) ( )rnpxApxn rnnx VC − =− 02 In this case we have pxm unknowns and pxn equations – px(n-m) constraints = pxm independent equations, i.e. a unique solution: † nxmpxn H AAApxnpxm AC UVCY rxmrxrnxr = Σ= − 1 1 11 ( )H AR ( )AN ( ) 0=H AN ( )ARCAY = C Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 C has to be in the Row Space of A H AC ( ) ( ) ( ) ( ) ( ) ( ) pxn H AApxnmxn † nxmnxnpxn H xnrn rnpx rnnx VVCAAIC orANC orARC 02 0 2 ==− Ο/=∩ ∈ − − −  Table of Contents
82 SOLO Matrices Particular case (2) r = n ≤ n: pxn equations ≥ pxm unknowns, meaning px(m-n) constraints ( ) [ ] ( ) H A xnnm A AA H AAAmxn nxn nxn nmmxmxnnxnmxnmxm VUUVUA        Σ =Σ= − − 0 1 21  ( )[ ] ( ) H AAAH A H A nmnxAA † nxm nxmnxnnxn xmnm nxm nxnnxn UV U U VA 1 1 2 11 0: − − − Σ=         Σ= − ( )[ ] ( ) ( ) [ ] ( ) nxn H A xnnm A AAH A H A nmnxAAmxn † nxm IVUU U U VAA nxn nxn nmmxmxn xmnm nxm nxnnxn =        Σ         Σ= − − − − − 0 0 1 21 2 11  Since solutions always exist( ) ( )rnpxApxn rnnx VC − ≡− 02 pxm unknowns ≥ pxn equations, meaning px(m-n) degrees of freedom ( )H AR ( ) 0=AN ( )H AN ( )ARCAY = C Y Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 C has to be in the Row Space of A H AC ( ) CAAC † = ( ) H A UXAAIW 2 =− + ( ) ( ) ( ) ( ) ( ) ( ) pxn H AApxnmxn † nxmnxnpxn H xnrn rnpx rnnx VVCAAIC orANC orARC 02 0 2 ==− Ο/=∩ ∈ − − −  ( ) ( )  ( )† nxmmxnmxm any pxm † nxmpxn H A any rmpx H AAApxnpxm AAIWAC UXUVCY xmrmrxmrxrnxr −+= +Σ= −− − 21 1 11  General Solution of YpxmAmxn = Cpxn Table of Contents
83 SOLO Matrices Inverse of Partitioned Matrices ( ) ( ) ( ) ( )         −−− −−− =      − × − ××× − × − ×××× − × − × − ×××× − × − × − ××× − ×× ×× 11111 111111 mnnnnmmmnmmmmnnnnmmm mnnnnmmmmnnnnmmmmnnn mmnm mnnn BADCDCBADC BADCBADCBA CD BA if and exist. 1− ×nnA 1− ×nnC Let find the inverse of such that:      ×× ×× mmnm mnnn PN ML       ×× ×× mmnm mnnn CD BA       =            ×× ×× ×× ×× ×× ×× mmnm mnnn mmnm mnnn mmnm mnnn I I PN ML CD BA 0 0 Proof: nnnmmnnnnn INBLA ××××× =+1 ( ) nnnnnmmmmnnn ILDCBA ××× − ××× =−→ 1 2 nmnmmmnnnm NCLD ××××× =+ 0 nnnmmmnm LDCN ×× − ×× −=→ 1 3 mnmmmnmnnn PBMA ××××× =+ 0 mmmnnnmn PBAM ×× − ×× −=→ 1 4 mmmmmmmnnm IPCMD ××××× =+ ( ) mmmmmnnnnmmm IPBADC ××× − ××× =−→ 1
84 SOLO Matrices Inverse of Partitioned Matrices ( ) ( ) ( ) ( )         −−− −−− =      − × − ××× − × − ×××× − × − × − ×××× − × − × − ××× − ×× ×× 11111 111111 mnnnnmmmnmmmmnnnnmmm mnnnnmmmmnnnnmmmmnnn mmnm mnnn BADCDCBADC BADCBADCBA CD BA if and exist. 1− ×nnA 1− ×nnC Let find the inverse of such that:      ×× ×× mmnm mnnn PN ML       ×× ×× mmnm mnnn CD BA       =            ×× ×× ×× ×× ×× ×× mmnm mnnn mmnm mnnn mmnm mnnn I I PN ML CD BA 0 0 Proof (continue – 1): 1 ( ) nnnnnmmmmnnn ILDCBA ××× − ××× =− 1 2 nnnmmmnm LDCN ×× − ×× −= 1 3 4 mmmnnnmn PBAM ×× − ×× −= 1 ( ) mmmmmnnnnmmm IPBADC ××× − ××× =− 1 ( ) 11 − × − ×××× −= nmmmmnnnnn DCBAL ( ) 111 − × − ×××× − ×× −−= nmmmmnnnnmmmnm DCBADCN ( ) 11 − × − ×××× −= mnnnnmmmmm BADCP ( ) 111 − × − ×××× − ×× −−= mnnnnmmmmnnnmn BADCBAM q.e.d.
85 SOLO Matrices Inverse of Partitioned Matrices       =            ×× ×× ×× ×× ×× ×× mmnm mnnn mmnm mnnn mmnm mnnn I I CD BA PN ML 0 0 From: ( ) 1− ××××× −=→ mmmnnmmmmm CBNIPmmmmmmmnnm ICPBN ××××× =+we get: ( ) 11 − × − ×××× −= mnnnnmmmmm BADCPSubstitute: and: ( ) 111 − × − ×××× − ×× −−= nmmmmnnnnmmmnm DCBADCN ( ) ( ) 1111111 − ×× − × − ×××× − × − × − × − ××× −+=− mmmnnmmmmnnnnmmmmmmnnnnmmm CBDCBADCCBADC to obtain: By inter-changing , in this identity, we obtain:nmmnnnmm DBAC ×××× ↔↔ , ( ) ( ) 1111111 − ×× − × − ×××× − × − × − × − ××× −+=− nnnmmnnnnmmmmnnnnnnmmmmnnn ADBADCBAADCBA
86 SOLO Pre-multiplying this identity by we obtain Matrices Inverse of Partitioned Matrices Let prove the identity: ( ) ( ) 111111 − ×× − × − ××× − × − ×××× − × −=− mmmnnmmmmnnnmnnnnmmmmnnn CBDCBABADCBA ( ) ( ) 1111 − × − ×××× − ×× − ××× −− mnnnnmmmmnnnnmmmmnnn BADCBADCBA Proof: ( )( ) 1111 − × − ×××× − ×× − ××× −−≡ mnnnnmmmmnnnnmmmmnmn BADCBADCBB ( )( ) 11111 − ×× − × − ×××× − ××× − ×× ≡−−≡ mmmnmnnnnmmmmnnnnmmmmmmn CBBADCBADCCB ( ) 11 − × − ××× − nmmmmnnn DCBA ( ) ( ) ( ) ( ) 111 111111 − ×× − × − ××× − × − ×××× − ×× − ××× − × − ××× −≡ −−− mmmnnmmmmnnn mnnnnmmmmnnnnmmmmnnnnmmmmnnn CBDCBA BADCBADCBADCBA q.e.d. ( ) ( ) 111111 − × − ×××× − × − ×× − × − ××× −=− nmmmmnnnnmmmnnnmmnnnnmmm DCBADCADBADC By inter-changing ,in the first identity, we obtain:nmmnnnmm DBAC ×××× ↔↔ , ( ) ( ) 111111 − ×× − × − ××× − × − ×××× − × −=− nnnmmnnnnmmmnmmmmnnnnmmm ADBADCDCBADC q.e.d.
87 SOLO Matrices Inverse of Partitioned Matrices By using the identities: ( ) ( ) 111111 − × − ×××× − × − ×× − × − ××× −=− nmmmmnnnnmmmnnnmmnnnnmmm DCBADCADBADC We obtain: ( ) ( ) ( ) ( )         −−− −−− =      − × − ××× − × − ×××× − × − × − ×××× − × − × − ××× − ×× ×× 11111 111111 mnnnnmmmnmmmmnnnnmmm mnnnnmmmmnnnnmmmmnnn mmnm mnnn BADCDCBADC BADCBADCBA CD BA if and exist. 1− ×nnA 1− ×nnC ( ) ( ) ( ) ( )         −−− −−−+ =      − × − ××× − ×× − × − ××× − × − ×××× − × − ×× − × − ×××× − × − × − ×× ×× 11111 111111111 mnnnnmmmnnnmmnnnnmmm mnnnnmmmmnnnnnnmmnnnnmmmmnnnnn mmnm mnnn BADCADBADC BADCBAADBADCBAA CD BA ( ) ( ) 1111111 − ×× − × − ×××× − × − × − × − ××× −+=− nnnmmnnnnmmmmnnnnnnmmmmnnn ADBADCBAADCBA
88 SOLO Matrices Inverse of Partitioned Matrices If and exist, performing the computation M-1 M, we can prove: 1− ×nnA 1− ×nnC         − =      − × − ×× − × × − × − ×× ×× 111 11 00 mmnnnmmm mnnn mmnm mnnn CADC A CD A 1 2         − =      − ×× − ×× − × − × − ×× ×× 1 1111 00 mmnm mmmnnnnn mmnm mnnn C CBAA C BA 3         =      − ×× × − × − ×× ×× 1 11 0 0 0 0 mmnm mnnn mmnm mnnn C A C A 4 If and : T nnnn AA ×× = T nnnn CC ×× = ( ) ( ) ( ) ( )         −−− −−− =      − × − ××× − × − ×××× − × − × − ×××× − × − × − ××× − ×× ×× 11111 111111 mnnnnm T mmnm T mmmnnnnm T mm mnnnnm T mmmnnnnm T mmmnnn mmnm T mnnn BABCBCBABC BABCBABCBA CB BA Because this is a symmetric matrix ( ) ( ) 111111 − ×× − × − ××× − × − ×××× − × −=− nnnm T mnnnnm T mmnm T mmmnnnnm T mm ABBABCBCBABC Also: ( ) ( ) 1111111 − ×× − × − ×××× − × − × − × − ××× −+=− mmmnnm T mmmnnnnm T mmmmmnnnnm T mm CBBCBABCCBABC
89 SOLO Matrices Inverse of Partitioned Matrices If m=n and and also exist: 1− ×nnB 1− ×nn D5 ( ) ( )[ ] 111111 − × − ×× − ××× − × − ×××× − × −−=−− nnnnnnnnnnnnnnnnnnnnnnnn ABBADCBADCBA ( ) ( )[ ] 111111 − × − ×× − ××× − × − ×××× − × −−=−− nnnnnnnnnnnnnnnnnnnnnnnn CDDCBADCBADC also we obtain ( ) ( ) ( ) ( )         −− −− =      − × − ××× − × − ××× − × − ××× − × − ××× − ×× ×× 1111 11111 nnnnnnnnnnnnnnnn nnnnnnnnnnnnnnnn nnnn nnnn BADCCDAB ABCDDCBA CD BA Table of Contents
90 SOLO Matrices Matrix Inverse Lemmas Identities ( ) ( ) 1111111 − ×× − × − ×× − ×× − × − × − ×××× +−=+ mmmnnmmmmnnnnmmmmmmnnnnmmm CBDCBADCCBADC1 Proof: ( ) ( ) 1111111 − ×× − × − ×××× − × − × − × − ××× −+=− mmmnnmmmmnnnnmmmmmmnnnnmmm CBDCBADCCBADC In the identity: substitute by . 1− ×nnA nnA ×− Substitute by and by in (1). 1− ×nnAnnA × mmC × 1− ×mmC2 ( ) ( ) mmmnnmmmmnnnnmmmmmmnnnnmmm CBDCBADCCBADC ×× − ××××××× − × − ×× − × +−=+ 1111 Substitute in (1) nnnn IA ×× =3 ( ) ( ) 111111 − ×× − × − ×××× − × − × − ××× +−=+ mmmnnmmmmnnnnmmmmmmnnmmm CBDCBIDCCBDC Substitute in (2) nnnn IA ×× =4 ( ) ( ) mmmnnmmmmnnnnmmmmmmnnmmm CBDCBIDCCBDC ×× − ××××××× − ×× − × +−=+ 111
91 SOLO Matrices Matrix Inverse Lemmas Identities 5Substitute in (1), (2), (3), (4) by . (We don’t assume symmetric and )nmD × nm T B × nnA × mmC × ( ) ( ) 1111111 − ×× − × − ×× − ×× − × − × − ×××× +−=+ mmmnnm T mmmnnnnm T mmmmmnnnnm T mm CBBCBABCCBABC ( ) ( ) mmmnnm T mmmnnnnm T mmmmmnnnnm T mm CBBCBABCCBABC ×× − ××××××× − × − ×× − × +−=+ 1111 ( ) ( ) 111111 − ×× − × − ×××× − × − × − ××× +−=+ mmmnnm T mmmnnnnm T mmmmmnnm T mm CBBCBIBCCBBC ( ) ( ) mmmnnm T mmmnnnnm T mmmmmnnm T mm CBBCBIBCCBBC ×× − ××××××× − ×× − × +−=+ 111 From this we get: 6Substitute in (3) mmmm IC ×× = ( ) ( ) mnnmmnnnnmmmmnnmmm BDBIDIBDI × − ××××× − ××× +−=+ 11 ( ) ( ) ( )( ) ( ) ( ) ( ) ( ) ( ) ( ) mnnmmnnmmmmnnmmmmnnmmmmnnmmm mnnmmmmnnmmnnmmmmnnmmmmnnmmm mnnmmmmmmnnmmnnnnm BDBDIBDIBDIBDI BDIBDBDIBDIBDI BDIIBDBID ×× − ××× − ×××××× − ××× − ××××× − ××× − ×××××× − ××××× − ×××× +=+−++= +=+−++= +−=+ 111 111 11
92 SOLO Matrices Matrix Inverse Lemmas Identities 7Substitute in the identity ( ) ( ) 111111 − ×× − × − ××× − × − ×××× − × −=− mmmnnmmmmnnnmnnnnmmmmnnn CBDCBABADCBA nnnn IA ×× −= and mmmm IC ×× = to obtain: ( ) ( ) mnnmmnnnmnnmmmmn BDBIBDIB × − ××× − ×××× +=+ 11 Pre-multiplying this by we get (6).nmD × By using a similar path with the identity ( ) ( ) 111111 − ×× − × − ××× − × − ×××× − × −=− nnnmmnnnnmmmnmmmmnnnnmmm ADBADCDCBADC nnnn IA ×× −= and mmmm IC ×× = to obtain:with ( ) ( ) nmmnnmmmnmmnnnnm DBDIDBID × − ××× − ×××× +=+ 11 Post-multiplying this by we get (6).mnB ×
93 SOLO Matrices By matrix manipulation we obtain: Matrix Inverse Lemmas Identities 8In the identity ( ) ( ) 111111 − ×× − × − ××× − × − ×××× − × −=− mmmnnmmmmnnnmnnnnmmmmnnn CBDCBABADCBA pre-multiplying Anxn by and post-multiplying by Cmxm we get: ( ) ( ) mnnmmmmnnnnnmmmnnnnmmmmn BDCBAACBADCB × − × − ××××× − × − ×××× −=− 1111 ( ) ( ) mnnnnmmmmnnnmnnnnmmmmmmn BADCBIBADCIB × −− ×× − ××× − × − ×× − ××× −=− 111111 Use now the identity ( ) ( ) 111111 − ×× − × − ××× − × − ×××× − × −=− nnnmmnnnnmmmnmmmmnnnnmmm ADBADCDCBADC Pre-multiplying by Cmxm and post-multiplying by Anxn we get: ( ) ( ) nmmnnnnmmmmmnnnmmmmnnnnm DBADCCADCBAD × − × − ××××× − × − ×××× −=− 1111 By matrix manipulation we obtain ( ) ( ) nmmmmnnnnmmmnmmmmnnnnnnm DCBADIDCBAID × −− ×× − ××× − × − ×× − ××× −=− 111111 Table of Contents
94 SOLO Matrices Matrix Schwarz Inequality ( ) ( ) ( )QPPPQPQQ TTTTT 1− ≥ Table of Contents Hermann Amandus Schwarz 1843 - 1921 yxyx ≤>< , Let x, y be the elements of an Inner Product space X, than : This is the Schwarz Inequality. Let Pmxn and Qmxl be two matrices such that PT P is nonsingular, then: ( ) ( ) ( ) CxxQPPPQPxxQQx TTTTTTT ∈∨≥ −1 i.e.,: Furthermore equality holds if and only if exists a matrix Snxl such that Q = P S. Proof: Start from the inequality: and choose( ) ( ) 0≥−− SPQSPQ T ( ) ( )QPPPS TT 1− = ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) 0 1 1111 ≥−= +−−= +−−=−− − −−−− QPPPQPQQ QPPPPPPPPQQPPPPQQPPPPQQQ SPPSQPSSPQQQSPQSPQ TTTTT TTTTTTTTTTTT TTTTTTT The inequality becomes equality if and only if : that is equivalent with ( ) ( ) 0=−− SPQSPQ T SPQ =
95 SOLO Matrices Trace of a Square Matrix The trace of a square matrix is defined as ( ) ( )T nn n i iinn AtraceaAtrace × = × == ∑1 : q.e.d. ( ) ( )ABtraceBAtrace =1 Proof: ( ) ∑ ∑= =         = n i n j jiij baBAtrace 1 1 ( ) ( )BAtracebaabABtrace n i n j jiij n j n i ijji ==      = ∑∑∑ ∑ = == = 1 11 1 ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )ABtraceBAtraceBAtraceABtraceABtraceBAtrace TTTT 111 =≠===2 Proof: ( ) ( ) ( )ABtraceBAtracebabaBAtrace n i n j jiij n i n j ijij T ==        ≠        = ∑ ∑∑ ∑ = == = 1 11 1 ( ) ( )T n j n i ijij T BAtraceabABtrace =      = ∑ ∑= =1 1 q.e.d.
96 SOLO Matrices Trace of a Square Matrix The trace of a square matrix is defined as ( ) ( )T nn n i iinn AtraceaAtrace × = × == ∑1 : 3 Proof: q.e.d. ( ) ( ) ( )∑= − == n i i APAPtraceAtrace 1 1 λ where P is the eigenvector matrix of A related to the eigenvalue matrix Λ of A by           =Λ= n PPPA λ λ    0 01 ( ) ( ) ( ) ( )AtraceAPPtracePAPtrace == −− 1 1 1           =Λ= n PPPA λ λ    0 01           =Λ=→ − n PAP λ λ    0 01 1 ( ) ( ) ∑= − =Λ=→ n i itracePAPtace 1 1 λ
97 SOLO Matrices Trace of a Square Matrix The trace of a square matrix is defined as ( ) ( )T nn n i iinn AtraceaAtrace × = × == ∑1 : Proof: q.e.d. Definition 4 ( )AtraceA ee =det ( )AtraceA eeePe P PePPePe n i i ====== ∑=ΛΛΛ−Λ− 1 detdetdet det 1 detdetdetdetdet 11 λ If aij are the coefficients of the matrix Anxn and z is a scalar function of aij, i.e.: ( ) njiazz ij ,,1, == then is the matrix nxn whose coefficients i,j areA z ∂ ∂ nji a z A z ijij ,,1,: = ∂ ∂ =      ∂ ∂ (see Gelb “Applied Optimal Estimation”, pg.23)
98 SOLO Matrices Trace of a Square Matrix The trace of a square matrix is defined as ( ) ( )T nn n i iinn AtraceaAtrace × = × == ∑1 : Proof: q.e.d. 5 ( ) ( ) ( ) A Atrace I A Atrace T n ∂ ∂ == ∂ ∂ 1 ( )    = ≠ == ∂ ∂ =      ∂ ∂ ∑= ji ji a aA Atrace ij n i ii ijij 1 0 1 δ 6 ( ) ( ) ( ) ( ) nmmnTTT RBRCCBBC A BCAtrace A ABCtrace ×× ∈∈== ∂ ∂ = ∂ ∂ 1 Proof: ( ) ( ) ( )[ ]ij T ji m p pijp ik jl n l m p n k klpklp ijij BCBCbcabc aA ABCtrace === ∂ ∂ =      ∂ ∂ ∑∑∑∑ = = = = = = 11 1 1 q.e.d. 7 If A, B, C ∈ Rnxn ,i.e. square matrices, then ( ) ( ) ( ) ( ) ( ) ( ) TTT CBBC A BCAtrace A CABtrace A ABCtrace == ∂ ∂ = ∂ ∂ = ∂ ∂ 11
99 SOLO Matrices Trace of a Square Matrix The trace of a square matrix is defined as ( ) ( )T nn n i iinn AtraceaAtrace × = × == ∑1 : Proof: q.e.d. 8 ( )( ) ( )( ) ( )( )( ) nmmn TTT RBRCBC A ABCtrace A BCAtrace A ABCtrace ×× ∈∈= ∂ ∂ = ∂ ∂ = ∂ ∂ 721 9 ( )( ) ( )( ) ( )( ) BC A BCAtrace A CABtrace A ABCtrace TTT 811 = ∂ ∂ = ∂ ∂ = ∂ ∂ If A, B, C ∈ Rnxn ,i.e. square matrices, then 10 ( ) T A A Atrace 2 2 = ∂ ∂ ( ) ( ) ( )ij T jiji n l n m mllm ijijij Aaaaa aa Atrace A Atrace 2 1 1 22 =+=      ∂ ∂ = ∂ ∂ =      ∂ ∂ ∑∑= = 11 ( ) ( ) 1− = ∂ ∂ kT k Ak A Atrace Proof: ( ) ( ) ( ) ( ) ( ) 1111 −−−− =+++= ∂         ⋅∂ = ∂ ∂ kT k kTkTkT k k AkAAA A AAAtrace A Atrace       q.e.d.
100 SOLO Matrices Trace of a Square Matrix The trace of a square matrix is defined as ( ) ( )T nn n i iinn AtraceaAtrace × = × == ∑1 : Proof: q.e.d. 12 ( ) T A A e A etrace = ∂ ∂ ( ) ( ) ( ) T A n k n k kT n kk kT n n k k n n k k n A eA k A k k k A trace Ak A trace AA etrace ===      ∂ ∂ =      ∂ ∂ = ∂ ∂ ∑ ∑∑∑ = = →∞ →− − →∞ = →∞ = →∞ 1 0 1 1 00 ! 1 lim ! lim ! lim ! lim 13 ( )( ) ( )( ) ( ) ( ) ( )( ) ( )( ) ( ) ( ) ( )( ) ( )( ) ( ) TT TTTTTTTTT TTTTT TTT BACBAC A ACABtrace A BACAtrace A ABACtrace A CABAtrace A BACAtrace A CABAtrace A ACABtrace A BACAtrace A ABACtrace += ∂ ∂ = ∂ ∂ = ∂ ∂ = ∂ ∂ = ∂ ∂ = ∂ ∂ = ∂ ∂ = ∂ ∂ = ∂ ∂ 111 21 11 ( ) ( ) ( ) ( ) ( ) ( ) TTTT TTT BACBACCABBAC A ABACtrace A ABACtrace A ABACtrace +=+== ∂ ∂ + ∂ ∂ = ∂ ∂ + 86 2 2 1 1 Proof: q.e.d. 14 ( ) ( )( ) A A AAtrace A AAtrace TT 2 13 = ∂ ∂ = ∂ ∂
101 SOLO Matrices Trace of a Square Matrix The trace of a square matrix is defined as ( ) ( )T nn n i iinn AtraceaAtrace × = × == ∑1 : Proof: 15 ( ) ( )TTTTT ABBAABBA A ABAtrace +=+= ∂ ∂ Table of Contents ( ) ( ) ( )ij TTTT n l jlli n k kijk n l n l n k klmklm ijijij ABBAbababaa aa ABAtrace A ABAtrace +=+=      ∂ ∂ = ∂ ∂ =      ∂ ∂ ∑∑∑∑∑ === = = 111 1 1 q.e.d. 16 ( ) TTTTTT CABBAC A ABACtrace += ∂ ∂ ( ) ( ) ( )ij TTTTTT n l n r lirljr n k n m mikmjk n l n r rljrli n k n m mikmjk n l n k n m n r rlmrkmlk ijijij CABBACcabbac abcbacabac aa ABACtrace A ABACtrace +=+= +=      ∂ ∂ = ∂ ∂ =      ∂ ∂ ∑∑∑∑ ∑∑∑∑∑∑∑∑ = == = = == == = = = 1 11 1 1 11 11 1 1 1 Proof: q.e.d.
102 SOLO References [1] Pease, “Methods of Matrix Algebra” ,Mathematics in Science and Engineering, Vol.16, Academic Press, 1965 Matrices [2] S. Hermelin, “Robustness and Sensitivity Design of Linear Time-Invariant Systems” PhD Thesis, Stanford University, 1986 Table of Contents
January 6, 2015 103 SOLO Technion Israeli Institute of Technology 1964 – 1968 BSc EE 1968 – 1971 MSc EE Israeli Air Force 1970 – 1974 RAFAEL Israeli Armament Development Authority 1974 – 2013 Stanford University 1983 – 1986 PhD AA Matrices
104 SOLO Derivatives of Matrices Matrices ljik ij kl a a δδ= ∂ ∂ For vector forms j i ijii i i y x y x y x y x y x y x ∂ ∂ =      ∂ ∂ ∂ ∂ =      ∂ ∂ ∂ ∂ =      ∂ ∂ ::: We have the following expressions: ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )( ) ( ) ( ) ( ) ( ) ( )HH TT XX XX XXtraceX XXtraceXX XXXX YXYXYX YXYXYX XtraceXtrace YXYX XX constAifA ∂=∂ ∂=∂ ∂=∂ ∂=∂ ∂−=∂ ∂⊗+⊗∂=⊗∂ ∂⋅+⋅∂=⋅∂ ∂=∂ ∂+∂=+∂ =∂ ==∂ − − −−− 1 1 111 detln detdet 0 αα
105 SOLO Derivatives of Determinants Matrices ( ) ( )       ∂ ∂ = ∂ ∂ − x Y YtraceY x Y 1 det det ( ) ( )             ∂ ∂       ∂ ∂ −       ∂ ∂       ∂ ∂ +             ∂ ∂ ∂ ∂ = ∂ ∂ ∂ ∂ −− −− − x Y Y x Y Ytrace x Y Ytrace x Y Ytrace x x Y YtraceY x x Y 11 11 1 det det General Form
106 SOLO Derivatives of Determinants Matrices ( ) ( )( ) ( ) ( ) ( ) ( )( ) ( )( ) 11 1 detdet det det det det det −− − == ∂ ∂ = ∂ ∂ = ∂ ∂ ∑ TT ij k jk ik T XBXAXBXA X BXA XX X X XX X X δ Linear Form Square Forms If X is Square and Invertible, then ( ) ( ) TT T XXAX X XAX − = ∂ ∂ det2 det If X is Not Square but A is Symmetric, then ( ) ( ) ( ) 1 det2 det − = ∂ ∂ XAXXAXAX X XAX TT T If X is Not Square and A is Not Symmetric, then ( ) ( ) ( ) ( )[ ]11 det det −− += ∂ ∂ XAXXAXAXXAXAX X XAX TTTT T
107 SOLO Derivatives of Determinants Matrices ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) Tk k T T T T T XXk X X XX X X X X XX X X XX − − = ∂ ∂ == ∂ ∂ −= ∂ ∂ = ∂ ∂ det det 22 detln 2 detln 2 detln 1T1- † † Nonlinear Form
108 SOLO Derivatives of an Inverse Matrices 11 1 −− − ∂ ∂ −= ∂ ∂ Y x Y Y x Y From this it follows ( ) ( ) ( ) ( ) ( ) ( )( ) ( ) ( ) ( )( ) ( ) ( )( )T T T TTT T jlki ij kl AXAX X AXtrace XABX X BXAtrace XX X X XbaX X bXa XX X X 11 1 11 1 11 1 1 11 1 det det −− − −− − −− − −− − −− − ++−= ∂ +∂ −= ∂ ∂ −= ∂ ∂ −= ∂ ∂ −= ∂ ∂
109 SOLO Derivatives of Matrices, Vectors and Scalar Forms Matrices First Order ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )ij nm mjin mn ij T ij mn njim mn ij ij ij T TTT T TT T T TT AJA X AX AJA X AX J X X aa X aXa X aXa ab X bXa ba X bXa a x xa x ax == ∂ ∂ == ∂ ∂ = ∂ ∂ = ∂ ∂ = ∂ ∂ = ∂ ∂ = ∂ ∂ = ∂ ∂ = ∂ ∂ δ δ n mJ mn ↑ ←                 = 000 010 000     
110 SOLO Derivatives of Matrices, Vectors and Scalar Forms Matrices Second Order ( ) ( ) ( ) ( )[ ] ( ) ( ) ( ) ( ) ( ) ( ) ( ) jlikkl ijijijT ij T ilkjkl T lj ij kl T TTT T TT TT kl kl klmn mnkl ij JXBJJBX X XBX XBBX X XBX bxBCDdxDCB x dxDCbxB bccbX X cXXb XXX X δδ δδ =+= ∂ ∂ += ∂ ∂ +++= ∂ ++∂ += ∂ ∂ = ∂ ∂ ∑∑ 2 n mJ mn ↑ ←                 = 000 010 000     
111 SOLO Derivatives of Matrices, Vectors and Scalar Forms Matrices Second Order (continue) ( ) ( ) ( ) ( ) ( )[ ] ( )( ) TTT TTT TT T T bcbXDDcbXDcbX X bcXDcbXD X cXDXb xBB x xBx ++=++ ∂ ∂ += ∂ ∂ += ∂ ∂ Assume W is symmetric ( ) ( )[ ] ( ) ( ) ( )[ ] ( ) ( ) ( )[ ] ( ) ( ) ( )[ ] ( ) ( ) ( )[ ] ( ) TT T T T TT ssAxWsAxWsAx A sAxWsAxWsAx x sxWsxWsx s sxWsxWsx x sAxWAsAxWsAx s −−=−− ∂ ∂ −=−− ∂ ∂ −−=−− ∂ ∂ −=−− ∂ ∂ −−=−− ∂ ∂ 2 2 2 2 2
112 SOLO Derivatives of Matrices, Vectors and Scalar Forms Matrices Higher Order and Nonlinear ( ) ( )∑ − = −− = ∂ ∂ 1 0 1 n r kl rnijr ij kl n XJX X X [ ] ( ) ( ) ( )[ ] ( ) ( ) ( )[ ]∑ ∑ − = −−−− −− − = += ∂ ∂ = ∂ ∂ 1 0 11 1 1 0 n r TrnTnTrrTnTrnnTnT Trn n r TTrnT XbaXXXXbaXbXXa X XbaXbXa X

More Related Content

PPT
2 classical field theories
Solo Hermelin
 
PPT
Inner outer and spectral factorizations
Solo Hermelin
 
PPT
Legendre functions
Solo Hermelin
 
PPT
Prime numbers
Solo Hermelin
 
PPT
Calculus of variation problems
Solo Hermelin
 
PPT
Analytic dynamics
Solo Hermelin
 
PPT
3 recursive bayesian estimation
Solo Hermelin
 
PPT
Rotation in 3d Space: Euler Angles, Quaternions, Marix Descriptions
Solo Hermelin
 
2 classical field theories
Solo Hermelin
 
Inner outer and spectral factorizations
Solo Hermelin
 
Legendre functions
Solo Hermelin
 
Prime numbers
Solo Hermelin
 
Calculus of variation problems
Solo Hermelin
 
Analytic dynamics
Solo Hermelin
 
3 recursive bayesian estimation
Solo Hermelin
 
Rotation in 3d Space: Euler Angles, Quaternions, Marix Descriptions
Solo Hermelin
 

What's hot (20)

PPT
Stabilization of linear time invariant systems, Factorization Approach
Solo Hermelin
 
PPT
Dyadics
Solo Hermelin
 
PPT
Matrices i
Solo Hermelin
 
PPTX
HERMITE SERIES
MANISH KUMAR
 
PPT
Equation of motion of a variable mass system3
Solo Hermelin
 
PPT
2 backlash simulation
Solo Hermelin
 
PDF
Introduction to Calculus of Variations
Delta Pi Systems
 
PPT
Gamma function
Solo Hermelin
 
PPT
Calculus of variations
sauravpatkotwar
 
PDF
Calculus of variations & solution manual russak
José Pallo
 
PPT
5 cramer-rao lower bound
Solo Hermelin
 
PPT
Calculus of variations
Solo Hermelin
 
PPTX
Seminar: Calculus of Variation
Subhajit Pramanick
 
PDF
Mech MA6351 tpde_notes
KIT-Kalaignar Karunanidhi Institute of Technology
 
PPT
Reduced order observers
Solo Hermelin
 
PDF
How to Solve a Partial Differential Equation on a surface
tr1987
 
PDF
Notions of equivalence for linear multivariable systems
Stavros Vologiannidis
 
PPT
4 stochastic processes
Solo Hermelin
 
PPT
Introduction to Mathematical Probability
Solo Hermelin
 
PDF
residue
Rob Arnold
 
Stabilization of linear time invariant systems, Factorization Approach
Solo Hermelin
 
Dyadics
Solo Hermelin
 
Matrices i
Solo Hermelin
 
HERMITE SERIES
MANISH KUMAR
 
Equation of motion of a variable mass system3
Solo Hermelin
 
2 backlash simulation
Solo Hermelin
 
Introduction to Calculus of Variations
Delta Pi Systems
 
Gamma function
Solo Hermelin
 
Calculus of variations
sauravpatkotwar
 
Calculus of variations & solution manual russak
José Pallo
 
5 cramer-rao lower bound
Solo Hermelin
 
Calculus of variations
Solo Hermelin
 
Seminar: Calculus of Variation
Subhajit Pramanick
 
Mech MA6351 tpde_notes
KIT-Kalaignar Karunanidhi Institute of Technology
 
Reduced order observers
Solo Hermelin
 
How to Solve a Partial Differential Equation on a surface
tr1987
 
Notions of equivalence for linear multivariable systems
Stavros Vologiannidis
 
4 stochastic processes
Solo Hermelin
 
Introduction to Mathematical Probability
Solo Hermelin
 
residue
Rob Arnold
 
Ad

Similar to Matrices ii (20)

PDF
Eigenvalue eigenvector slides
AmanSaeed11
 
PPTX
L04 Singular Value Decomposition Moore Penrose .pptx
ranjithkumar379904
 
PPTX
Eigenvalue problems-numerical methods.pptx
AnithaRaghavan3
 
PPT
lec17.ppt
Rai Saheb Bhanwar Singh College Nasrullaganj
 
PPTX
eigenvalue
Asyraf Ghani
 
PDF
Linear Algebra for AI & ML
Sandip Ladi
 
PPTX
Maths Assignment Help
Math Homework Solver
 
PDF
1- Matrices and their Applications.pdf
d00a7ece
 
PPTX
Maths Assignment Help
Maths Assignment Help
 
PPT
Eigen values and eigen vectors engineering
shubham211
 
ODP
Seismic data processing (mathematical foundations)
Amin khalil
 
PDF
An Introduction to Linear Algebra.pdf
Ashley Carter
 
PPT
Eigen value , eigen vectors, caley hamilton theorem
gidc engineering college
 
PPTX
Matrix PPT.pptx______________________________________________________________...
Vikrant168800
 
PPTX
MODULE_05-Matrix Decomposition.pptx
AlokSingh205089
 
PPT
Vcla - Inner Products
Preetshah1212
 
PDF
Linear Algebra
Ravinder Singh
 
PPTX
orthogonal.pptx
JaseSharma
 
PPTX
Maths-->>Eigenvalues and eigenvectors
Jaydev Kishnani
 
PDF
Partial midterm set7 soln linear algebra
meezanchand
 
Eigenvalue eigenvector slides
AmanSaeed11
 
L04 Singular Value Decomposition Moore Penrose .pptx
ranjithkumar379904
 
Eigenvalue problems-numerical methods.pptx
AnithaRaghavan3
 
lec17.ppt
Rai Saheb Bhanwar Singh College Nasrullaganj
 
eigenvalue
Asyraf Ghani
 
Linear Algebra for AI & ML
Sandip Ladi
 
Maths Assignment Help
Math Homework Solver
 
1- Matrices and their Applications.pdf
d00a7ece
 
Maths Assignment Help
Maths Assignment Help
 
Eigen values and eigen vectors engineering
shubham211
 
Seismic data processing (mathematical foundations)
Amin khalil
 
An Introduction to Linear Algebra.pdf
Ashley Carter
 
Eigen value , eigen vectors, caley hamilton theorem
gidc engineering college
 
Matrix PPT.pptx______________________________________________________________...
Vikrant168800
 
MODULE_05-Matrix Decomposition.pptx
AlokSingh205089
 
Vcla - Inner Products
Preetshah1212
 
Linear Algebra
Ravinder Singh
 
orthogonal.pptx
JaseSharma
 
Maths-->>Eigenvalues and eigenvectors
Jaydev Kishnani
 
Partial midterm set7 soln linear algebra
meezanchand
 
Ad

More from Solo Hermelin (20)

PPT
5 introduction to quantum mechanics
Solo Hermelin
 
PPT
Slide Mode Control (S.M.C.)
Solo Hermelin
 
PPT
Sliding Mode Observers
Solo Hermelin
 
PPT
Keplerian trajectories
Solo Hermelin
 
PPT
Anti ballistic missiles ii
Solo Hermelin
 
PPT
Anti ballistic missiles i
Solo Hermelin
 
PPT
12 performance of an aircraft with parabolic polar
Solo Hermelin
 
PPT
11 fighter aircraft avionics - part iv
Solo Hermelin
 
PPT
10 fighter aircraft avionics - part iii
Solo Hermelin
 
PPT
9 fighter aircraft avionics-part ii
Solo Hermelin
 
PPT
8 fighter aircraft avionics-part i
Solo Hermelin
 
PPT
6 computing gunsight, hud and hms
Solo Hermelin
 
PPT
4 navigation systems
Solo Hermelin
 
PPT
3 earth atmosphere
Solo Hermelin
 
PPT
2 aircraft flight instruments
Solo Hermelin
 
PPT
3 modern aircraft cutaway
Solo Hermelin
 
PPT
2Anti-aircraft Warhead
Solo Hermelin
 
PPT
1 susceptibility vulnerability
Solo Hermelin
 
PPT
15 sky cars
Solo Hermelin
 
PPT
14 fixed wing fighter aircraft- flight performance - ii
Solo Hermelin
 
5 introduction to quantum mechanics
Solo Hermelin
 
Slide Mode Control (S.M.C.)
Solo Hermelin
 
Sliding Mode Observers
Solo Hermelin
 
Keplerian trajectories
Solo Hermelin
 
Anti ballistic missiles ii
Solo Hermelin
 
Anti ballistic missiles i
Solo Hermelin
 
12 performance of an aircraft with parabolic polar
Solo Hermelin
 
11 fighter aircraft avionics - part iv
Solo Hermelin
 
10 fighter aircraft avionics - part iii
Solo Hermelin
 
9 fighter aircraft avionics-part ii
Solo Hermelin
 
8 fighter aircraft avionics-part i
Solo Hermelin
 
6 computing gunsight, hud and hms
Solo Hermelin
 
4 navigation systems
Solo Hermelin
 
3 earth atmosphere
Solo Hermelin
 
2 aircraft flight instruments
Solo Hermelin
 
3 modern aircraft cutaway
Solo Hermelin
 
2Anti-aircraft Warhead
Solo Hermelin
 
1 susceptibility vulnerability
Solo Hermelin
 
15 sky cars
Solo Hermelin
 
14 fixed wing fighter aircraft- flight performance - ii
Solo Hermelin
 

Recently uploaded (20)

PPTX
AP CHEM 1.2 Mass spectroscopy of elements
Mohamed Harb
 
PDF
ECG Practice from Passmedicine for MRCP Part 2 2024.pdf
AhmedAbdelmawla15
 
PDF
THE-VITAL-ROLE-OF-MITOCHONDRIAL-RESPIRATION-IN-PLANT-GROWTH-AND-DEVELOPMENT.pdf
SavithaRajagopal3
 
PPTX
BPharm_Hospital_Organization_Complete_PPT.pptx
Vijay Meti
 
PPTX
Presentation1 INTRODUCTION TO ENZYMES.pptx
JIBY2
 
PPT
Biochemestry- PPT ON Protein,Nitrogenous constituents of Urine, Blood, their ...
SuchitraRati1
 
PPT
Cell Structure Description and Functions
nctpcctv
 
PPTX
Preformulation.pptx Preformulation studies-Including all parameter
manjushamhaske
 
PPTX
HAEMATOLOGICAL DISEASES lack of red blood cells, which carry oxygen throughou...
pavanteja2396
 
PPTX
02_OpenStax_Chemistry_Slides_20180406 copy.pptx
nevartmooradian
 
PDF
The Future of Telehealth: Engineering New Platforms for Care (www.kiu.ac.ug)
publication11
 
PDF
Packaging materials of fruits and vegetables
SavithaRajagopal3
 
PPTX
LIPID & AMINO ACID METABOLISM UNIT-III, B PHARM II SEMESTER
ARUN KUMAR
 
PPTX
endocrine - management of adrenal incidentaloma.pptx
csushrut2511
 
PPTX
Toxicity Studies in Drug Development Ensuring Safety, Efficacy, and Global Co...
VaidehiVadhvana1
 
PPTX
Cells and Organs of the Immune System (Unit-2) - Majesh Sir.pptx
tkgauri1
 
PDF
Integrative Oncology: Merging Conventional and Alternative Approaches (www.k...
publication11
 
PPT
Enhancing Laboratory Quality Through ISO 15189 Compliance
mdsgift
 
PPTX
2currentelectricity1-201006102815 (1).pptx
faisalahemd13
 
PPTX
Platelet disorders - thrombocytopenia.pptx
AnnaMarie531858
 
AP CHEM 1.2 Mass spectroscopy of elements
Mohamed Harb
 
ECG Practice from Passmedicine for MRCP Part 2 2024.pdf
AhmedAbdelmawla15
 
THE-VITAL-ROLE-OF-MITOCHONDRIAL-RESPIRATION-IN-PLANT-GROWTH-AND-DEVELOPMENT.pdf
SavithaRajagopal3
 
BPharm_Hospital_Organization_Complete_PPT.pptx
Vijay Meti
 
Presentation1 INTRODUCTION TO ENZYMES.pptx
JIBY2
 
Biochemestry- PPT ON Protein,Nitrogenous constituents of Urine, Blood, their ...
SuchitraRati1
 
Cell Structure Description and Functions
nctpcctv
 
Preformulation.pptx Preformulation studies-Including all parameter
manjushamhaske
 
HAEMATOLOGICAL DISEASES lack of red blood cells, which carry oxygen throughou...
pavanteja2396
 
02_OpenStax_Chemistry_Slides_20180406 copy.pptx
nevartmooradian
 
The Future of Telehealth: Engineering New Platforms for Care (www.kiu.ac.ug)
publication11
 
Packaging materials of fruits and vegetables
SavithaRajagopal3
 
LIPID & AMINO ACID METABOLISM UNIT-III, B PHARM II SEMESTER
ARUN KUMAR
 
endocrine - management of adrenal incidentaloma.pptx
csushrut2511
 
Toxicity Studies in Drug Development Ensuring Safety, Efficacy, and Global Co...
VaidehiVadhvana1
 
Cells and Organs of the Immune System (Unit-2) - Majesh Sir.pptx
tkgauri1
 
Integrative Oncology: Merging Conventional and Alternative Approaches (www.k...
publication11
 
Enhancing Laboratory Quality Through ISO 15189 Compliance
mdsgift
 
2currentelectricity1-201006102815 (1).pptx
faisalahemd13
 
Platelet disorders - thrombocytopenia.pptx
AnnaMarie531858
 

Matrices ii

  • 1. 1 Matrices II SOLO HERMELIN Updated: 20.07.07http://www.solohermelin.com
  • 2. 2 SOLO Matrices Table of Content Singular Values Definitions Domain and Codomain of a Matrix A Properties of Square Orthogonal Matrices Definition of the Singular Values Geometric Interpretation of Singular Values Properties of Singular Values Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Householder Transformation Projection of a vector on a vector .b  a  111 min&min 1 nxmxnxmxn x xbxAd nx   −= Computation of Moore-Penrose Pseudoinverse Matrix, A † Properties of Moore-Penrose Pseudoinverse Matrix, A † Description of Projections Related to Moore-Penrose Pseudoinverse Particular case (1) r = n ≤ m: Particular case (2) r = m ≤ n:
  • 3. 3 SOLO Matrices Table of Content (continue – 1) General Solution of Amxn Xnxp = Bmxp Particular case (1) r = m ≤ n Particular case (2) r = n ≤ n General Solution of YpxmAmxn = Cpxn Particular case (1) r = m ≤ n Particular case (2) r = n ≤ n Inverse of Partitioned Matrices Matrix Inverse Lemmas Identities Matrix Schwarz Inequality Trace of a Square Matrix References
  • 4. 4 SOLO Matrices Singular Values Definitions: Any complex matrix A with n rows (r1, r2,…,rn) and m columns (c1,c2,…,cm) [ ]m n nxm ccc r r r A ,,, 21 2 1   =               = can be considered as a linear function (or mapping or transformation) for a m-dimensional domain to a n-dimensional codomain. ( ) ( ){ }AcodomyAdomxxAyA nxmxnxm ∈⇒∈= 11;: In the same way its conjugate transpose: [ ]H n HH H m H H H mxn rrr c c c A ,,, 21 2 1   =               = is a linear function (or mapping or transformation) for an-dimensional codomain to a m-dimensional domain. ( ) ( ){ }AcdomxAcodomyyAxA mxnx HH mxn ∈⇒∈= 111111 ;: Table of Contents
  • 5. 5 SOLO Matrices Domain and Codomain of a Matrix A The domain of A can be decomposed into orthogonal subspaces: ( ) ( ) ( )ANARAdom H ⊥ ⊕= ( )H AR ( )AN ( )H AN ( )AR xAy = 11 yAx H = ( )Adomxmx ∈1 11mxx ( )Acodomy nx ∈11 1nxyR (AH ) – is the row space of AH (dimension r) N (A) – is the null-space of A (x∈ N (A) ⇔ A x = 0) or the kernel of A (ker (A)) (dimension m-r) The codomain of A (domain of AH ) can be decomposed into orthogonal subspaces: ( ) ( ) ( )H ANARAcodom ⊥ ⊕= R (A) – is the column space of A (dimension r) N (AH ) – is the null-space of AH (dimension n-r) Singular Values Table of Contents
  • 6. 6 SOLO Hermitian = Symmetric if A has real components Hermitian Matrix: AH = A, Symmetric Matrix: AT = A Matrices Properties of Square Orthogonal Matrices Use Pease, “Methods of Matrix Algebra”, Mathematics in Science and Engineering Vol.16, Academic Press 1965 Definitions: Adjoint Operation (H): AH = (A*)T (* is complex conjugate and T is transpose of the matrix) Skew-Hermitian = Anti-Symmetric if A has real components. Skew-Hermitian: AH = -A, Anti-Symmetric Matrix: AT =-A Unitary Matrix: UH = U-1, Orthonormal Matix: OT = O-1 Unitary = Orthonormal if A has real components. Charles Hermite 1822 - 1901
  • 7. 7 SOLO Matrices Properties of Square Orthogonal Matrices (continue – 1) Lemma1: All the eigenvalues of a hermitian matrix H are real and the eigenvectors are orthogonal. Proof of Lemma1: Pre-multiply by :iii xHx λ= H ix i H iii H i xxHxx λ= and take the conjugate transpose: ( ) i H iii HH i H i H i xxxHxHxx * λ== This proves that the eigenvalues of H are real. Subtract those two equations: ( ) 00 ** ≠=→=− i H iiii H iii xxsincexx λλλλ From ( ) H ij H jj H j HH j H j xxHxHxHx λλ ==== * Pre-multiply by and post-multiply by and subtractiii xHx λ= H jx H ij H j xHx λ= ix ( ) 0=−→     = = i H jji i H jji H j i H jii H j xx xxHxx xxHxx λλ λ λ 0=→≠ i H jji xxλλIf If we can use the Gram-Schmidt procedure to obtain an eigenvector orthogonal to . ji λλ = jx~ ix ( ) ( ) i i H i j H i jj x xx xx xx −=~
  • 8. 8 SOLO Matrices Properties of Square Orthogonal Matrices (continue – 2) Lemma1: All the eigenvalues of a hermitian matrix H are real and the eigenvectors are orthogonal. Proof of Lemma1 (continue – 1): ( ) 0=−→     = = i H jji i H jji H j i H jii H j xx xxHxx xxHxx λλ λ λ 0=→≠ i H jji xxλλIf If we can use the Gram-Schmidt procedure to obtain an eigenvector orthogonal to . ji λλ = jx~ ix ( ) ( ) i i H i j H i jj x xx xx xx −=~ we can see that ( ) ( ) 0~ =−= i H i i H i j H i j H ij H i xx xx xx xxxx ( ) ( ) ( ) ( ) ( ) ( ) jii i H i j H i jiii i H i j H i jji i H i j H i jj xx xx xx xx xx xx xxH xx xx xHxH ~~ λλλλ =         −=−=−= q.e.d.
  • 9. 9 SOLO Matrices Properties of Square Orthogonal Matrices (continue – 3) Lemma2: Any hermitian matrix H can be factored in H = U Λ UH where Λ=diag (λ1,λ2,…,λn) and U is unitary i.e. U UH = UH U = In. Proof of Lemma2: Let normalize the orthogonal eigenvectors of H ;i.e. iii xxu /:= or H U = U Λ where U = [u1,u2,…,un] Because U is a square matrix having orthonormal columns, and is a square matrix, U is also a unitary matrix satisfying UH U=U UH =In. q.e.d. [ ] [ ]             = 000 00 00 ,,,,,, 2 1 2121      λ λ nn uuuuuuH
  • 10. 10 SOLO Matrices Properties of Square Orthogonal Matrices (continue – 4) Lemma3: A AH and AH A are hermitian matrices that have the same nonzero real positive eigenvalues. Proof of Lemma3: q.e.d. ( ) ( ) HHHHHH AAAAAA == i.e. A AH is hermitian, therefore the eigenvalues λi (A AH ) are real and positive according to Lemma 1. Suppose ui is a normalized eigenvector of λi (A AH ) ≠0 ( ) i H ii H uAAuAA λ= Pre-multiply by AH and define ( ) i H H i i uA AA v λ 1 := ( ) ( ) ( ) ( ) ( ) ( ) ( ) i H ii H H i i H H i H i i H H i HH ii HH vAAvAA AA uA AA AA uA AAuAAAuAAA λ λ λ λ λ =→ →=→= we get We can see that νi is the eigenvector of AH A and λi (A AH ) is the corresponding eigenvalue, meaning that both AH A and A AH have the same nonzero eigenvalues. From ( ) ( ) ( ) ( ) ( ) ( ) 02 2 >==→=→= i i i H i i H iH ii H i H ii HH ii H ii H v vA vv vAvA AAvvAAvAAvvAAvAA λλλ Therefore we can define ( ) 0: >= H ii AAλσ
  • 11. 11 SOLO Matrices Properties of Square Orthogonal Matrices (continue – 5) Lemma4: If U is a unitary matrix then all its eigenvalues have unit modulus. . Proof of Lemma4: form the inner product IUUUU HH == Consider the set of eigenvalues x1, x2, …, xn which we know to be complete and iii xUx λ= nixxIxxUxUxxx iiii H ii H ii HH ii H iii ,11 ** =∀==→=== λλλλλ q.e.d.
  • 12. 12 SOLO Matrices Properties of Square Orthogonal Matrices (continue – 6) Proof of Lemma5: Lemma5: Every unitary matrix U can be expressed as an exponential matrix: where H is hermitian (jH is skew-hermitian) jH eU = Since the eigenvalues of U have unit modulus; i.e. we can writenii ,11 =∀=λ niej Hij HiHii ,1sincos =∀=+= λ λλλ ( ){ } jHjjj eSeeediagSU HnHH == −1 ,,, 21 λλλ  ( ){ } 1 21 ,,, − = SdiagSH HnHH λλλ where: q.e.d.
  • 13. 13 SOLO Matrices Properties of Square Orthogonal Matrices (continue – 7) Table of Contents Decomposition of Square Matrices: ( ) ( ) ( ) ( )    − − ++=−++= HHHH AA j jAAAAAAA 22 1 2 1 2 1 ( ) ( )H H H AAAA +=    + 2 1 2 1 ( ) ( ) ( )HH H H AAAAAA −−=−=      − 2 1 2 1 2 1 here: Hermitian Skew-Hermitian ( ) ( ) ( )HH H H AA j AA j AA j − − =−=    − − 222 Hermitian ( ) ( )      − − ++= HH AA j jAAA 22 1 the matrix generalization of the decomposition of a complex number in the real and imaginary part.
  • 14. 14 SOLO Matrices Lemma6: (6.1) Every complex nxm matrix of rank can be factored into: Definition of the Singular Values ( ) [ ] ( ) ( ) ( ) ( ) ( )                Σ =Σ= − − −−− − H H rmxrnxrrn rmrxH mxmnxmnxnnxm xmrm rxmrxr rnnxnxr V V UUVUA 2 11 21 00 0    where ( ) 0,,, 21211 >≥≥≥=Σ rrdiagrxr σσσσσσ  Unxn and Vmxm are unitary matrices, i.e.: [ ] [ ] ( ) ( )       =         ==         = −− rnxrn rxr H H H H H H I I UU U U UU U U UUUU 0 0 21 2 1 2 1 21 [ ] [ ] ( ) ( )       =         ==         = −− rmxrm rxr H H H H H H I I VV V V VV V V VVVV 0 0 21 2 1 2 1 21 (6.2) σi i=1,…,r are the positive square roots of the nonzero eigenvalues of AH A or A AH and are called the singular values of A. (6.3) The dyadic expansion of A is: where ui and vi are the columns of U1 and V1 respectively. ∑= = r i H iii vuA 1 σ Singular Values
  • 15. 15 SOLO Matrices Lemma6 (continue – 1): Definition of the Singular Values (6.5) The columns of V are orthonormal eigenvectors of AH A: [ ] [ ] ( ) ( ) ( ) ( )        Σ = −−− − rmxrmxrrm rmrxH rxr VVVVAA 00 0 2 1 2121    i.e. the columns of V1 are the eigenvectors of the nonzero eigenvalues, and the columns of V2 are the eigenvectors of the zero eigenvalues of AH A. (6.6) The following relations exist between U1 and V1 1 111 1 111 − − Σ= Σ= rxrnxrmxr rxrmxrnxr UAV VAU H nxm nxm i.e. the columns of U1 are the eigenvectors of the nonzero eigenvalues, and the columns of U2 are the eigenvectors of the zero eigenvalues of A AH . (6.4) The columns of U are orthonormal eigenvectors of A AH : [ ] [ ] ( ) ( ) ( ) ( )        Σ = −−− − rmxrnxrrn rmrxH rxr UUUUAA 00 0 2 1 2121    Singular Values
  • 16. 16 SOLO Matrices Lemma6 (continue – 2): Definition of the Singular Values ( )H AR ( )AN ( )H AN ( )AR xAy = 11 yAx H = ( )Adomxmx ∈1 11mx x ( )Acodomy nx ∈11 1nxy (6.7) The columns U1 of form an orthonormal basis for the column space of A: ( ) ( )ARUR =1 The columns of U2 form an orthonormal basis for the nullspace of AH : ( ) ( ) ( )HH AANUR ker2 == The columns of V1 form an orthonormal basis for the column space of AH : ( ) ( )H ARVR =1 The columns of V2 form an orthonormal basis for the nullspace of A: ( ) ( ) ( )AANVR ker2 == Singular Values
  • 17. 17 SOLO Matrices Lemma6 (continue – 3): Proof of Lemma 6: Definition of the Singular Values and From Lemma 3 we have ( ) [ ] ( ) ( ) ( ) ( ) ( )                 Σ =Λ= − − −−− − H H rnxrnxrrn rnrxH nxnnxn H mxnnxm xnrn rxnrxr rnnxnxrnxn U U UUUUAA 2 1 2 1 211 00 0 ( ) [ ] ( ) ( ) ( ) ( ) ( )                 Σ =Λ= − − −−− − H H rmxrmxrrm rmrxH mxmmxmnxm H mxn xmrm rxmrxr rmmxmxrmxm V V VVVVAA 2 1 2 1 212 00 0 where ( ) 0,,, 21211 >≥≥≥=Σ rrdiagrxr σσσσσσ  and ( ) ( ) riAAAA H i H ii ,,2,10: =>== λλσ Those equations can be rewritten as: [ ] [ ] [ ] [ ]0 0 1121 1121 Σ= Σ= VVVAA UUUAA H H or 0 0 2 2 111 111 = = Σ= Σ= VAA UAA VVAA UUAA H H H H H U2 H V2 ( ) ( ) 00 00 22222 22222 =→== =→== VAVAVAVAAV UAUAUAUAAU HHH HHHHHH Singular Values
  • 18. 18 SOLO Matrices Lemma6 (continue – 4): Proof of Lemma 6 (continue – 1): Definition of the Singular Values 02 =UAH The columns of U2 form an orthonormal basis for the nullspace of AH : ( ) ( ) ( )HH AANUR ker2 == 02 =VA The columns of V2 form an orthonormal basis for the nullspace of A: ( ) ( ) ( )AANVR ker2 == 111 Σ= UUAA H The columns U1 of form an orthonormal basis for the column space of A: ( ) ( )ARUR =1 111 Σ= VVAAH The columns of V1 form an orthonormal basis for the column space of AH : ( ) ( )H ARVR =1 ( )H AR ( )AN ( )H AN ( )AR xAy = 11 yAx H = ( )Adomxmx ∈1 11mx x ( )Acodomy nx ∈11 1nx y Singular Values
  • 19. 19 SOLO Matrices Lemma6 (continue – 5): Proof of Lemma 6 (continue – 2): Definition of the Singular Values ( )H AR ( )AN ( )H AN ( )AR xAy = 11 yAx H = ( )Adomxmx ∈1 11mx x ( )Acodomy nx ∈11 1nx y From ( ) riuAuA AA v i H i i H H i i ,,2,1 11 : === σλ we have [ ] [ ]               = r r H r uuuAvvv σ σ σ /100 0/10 00/1 2 1 2121      or 1 111 − Σ= UAV H 11 1 1 2 11 1 111 2 111 Σ=ΣΣ=Σ= − Σ= − UUUAAVA UUAA H H riuuuAAvA iiii i i H i i ,,2,1 11 2 ==== σσ σσ from which 1 111 − Σ= VAU rivAu i i i ,,2,1 1 == σ 111 Σ=VAU H Singular Values
  • 20. 20 SOLO Matrices Lemma6 (continue – 6): Proof of Lemma 6 (continue – 3): Definition of the Singular Values ( )H AR ( )AN ( )H AN ( )AR xAy = 11 yAx H = ( )Adomxmx ∈1 11mx x ( )Acodomy nx ∈11 1nx y Using and let compute AH A V111 Σ=VAU H 02 =VA ( ) ( ) [ ] ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )         Σ =         =         = −−− − −−− − − − rmxrnxrrn rmrx nxm H nxm H nxm H nxm H nxmH H mxmnxm H nxn rxr rmmxxrrnmxrxrrn rmmxrxnmxrrxn rmmxmxr xnrn rxn VAUVAU VAUVAU VVA U U VAU 00 01 2212 2111 21 2 1 From this equation we obtain: ( ) [ ] ( ) ( ) ( ) ( ) ( ) ∑= −−− − =Σ=                 Σ = − − r i H iii H H H rmxrnxrrn rmrx nxm vuVU V V UUA rxmrxrnxr xmrm rxmrxr rnnxnxr 1 111 2 11 21 00 0 σ Singular Values Table of Contents
  • 21. 21 SOLO Matrices Let perform the following transformations in the domain and codomain : Geometric Interpretation of Singular Values Suppose, first, that A is square and r = n = m, and consider the spherical hypersurface in the domain of A for which: 1v 1x 2 x 2 v 1=x 111 vAv =σ 111 xAx =λ 222 xAx =λ 222 vAv =σ The Indicator Ellipsoid of a 2 x 2 Matrix11 11 nxnxnnx mxmxmmx Uy Vx η ς = = Because y = A x: ( ) ςςςη Σ=Σ=== UVVUVAUy H From which: 11 mxnxmnx ςη Σ= 1 1 22 2 2 2 ===== ∑= r i i HHH VVxxx ςςςς From we have and the mapping of the spherical hypersurface, in the codomain of A is the hypersurface of an ellipsoid: 11 mxnxmnx ςη Σ= iii σης /= 1 1 2 =        ∑= r i i i σ η This ellipsoid is called the indicator ellipsoid of A and the singular values are the lengths of the principal axes of this ellipsoid. Singular Values
  • 22. 22 SOLO Matrices If the square matrix A is singular, i.e., r < n = m, the indicator ellipsoid shrinks to zero in the directions of the principal axes vi for which σi = 0. In this case: Geometric Interpretation of Singular Values (continue – 1) 01 1 1 2 ===<        + = ∑ nr r i i i ηη σ η  If the general case of nonsquare matrices with r < n ≠ m, if we choose the cylindrical hypersurface that has a circular hypersurface projection in R (AH ): 01 1 1 2 ==== + = ∑ mr r i i ςςς  then its mapping will be the surface of the ellipsoid in R (A). 01 1 1 2 ====        + = ∑ nr r i i i ηη σ η  ( )H AR ( )AN ( )H AN ( )AR Singular Values Table of Contents
  • 23. 23 SOLO Matrices Properties of Singular Values (1) The maximum singular value of Anxm is: [ ] [ ] 2 2 02121 maxmaxmaxmax: 22 x xA xAxAAA xxx i i ≠=≤ ==== σσ (2) The minimum singular value of Anxm is: [ ] [ ] 2 2 02121 minminminmin: 22 x xA xAxAAA xxx i i ≠=≤ ==== σσ Proof of (1) and (2) Using x = V ζ we can write: ( ) ∑= = =Σ=Σ=== m i ii H xV HHHHH xVVxxAAxxAxAxA 1 22222 2 ςσςς ς 1 1 22 2 ≤==== ∑= = m i i H xV HHH xVVxxxx ςςς ς To obtain the maximum/minimum of that satisfy the condition we construct the Hamiltonian by adjoin the constraint to the extremum problem: 2 2 xA 1 2 2 ≤x ( )       −+±= ∑∑ == 1:, 1 2 1 22 m i i m i ii H ςλςσλς + for maximum - for minimum Singular Values
  • 24. 24 SOLO Matrices Properties of Singular Values Proof of maximum/minimum singular value of Anxm (continue – 1) The necessary conditions for extremum are: ( )       −+±= ∑∑ == 1:, 1 2 1 22 m i i m i ii H ςλςσλς + for maximum - for minimum ( ) mi H ii i ,,2,102 2 ==+±= ∂ ∂ ςλσ ς Kuhn-Tucker Condition ( ) mi minimumfor maximumforH i i ,,2,1 0 0 2 2 2 2 =    > < +±= ∂ ∂ λσ ς Maximization problem solution: ( ) [ ]AH 22 1,max σσλς == with 0 10,1 2 1 1 2 21 <−= =⇒==== ∑= σλ ςςςς m i im  Kuhn-Tucker Condition Minimization problem solution: ( ) [ ]AH m 22 ,min σσλς == with 0 10,1 2 1 2 11 ≥= =⇒==== ∑= − m m i imm σλ ςςςς  Kuhn-Tucker Condition Singular Values
  • 25. 25 SOLO Matrices Properties of Singular Values Proof of maximum/minimum singular value of Anxm (continue – 2) For any x ≠ 0 we have: [ ] [ ] ( ) 0max max 2 2 2 2 2 ≤ − =−=− xx xIAAx AA xx xAAx A x xA H HH H H HH λ λσ The inequality holds because (AH A-I λmax [AH A]) is non-positive definite. We can see that the equality is satisfied for x = eigenvector (AH A) that corresponds to λmax [AH A], therefore: [ ] 0max 2 2 0 =         − ≠ A x xA x σ In the same way, or any x ≠ 0 we have: [ ] [ ] ( ) 0min min 2 2 2 2 2 ≥ − =−=− xx xIAAx AA xx xAAx A x xA H HH H H HH λ λσ We can see that the equality is satisfied for x = eigenvector (AH A) that corresponds to λmin [AH A], therefore: [ ] 0min 2 2 0 =         − ≠ A x xA x σ [ ]A x xA x σ= ≠ 2 2 0 max [ ]A x xA x σ= ≠ 2 2 0 min Singular Values
  • 26. 26 SOLO Matrices Properties of Singular Values (3) is a norm of Anxm, because it satisfies the norm properties:[ ]Aσ (3.1) is non-negative and if and only if A = 0. Proof of (3.1): From Lemma 3: [ ]Aσ [ ] 0=Aσ [ ] 00 =⇔= AAσ (3.2) Multiplication by a complex constant α: [ ] [ ]AA σαασ = (3.3) Triangle Inequalities: (3.4) Schwarz Inequality: [ ] [ ] [ ] [ ] [ ]BABABA σσσσσ +≤+≤− [ ] [ ] [ ]BABA σσσ ≤ [ ] 0≥Aσ [ ] 000 =Σ=⇔=Σ⇔= H VUAAσ Proof of (3.2): [ ] ( ) [ ]AAAA xx σαααασ === ≤≤ 2121 22 maxmax Singular Values
  • 27. 27 SOLO Matrices Properties of Singular Values (3) is a norm of Anxm, because it satisfies the norm properties:[ ]Aσ (3.1) is non-negative and if and only if A = 0.[ ]Aσ [ ] 0=Aσ [ ] 00 =⇔= AAσ (3.2) Multiplication by a complex constant α: [ ] [ ]AA σαασ = (3.3) Triangle Inequalities: (3.4) Schwarz Inequality: [ ] [ ] [ ] [ ] [ ]BABABA σσσσσ +≤+≤− [ ] [ ] [ ]BABA σσσ ≤ Proof of (3.3): [ ] ( ) ( ) [ ] [ ]BAxBxA xBxAxBABA xx xx σσ σ +=+≤ +≤+=+ ≤≤ ≤≤ 2121 22121 22 22 maxmax maxmax [ ] ( )[ ] [ ] [ ]BBABBAA σσσσ ++≤−+=From which [ ] [ ] [ ]BABA +≤− σσσ In the same way [ ] [ ] [ ]BAAB +≤− σσσ [ ] [ ] [ ]BABA +≤− σσσ Proof of (3.4): [ ] ( ) ( ) [ ] [ ]BA x xB y yA x xB xB xBA x xBA BA xy xx σσ σ =≤ == ≠≠ ≠≠ 2 2 0 2 2 0 2 2 2 2 0 2 2 0 maxmax maxmax Hermann Amandus Schwarz 1843 - 1921 Singular Values
  • 28. 28 SOLO Matrices Properties of Singular Values (4) The absolute value of the eigenvalues of a square matrix Anxn are bounded between the minimum and the maximum singular values: [ ] [ ] [ ] niAAA i ,,2,1 =≤≤ σλσ Proof of (4): We have: [ ] [ ] 0 2 2 ≠∀≤≤ xA x xA A σσ If xi is any normalized eigenvector: A xi = λi xi, then ni x x x x x xA i i ii i ii i i ,,2,1 2 2 2 2 2 2 ==== λ λλ Therefore: [ ] [ ] [ ] niAAA i ,,2,1 =≤≤ σλσ Singular Values
  • 29. 29 SOLO Matrices Properties of Singular Values (5) A square matrix Anxn is singular iff its minimal singular value is zero. [ ] 0=⇔ ASingularA σ Proof of (5): ( ) [ ] [ ]AAVdiagUA n H n σσσσσσσσ =≥≥≥==  2121 ,,, Therefore: [ ] [ ] [ ] 00det ==⇔=⇔ AAASingularA nσσ ( )[ ] [ ] ∏= == n i i H n VdiagUA 11 21 1 det,,,detdetdet σσσσ   (6) For a nonsingular square matrix Anxn we have [ ] [ ] [ ] [ ]11 1 & 1 −− ==⇔ A A A ArNonsingulaA σ σ σ σ Proof of (6): ( ) ( ) H n H n UdiagVAVdiagUA σσσσσσ /1,,/1,/1,,, 21 1 21  =⇒= − [ ] [ ] [ ] [ ]1 1 1 21 /1/10 −− =≥≥=⇒>=≥≥≥= AAAA nn σσσσσσσσσ  Hence: [ ] [ ] [ ] [ ]11 1 & 1 −− == A A A A σ σ σ σ Singular Values
  • 30. 30 SOLO Matrices Properties of Singular Values (7) If the square matrix (A+B) is singular then the maximum singular values of A and of B are greater or equal than the minimum singular value of B and A, respectively. The opposite is not true. ( ) [ ] [ ] [ ] [ ]ABBASingularBA σσσσ ≥≥⇒+ & Proof of (7): If (A+B) is singular, there exists a normalized eigenvector u (║u║2=1), s.t.: ( ) 22 0 uBuAuBuAuBA =⇒−=⇒=+ From this equation we obtain: [ ] [ ]BxBuBuAxAA xx σσ =≥=≥= ≤≤ 212221 22 minmax [ ] [ ]AxAuAuBxBB xx σσ =≥=≥= ≤≤ 212221 22 minmax To prove that the opposite is not true, consider a counterexample: [ ] [ ] [ ] [ ] 15&34 30 05 10 04 =>==>=     − =      = ABBABA σσσσ The right side is satisfied, but is nonsingular.( )      − =+ 40 01 BA Singular Values
  • 31. 31 SOLO Matrices Properties of Singular Values ( ) [ ] [ ] [ ] [ ]ABBASingularBA σσσσ ≥≥⇒+ & Proof of (8): (8) A sufficient condition that the square matrix (A+B) is nonsingular is: We just proved: ( ) [ ] [ ] [ ] [ ]ABorBAingularonsNBA σσσσ <<⇒+ The proof follows directly from property (7). If (A+B) is singular then [ ] [ ] [ ] [ ]ABBA σσσσ ≥≥ & ; hence if then (A+B) is nonsingular. [ ] [ ] [ ] [ ]ABorBA σσσσ << (7) If the square matrix (A+B) is singular then the maximum singular values of A and of B are greater or equal than the minimum singular value of B and A, respectively. The opposite is not true. Singular Values
  • 32. 32 SOLO To prove this we will consider the following three cases: - (A+B) singular, - (A+B) nonsingular but A and B are singular, - (A+B) nonsingular but A or B, or both are nonsingular. Matrices Properties of Singular Values (9) The minimum singular value of a square matrix (A+B) satisfies the inequalities: [ ] [ ] [ ] [ ]( ) [ ] [ ] [ ] [ ] [ ]( )ABBABAABBA σσσσσσσσσ ++≤+≤−− ,min,max Proof of (9): (9.1) - (A+B) singular According to property (5) [ ] 0=+ BAσ Since (A+B) is singular use property (7) [ ] [ ] [ ] [ ] [ ]BABAAB +=≤−⇒≥ σσσσσ 0 [ ] [ ] [ ] [ ] [ ]BAABBA +=≤−⇒≥ σσσσσ 0 This completes the proof when (A+B) is singular. Singular Values
  • 33. 33 SOLO (9.2) - (A+B) nonsingular but A and B are singular, Matrices Properties of Singular Values (9) The minimum singular value of a square matrix (A+B) satisfies the inequalities: [ ] [ ] [ ] [ ]( ) [ ] [ ] [ ] [ ] [ ]( )ABBABAABBA σσσσσσσσσ ++≤+≤−− ,min,max Proof of (9) (continue – 1): ( )  ( ) 22 0 uBuBAuBuBuAuBA =+⇒=+=+ If A is singular, ,there exists a normalized eigenvector u (║u║2=1), s.t. A u=0:[ ] 0=Aσ [ ] ( ) ( ) [ ] [ ] [ ]ABBxBuBuBAxBABA xx σσσσ +==≤=+≤+=+ ≤≤ 212221 22 maxmin and [ ] [ ] [ ] [ ] [ ]BABABA σσσσσ +≤+<− In the same way for (A+B) nonsingular and B singular: [ ] [ ] [ ] [ ] [ ]ABBAAB σσσσσ +≤+<− This completes the proof when (A+B) is nonsingular but A and B are singular. Singular Values
  • 34. 34 SOLO (9.3) - (A+B) nonsingular but A or B, or both are nonsingular. Matrices Properties of Singular Values (9) The minimum singular value of a square matrix (A+B) satisfies the inequalities: [ ] [ ] [ ] [ ]( ) [ ] [ ] [ ] [ ] [ ]( )ABBABAABBA σσσσσσσσσ ++≤+≤−− ,min,max Proof of (9) (continue – 2): BAC +=:Suppose that (A+B) and A are nonsingular, and define: Pre-multiply by C-1 and post-multiply by A-1 : 1111 −−−− += ABCCA Let take any norm of this equation and write triangle an Schwarz inequalities: BACABC ABCCAABCC 1111 1111111 −−−− −−−−−−− ≤ −≤≤− BACCABACC 1111111 −−−−−−− −≤≤− ( ) B ABA B A +≤ + ≤− −−− 111 111 Using property (3), we can define , and because property (6) the previous equation is equivalent to: [ ]** σ= [ ] [ ] [ ] [ ] [ ]BABABA σσσσσ +≤+<− If B is nonsingular in the same way we can prove that: [ ] [ ] [ ] [ ] [ ]ABBAAB σσσσσ +≤+<− This completes the proof when (A+B) is nonsingular but A or B, or both are nonsingular. Singular Values
  • 35. 35 SOLO Using this and property (3.3): Matrices Properties of Singular Values (10) If the square matrix A is a big matrix relative to the square matrix B, then (A+B) can be approximated by A: [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]ABAABABABAIf σσσσσσσσ ≈+≈+≤+⇒>> & Proof of (10): We have: [ ] [ ] [ ] [ ]BBAA σσσσ ≥>>≥ [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]ABAABABABAA σσσσσσσσσ ≈+⇒≈+≤+≤−≈ [ ] [ ] [ ] [ ] [ ]BABABA σσσσσ +≤+≤− Using: and property (9):[ ] [ ] [ ] [ ]BBAA σσσσ ≥>>≥ [ ] [ ] [ ] [ ]( ) [ ] [ ] [ ] [ ] [ ]( )ABBABAABBA σσσσσσσσσ ++≤+≤−− ,min,max we have: [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]ABAABABABAA σσσσσσσσσ ≈+⇒≈+≤+≤−≈ Singular Values
  • 36. 36 SOLO Matrices Properties of Singular Values (11) Multiplicative Inequalities for square matrices: [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]BABABABABABA σσσσσσσσσσ ≤≤≤≤ & Proof of (11): The proof is given in the following steps: (11.1) is the Schwarz inequality of property (3)[ ] [ ] [ ]BABA σσσ ≤ (11.2) prove that :[ ] [ ] [ ]BABA σσσ ≤ If A or B is singular ( or is zero) then A B is singular ( det [A B] = det [A]. det [B]=0 and ) and we have equality. [ ]Aσ [ ]Bσ [ ] 0=BAσ If A or B is nonsingular then A B is nonsingular ( det [A B] = det [A]. det [B]≠0 ) and: ( ) ( ) 11111111 −−−−−−−− ≤=== ABABBAABBA We define , and use the property (6):[ ]** σ= [ ] [ ] [ ] [ ]11 1 & 1 −− == A A A A σ σ σ σ to obtain: [ ] [ ] ( ) [ ]BA BABA BA σσσ ≤≤= −−− 111 111 This result is opposite to Schwarz inequality, proving that is not a norm.[ ]Aσ Singular Values
  • 37. 37 SOLO Matrices Properties of Singular Values (11) Multiplicative Inequalities for square matrices: [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]BABABABABABA σσσσσσσσσσ ≤≤≤≤ & Proof of (11) (continue – 1): (11.3) prove that : If A is singular then: [ ] [ ] [ ] [ ] [ ]BABAorBA σσσσσ ≤ [ ] [ ] [ ]BABA σσσ ≤= 0 If A is nonsingular then: [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]BAB A BA A BAABAAB σσ σ σ σ σσσσ ≤⇒=≤= −− 1111 If B is singular then: [ ] [ ] [ ]BABA σσσ ≤= 0 If B is nonsingular then: [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]BABA B BA BBABBAA σσσ σ σ σσσσ ≤⇒=≤= −− 11 Singular Values
  • 38. 38 SOLO Matrices Properties of Singular Values (11) Multiplicative Inequalities for square matrices: [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]BABABABABABA σσσσσσσσσσ ≤≤≤≤ & Proof of (11) (continue – 2): (11.4) prove that : If A or B are singular then A B is singular, and: [ ] [ ] [ ] [ ] [ ]BABAorBA σσσσσ ≤ [ ] [ ] [ ] [ ] [ ]( )BAorBABA σσσσσ ≤= 0 If B is nonsingular then: [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]BABA B BA BBABBAA σσσ σ σ σσσσ ≤⇒=≥= −− 11 We also have: [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]BABA B BA BBABBABBAA σσσ σ σ σσσσσσ ≤⇒ =≥≥= −−− 111 If A is nonsingular then: [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]BABA A BA BAABAAB σσσ σ σ σσσσ ≤⇒=≥= −− 11 We also have: [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]BABA A BA BAABAABAAB σσσ σ σ σσσσσσ ≤⇒ =≥≥= −−− 111 q.e.d. Singular Values
  • 39. 39 SOLO Matrices Properties of Singular Values (12) Any unitary matrix U (U UH = UH U = I) has all the singular values equal to 1. Proof of (12): [ ] [ ] [ ] iIUUU i H ii ∀=== 12/12/1 λλσ (13) If U is a unitary matrix (U UH = UH U = I) then: Proof of (13): [ ] [ ] [ ] iAUAAU iii ∀== σσσ [ ] ( ) ( )[ ] [ ] [ ] [ ] iAAAAUUAAUAUAU i H i HH i H ii ∀==== σλλλσ 2/12/12/1 [ ] ( ) ( )[ ] [ ] [ ] [ ] [ ] iAAAAAUUUAAUUAUAUA i H i HH i HH i H ii ∀===== σλλλλσ 2/12/12/12/1 q.e.d. Singular Values q.e.d.
  • 40. 40 SOLO Matrices Properties of Singular Values - Summary (4) The absolute value of the eigenvalues of a square matrix Anxn are bounded between the minimum and the maximum singular values: [ ] [ ] [ ] niAAA i ,,2,1 =≤≤ σλσ (3) is a norm of Anxm, because it satisfies the norm properties:[ ]Aσ (3.1) is non-negative and if and only if A = 0.[ ]Aσ [ ] 0=Aσ [ ] 00 =⇔= AAσ (3.2) Multiplication by a complex constant α: [ ] [ ]AA σαασ = (3.3) Triangle Inequalities: (3.4) Schwarz Inequality: [ ] [ ] [ ] [ ] [ ]BABABA σσσσσ +≤+≤− [ ] [ ] [ ]BABA σσσ ≤ (1) The maximum singular value of Anxm is: [ ] [ ] 2 2 02121 maxmaxmaxmax: 22 x xA xAxAAA xxx i i ≠=≤ ==== σσ (2) The minimum singular value of Anxm is: [ ] [ ] 2 2 02121 minminminmin: 22 x xA xAxAAA xxx i i ≠=≤ ==== σσ Singular Values
  • 41. 41 SOLO Matrices Properties of Singular Values – Summary (continue – 1) (5) A square matrix Anxn is singular iff its minimal singular value is zero. [ ] 0=⇔ ASingularA σ (6) For a nonsingular square matrix Anxn we have [ ] [ ] [ ] [ ]11 1 & 1 −− ==⇔ A A A ArNonsingulaA σ σ σ σ (9) The minimum singular value of a square matrix (A+B) satisfies the inequalities: [ ] [ ] [ ] [ ]( ) [ ] [ ] [ ] [ ] [ ]( )ABBABAABBA σσσσσσσσσ ++≤+≤−− ,min,max (8) A sufficient condition that the square matrix (A+B) is nonsingular is: ( ) [ ] [ ] [ ] [ ]ABorBAingularonsNBA σσσσ <<⇒+ ( ) [ ] [ ] [ ] [ ]ABBASingularBA σσσσ ≥≥⇒+ & (7) If the square matrix (A+B) is singular then the maximum singular values of A and of B are greater or equal than the minimum singular value of B and A, respectively. The opposite is not true. Singular Values
  • 42. 42 SOLO Matrices Properties of Singular Values – Summary (continue – 2) (12) Any unitary matrix U (U UH = UH U = I) has all the singular values equal to 1. (13) If U is a unitary matrix (U UH = UH U = I) then: [ ] [ ] [ ] iAUAAU iii ∀== σσσ (11) Multiplicative Inequalities for square matrices: [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]BABABABABABA σσσσσσσσσσ ≤≤≤≤ & (10) If the square matrix A is a big matrix relative to the square matrix B, then (A+B) can be approximated by A: [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ]ABAABABABAIf σσσσσσσσ ≈+≈+≤+⇒>> & Singular Values Table of Contents
  • 43. 43 SOLO Matrices Householder Transformation nˆ ( )xnn T  ˆˆ ( )xnn T  ˆˆ x  'x  O A We want to compute the reflection of over a plane defined by the normal ( )1ˆˆˆ =nnn T x  From the Figure we can see that: ( ) ( ) xHxnnIxnnxx TT  =−=−= ˆˆ2ˆˆ2' 1ˆˆˆˆ2: =−= nnnnIH TT We can see that H is symmetric: ( ) HnnInnIH TTTT =−=−= ˆˆ2ˆˆ2 In fact H is also a rotation of around OA so it must be orthogonal, i.e. HT H=H HT =I. x  ( ) ( )  InnnnnnInnInnIHHHH TTTTTT =+−=−−== ˆˆˆˆ4ˆˆ4ˆˆ2ˆˆ2 1 Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Table of Contents Alston Scott Householder 1904 - 1993
  • 44. 44 SOLO Matrices The same result is obtained if we compute α that minimizes: a  b  P a  α ab  α−p  We want to find such that ( )pba  −⊥ap  α= Projection of a vector on a vector .b  a  or: ( ) ( ) ( ) baaaabapba TTTT  1 0 − =⇒−=−= αα and: ( ) ( )[ ] bPbaaaabaaaaap TTTT  ==== −− 11 α ( )[ ]TT aaaaP  1 : − = Projection Matrix ( ) ( ) ( )aababbabababd TTT T  2 2 2 2minminminmin ααααα αααα +−=−−=−= ( ) ( ) ( ) baaa aa d aaba d TT T TT    1 min 2 22 2 0 022 − =⇒        >= ∂ ∂ =+−= ∂ ∂ α α α α Properties of Projection Matrix (1) P is idempotent P2 = P (2) P is symmetric PT = P ( ) cbcPIcPcbP  ,∀−=−⊥ Proof: ( ) ( ) ( ) cbcPIPbcPIbP TT T  ,0 ∀−=−= ( ) 0=− PIPT Hence: PPP TT = ( ) PPPPP TTT == 2 PPPPP TT === b  bP  cP  cP c   − c  Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Table of Contents
  • 45. 45 SOLO Note: If A and b were real, instead of H (transpose & complex conjugate) we have only T (transpose). Matrices Given: Amxn of rank (Amxn) = r ≤ min (m,n) and 1mx b  Find: such that is minimal1nx x  11 mxnxmxn bxAd  −= If the solution is not unique choose such that is minimal1nx x  1nx x  Solution: The minimum is obtained when ( ) ( ) ( )        >= ∂ ∂ =−=−=−= ∂ ∂ 0 0 2 22 2 11 2 AA x d bAxAAbxAAbxA x d H HHH mxnxmxn    ( ) bAAAx HH  1− = A unique solution exists if AH A is positive definite, or rank (AH A) = n, or det|AH A| ≠ 0 ( ) ( )1111 2 11 2 mxnxmxn H mxnxmxnmxnxmxn bxAbxAbxAd  −−=−=Analytic: Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . 111 min&min 1 nxmxnxmxn x xbxAd nx   −=
  • 46. 46 SOLO Matrices Given: Amxn of rank (Amxn) = r ≤ min (m,n) and 1mx b  Find: such that is minimal1nx x  11 mxnxmxn bxAd  −= If the solution is not unique choose such that is minimal1nx x  1nx x  Solution (continue – 1): Geometric: We have A x∈ R (A) for all x ∈ domain (A). We want to find x0 ∈ domain (A), such that is normal to A x.0 xAbpb  −=− ( ) ( ) ( )AdomainxxAbpbxA ∈∀−=−⊥  0 ( ) ( ) ( ) ( )AdomainxxAAbAxxAbxA HHHH ∈∀−=−=  00 0 Hence: 00 =− bAxAA HH  ( )H AR ( )AN ( )H AN ( )AR xA  ( )Adomx ∈0  ( )AcodomY ∈ Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 b  R x0  Nx0  0 xAp  = pb  − xA  ( )Adomx ∈  0 xA  Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . 111 min&min 1 nxmxnxmxn x xbxAd nx   −=
  • 47. 47 SOLO Matrices Let decompose as0 x  ( )H AR ( )AN ( )H AN ( )AR xA  ( )Adomx ∈0  ( )AcodomY ∈ Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 b  R x0  N x0  0xAp  = pb  − xA  ( )Adomx ∈  0xA  ( ) ( ) NR N H R NR xx AofspaceNullANx AofspaceRowARx xxx 00 0 0 000     ⊥    ∈ ∈ += Therefore: RNR xAxAxAxAp 0 0 000    =+== Hence if: (a) N (A) = 0 or (b) The rows of A are linearly dependent or (c) rank (A) = r < m (d) AH A is singular there are a infinity of solutions NRNR xxxxx 00000  ⊥+= The norm of is:0x  NR xx NR xxxxx NR 00000 00   +=+= ⊥ Hence: 0&min 000 == NR xxx  Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . 111 min&min 1 nxmxnxmxn x xbxAd nx   −=
  • 48. 48 SOLO Matrices Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . ( )T AR ( )AN ( )H AN ( )AR xA  ( )Adomx ∈0  ( )AcodomY ∈ Null Space of A Ker (A) span by VA2 T Row Space of A span by VA1 T Column Space of A span by UA1 Left Null Space of A span by UA2 b  R x0  N x0  0xAp  = pb  − xA  ( )Adomx ∈  0xA  bAx R  + =0 0&min 000 == NR xxx  Define the Linear Transformation (Matrix), that gives from , as the Pseudoinverse of A. (A is the direct transformation that gives from : Rx0  b  p  x  xAp  = bAx † R  =0 A† is called Moore-Penrose Pseudoinverse Matrix, because was defined independently by E.H.Moore in 1920 and Roger Penrose in 1955. Eliakim Hastings Moore 1862 - 1932 Roger Penrose 1931 - 111 min&min 1 nxmxnxmxn x xbxAd nx   −= Table of Contents
  • 49. 49 SOLO Matrices Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn .a  ( )H AR ( )AN ( )H AN ( )AR xA  ( )Adomx ∈0  ( )AcodomY ∈ Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 b  R x0  N x0  0xAp  = pb  − xA  ( )Adomx ∈  0xA  bAx † R  =0 bAx † R  =0 Computation of Moore-Penrose Pseudoinverse Matrix, A † Perform Singular Value Decomposition (S.V.D.) of Amxn: where ( ) 0,,, 21211 >≥≥≥=Σ rrA diagrxr σσσσσσ  UAmxm and VAnxn are unitary matrices, i.e.: ( ) [ ] ( ) ( ) ( ) ( ) ( )                Σ =Σ= − − −−− − H A H A rnxrmxrrm rnrxA AA H AAAmxn xnrn rxnrxr rmmxmxrnxnmxnmxm V V UUVUA 2 11 21 00 0  [ ] ( ) ( ) [ ] H AA H AA H AAH A H A AA rmxrm rxr AAH A H A A H A UUUUUU U U UU I I UU U U UU =+=      =      =      = −− 2211 2 1 2121 2 1 0 0 [ ] ( ) ( ) [ ] H AA H AA H AAH A H A AA rnxrn rxr AAH A H A A H A VVVVVV V V VV I I VV V V VV =+=      =      =         = −− 2211 2 1 2121 2 1 0 0
  • 50. 50 SOLO Matrices ( )H AR ( )AN ( )H AN ( )AR xA  ( )Adomx ∈0  ( )AcodomY ∈ Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 b  R x0  N x0  0xAp  = pb  − xA  ( )Adomx ∈  0xA  bAx † R  =0 Since the norm is invariant to the product of orthogonal matrices ( ) bUxVbxVUUbxVUbxA HHHHH  −Σ=−Σ=−Σ=− Introduce the new unknown: R H N H R HH xVxVxVxVy  =+==: But  ( ) 0 0 =⇒Σ=+Σ=+= N H R H N H R H NR xVxVUxVxVUxAxAxA  RR I HH RR HH xxVVxxVxVy    =         === 2/1 ( ) ( ) ( ) ( ) bUybUybxA H rnxrmxrrm r rnrx y H yx         −                 =−Σ=− −−− − 00 0 0 0 minminmin 1 σ σ Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Computation of Moore-Penrose Pseudoinverse Matrix, A †
  • 51. 51 SOLO Matrices ( )H AR ( )AN ( )H AN ( )AR xA  ( )Adomx ∈0  ( )AcodomY ∈ Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 b  Rx0  Nx0  0xAp  = pb  − xA  ( )Adomx ∈  0xA  bAx † R  =0 Therefore: R HH xVxVy  ==: RR I HT RR HH xxVVxxVxVy    =         === 2/1 ( ) ( ) ( ) ( ) bUybUybxA H rnxrmxrrm r rnrx y H yx         −                 =−Σ=− −−− − 00 0 0 0 minminmin 1 σ σ ( ) ( ) ( ) ( ) ( )                     +                 = − −−− −         any xrm rx H rmxrnxrrn r rmrx XbUy 1 1 1 0 00 /10 0 0/1 σ σ R xyx 0 minmin  == ( ) ( ) ( ) ( ) R HH rmxrnxrrn r rmrx xVbUy 0 1 00 /10 0 0/1           =                 = + Σ −−− − σ σ bAbUVx †H† R  =Σ=0 H†† UVA Σ= Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Computation of Moore-Penrose Pseudoinverse Matrix, A †
  • 52. 52 SOLO Matrices ( )H AR ( )AN ( )H AN ( )AR xA  ( )Adomx ∈0  ( )AcodomY ∈ Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 b  R x0  N x0  0 xAp  = pb  − xA  ( )Adomx ∈  0xA  bAx † R  =0 Where: ( ) ( ) ( ) ( )                   =Σ −−− − − = rmxrnxrrn r rmrx † nxm 00 0 0 0 : 1 1 1      σ σ H mxm † nxmnxn † nxm UVA Σ= Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Computation of Moore-Penrose Pseudoinverse Matrix, A † ( ) [ ] ( ) ( ) ( ) ( ) ( ) H AAAH A H A rmxrnxrrn rmrxA AnA † nxm rxmrxrnxr xmrm rxmrxr rnnxmxr UV U U VVA 1 1 11 2 1 1 1 21 00 0 : − −−− − − Σ=                Σ = − −            =Σ − = 1 1 1 1 0 0 : r 1- A rxr σ σ    Table of Contents
  • 53. 53 SOLO Matrices Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Properties of Moore-Penrose Pseudoinverse Matrix, A † ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )         =                                   =ΣΣ −−− − −−− − −−− − − = rnxrnxrrn rnrxrxr rnxrmxrrm r rnrx rmxrnxrrn r rmrx mxn † nxm I 00 0 00 0 0 0 00 0 0 0 1 1 1 1           σ σ σ σ ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )         =                                   =ΣΣ −−− − −−− − − = −−− − rmxrmxrrm rmrxrxr rmxrnxrrn r rmrx rnxrmxrrm r rnrx † nxmmxn I 00 0 00 0 0 0 00 0 0 0 1 1 11           σ σ σ σ ( ) ( ) ( ) ( ) ( ) † nxmmxn rmxrmxrrm rmrxrxr†† nxmmxn I ΣΣ=         =ΣΣ −−− − 00 0 ( ) ( ) ( ) ( ) ( ) mxn † nxm rnxrnxrrn rnrxrxr† mxn † nxm I ΣΣ=         =ΣΣ −−− − 00 0 Using the definition of the Pseudoinverse we can see that
  • 54. 54 SOLO Matrices Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Properties of Moore-Penrose Pseudoinverse Matrix, A † ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) † nxm rmxrnxrrn r rmrx rmxrnxrrn r rmrx rnxrnxrrn rnrxrxr† nxmmxn † nxm I Σ=                   =                           =ΣΣΣ −−− − − = −−− − − = −−− − 00 0 0 0 00 0 0 0 00 0 1 1 1 1 1 1           σ σ σ σ ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) mxn rnxrmxrrm r rnrx rnxrmxrrm r rnrx rmxrmxrrm rmrxrxr mxn † nxmmxn I Σ=                 =                         =ΣΣΣ −−− − −−− − −−− − 00 0 0 0 00 0 0 0 00 0 11           σ σ σ σ ( ) ( ) ( ) ( ) † nxmmxn H†H†† Def† †H††H†H†† nxmmxn AAUUUUUUUVVUAA =ΣΣ=ΣΣ=ΣΣ=ΣΣ= ( ) ( ) ( ) ( ) mxn † nxm H†H†† Def† †H††HH†† mxn † nxm AAVVVVVVVUUVAA =ΣΣ=ΣΣ=ΣΣ=ΣΣ= Also: Let check the same operations for Matrix A † ( ) ( ) ( ) ( ) mxn HH†HH†H mxn † nxmmxn AVUVUVUUVVUAAA =Σ=ΣΣΣ=ΣΣΣ= ( ) ( ) ( ) ( ) † nxm H†H††H†HH†† nxmmxn † nxm AUVUVUVVUUVAAA =Σ=ΣΣΣ=ΣΣΣ=
  • 55. 55 SOLO Matrices Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Properties of Moore-Penrose Pseudoinverse Matrix, A † - Summary) ( ) ††† nxmmxnnxmmxn AAAA = ( ) mxn † nxm † mxn † nxm AAAA = mxnmxn † nxmmxn AAAA = † nxm † nxmmxn † nxm AAAA = 1 2 3 4 Table of Contents
  • 56. 56 SOLO Matrices Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Description of Projections Related to Moore-Penrose Pseudoinverse bPbAAxAp † R  1 ===1 ( )H AR ( )AN ( )H AN ( )AR Rx  ( )Adomx∈ ( )Acodomb ∈  Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 b  bAx † R  = ( ) R † xA bAAp   = = R xAp  = P1 is a projection matrix because ( ) ( ) ( )    =ΣΣ=ΣΣ== === 11 1 2 1 PUUUVVUAAP PAAAAAAP HH†HH†HH†H ††† P1=A A† projects into column space of A, R (A)b  H†H†H† UUUVVUAAP ΣΣ=ΣΣ==:1
  • 57. 57 SOLO Matrices Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Description of Projections Related to Moore-Penrose Pseudoinverse 2 ( )H AR ( )AN ( )H AN ( )AR R x  ( )Adomx ∈  ( )Acodomb ∈  Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 b  bAx † R  = ( ) R † xA bAAp   = = RxAp  = ( )bAAIpb †  −=− ( ) 0=− pbA†  pAx † R  = P2=(I - A A † ) is a projection matrix because ( ) ( ) ( ) ( )     =−=−=−=−= =+−−=−−= 21112 2 2 2 PPIPIPIAAIP PAAAAAAAAIAAIAAIP HHH†H † A †††††  Because , is the projection of into . ( ) ( )H ANAR ⊥ pb  − ( )H ANb  We can see, also, that: ( ) H†H† † UIUUUI AAIPIP ΣΣ−=ΣΣ−= −=−= 12 : ( ) ( ) 00    ==         −=−=− bbAAAAbAAIApbA † A †††††† pAbAx †† R  == ( ) ( ) bPbAAIpb †  2 =−=−
  • 58. 58 SOLO Matrices Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Description of Projections Related to Moore-Penrose Pseudoinverse 3 ( ) ( ) NR N H R NR xx AofspaceNullANx AofspaceRowARx xxx     ⊥    ∈ ∈ += ( ) ( ) xPxAAxAApAbAx †††† R  3 ===== P3=A † A is a projection matrix of in R (AH )x  ( ) ( ) ( ) ( ) ( )     =ΣΣ=ΣΣ== ==== 33 3 2 3 PVVVUUVAAP PAAAAAAAAAAP HH†HHH†H†H † A †††† †  4 ( ) ( ) xPxAAIxAAxxxx †† RN  4 =−=−=−= P4=I-A † A is a projection matrix of in N (A)x  ( )  ( ) ( )     =−=−=−= =+−=−= 43333 4 2 33 2 3 2 4 3 2 PPIPIPIP PPPIPIP HHHH P ( )H AR ( )AN ( )H AN ( )AR ( )Adomx ∈  ( )Acodomb ∈  Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 b  bAx † R  = ( ) R † xA bAAp   = = R xAp  = ( )bAAIpb †  −=− ( ) 0=− pbA†  pAx † R  = ( ) xAAx † R  = ( )H AR ( )AN ( )H AN ( )AR ( )Adomx ∈  ( )Acodomb ∈  Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 b  bAx † R  = ( ) R † xA bAAp   = = R xAp  = ( )bAAIpb †  −=− ( ) 0=− pbA†  pAx † R  = ( ) xAAx † R  = ( ) xAAIx † N  −= 0=NxA 
  • 59. 59 SOLO Matrices Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Description of Projections Related to Moore-Penrose Pseudoinverse (Summary) 3 ( ) ( ) ( )H†††† R ARxPxAAxAApAbAx ∈=====  3 4 ( )H AR ( )AN ( )H AN ( )AR ( )Adomx ∈  ( )Acodomb ∈  Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 b  bAx † R  = ( ) R † xA bAAp   = = R xAp  = ( )bAAIpb †  −=− ( ) 0=− pbA†  pAx † R  = ( ) xAAx † R  = ( ) xAAIx † N  −= 0=NxA  ( ) ( ) ( )ANxPxAAIxAAxxxx †† RN ∈=−=−=−=  4 ( ) ( ) ( )H† ANbPbAAIpb ∈=−=−  2 2 ( )ARbPbAAxAp † R ∈===  1 1 H†† UUAAP ΣΣ==:1 † AAIPIP −=−= 12 : H†† VVAAP ΣΣ==:3 AAIPIP † −=−= 34 : Table of Contents
  • 60. 60 SOLO Matrices Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Particular case (1) r = n ≤ m: ( ) [ ] ( ) H A xnnm A AA H AAAmxn nxn nxn nmmxmxnnxnmxnmxm VUUVUA        Σ =Σ= − − 0 1 21  (a) rank (Amxn) = n or (b) columns of Amxn are linear independent or (c) N (Amxn) = 0 or (d) Anxm H Amxn is nonsingular This is equivalent to: where ( ) 0,,, 21211 >≥≥≥=Σ nnA diagnxn σσσσσσ  [ ] ( ) ( ) [ ] H AA H AA H AAH A H A AA nmxnm nxn AAH A H A A H A UUUUUU U U UU I I UU U U UU =+=      =      =      = −− 2211 2 1 2121 2 1 0 0 ( )[ ] ( ) H AAAH A H A nmnxAA † nxm nxmnxnnxn xmnm nxm nxnnxn UV U U VA 1 1 1 2 11 1 0: − − − Σ=         Σ= − ( )H AR ( ) 0=AN ( )H AN ( )AR x  b  Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 ( )bAAxAp †  == pAx †  = ( )bAAIpb  + −=− xAp  = bAx †  = ( ) 0=− pbA† 
  • 61. 61 SOLO Matrices Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Particular case (1) r = n ≤ n: (continue – 1) ( )[ ] ( ) H AAAH A H A nmnxAA † nxm nxmnxnnxn xmnm nxm nxnnxn UV U U VA 1 1 1 2 11 1 0: − − − Σ=         Σ= − ( ) [ ] ( ) H A xnnm A AA H AAAmxn nxn nxn nmmxmxnnxnmxnmxm VUUVUA        Σ =Σ= − − 0 1 21  [ ] [ ] H AAA H A A AAH A H A AA H VVVUU U U VAA 2 1 1 21 2 1 1 0 0 Σ=     Σ       Σ=  ( ) H AAA H VVAA 2 1 1 −− Σ= ( ) †H AAA H AAA H AAA HH AUVUVVVAAA =Σ=ΣΣ= −−− 1 1 111 2 1 1 or ( ) H nxmmxn H nxm † nxm AAAA 1− = ( )H AR ( ) 0=AN ( )H AN ( )AR x  b  Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 ( )bAAxAp †  == pAx †  = ( )bAAIpb  + −=− xAp  = bAx †  = ( ) 0=− pbA†  We have only one solution that minimize 11 mxnxmxn bxAd  −= x  and is given by: ( ) 1 1 11 mx H nxmmxn H nxmmx † nxmnx bAAAbAx  − == Table of Contents
  • 62. 62 SOLO Matrices Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Particular case (2) r = m ≤ n: (a) rank (Amxn) = m or (b) rows of Amxn are linear independent or (c) N (Anxm H ) = 0 or (d) AmxnAnxm H is nonsingular This is equivalent to: where ( ) 0,,, 21211 >≥≥≥=Σ mmA diagmxm σσσσσσ  ( ) [ ] ( ) H AAA H A xmmn A AnA † nxm mxmmxmnxmmxm mxm mnnxmxr UVUVVA 1 11 1 1 21 0 : − − − Σ=        Σ = −  ( )[ ] ( ) H AAAH A H A mnmxAA H AAAmxn mxnmxmmxm xnmn mxn mxmmxmnxnmxnmxm VU V V UVUA 11 2 1 1 0 Σ=         Σ=Σ= − − [ ] ( ) ( ) [ ] H AA H AA H AAH A H A AA mnxmn mxm AAH A H A A H A VVVVVV V V VV I I VV V V VV =+=      =      =         = −− 2211 2 1 2121 2 1 0 0 ( )H AR ( )AN ( ) ( )ARBAN H ∈≡ &0 ( )AR xAb  = ( ) N † nxn xxAAI  =− ( ) R † xxAA  = Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 bAx † R  = R xAb  = b  0=N xA 
  • 63. 63 SOLO Matrices Moore-Penrose Pseudoinverse Matrix Anxm † of Amxn . Particular case (2) r = m ≤ n: (continue – 1) ( ) [ ] ( ) H AAA H A xmmn A AnA † nxm mxmmxmnxmmxm mxm mnnxmxr UVUVVA 1 11 1 1 21 0 : − − − Σ=        Σ = −  ( )[ ] ( ) H AAAH A H A mnmxAA H AAAmxn mxnmxmmxm xnmn mxn mxmmxmnxnmxnmxm VU V V UVUA 11 2 1 1 0 Σ=         Σ=Σ= − − H AAA H AA I A H AAA H UUUVVUAA m 2 11111 Σ=ΣΣ=  ( ) H AAA H UUAA 2 1 1 −− Σ= ( ) †H AAA H AAA H AAA HH AUVUUUVAAA =Σ=ΣΣ= −−− 1 11 2 111 1 or ( ) 1− = HH† AAAA ( )H AR ( )AN ( ) ( )ARBAN H ∈≡ &0 ( )AR xAb  = ( ) N † nxn xxAAI  =− ( ) R † xxAA  = Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 bAx † R  = R xAb  = b  0=N xA  We have an infinite number of solutions that minimize 11 mxnxmxn bxAd  −= ( ) bAAAbAx HH† R  1− == The solution that minimizes the norm is given by: Rx  Rx  Table of Contents
  • 64. 64 SOLO Matrices General Solution of Amxn Xnxp = Bmxp X - nxp unknowns with mxp equations mxpnxpmxn BXA = Perform Singular Value Decomposition (S.V.D.) of Amxn: where ( ) 0,,, 21211 >≥≥≥=Σ rrA diagrxr σσσσσσ  UAmxm and VAnxn are unitary matrices, i.e.: ( )H AR ( )AN ( )H AN ( )ARBXA = 11 yAx H = ( )AdomX ∈ ( )AcodomY ∈ 1nx y Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 B ( ) [ ] ( ) ( ) ( ) ( ) ( )                Σ =Σ= − − −−− − H A H A rnxrmxrrm rnrxA AA H AAAmxn xnrn rxnrxr rmmxmxrnxnmxnmxm V V UUVUA 2 11 21 00 0  [ ] ( ) ( ) [ ] H AA H AA H AAH A H A AA rmxrm rxr AAH A H A A H A UUUUUU U U UU I I UU U U UU =+=      =      =      = −− 2211 2 1 2121 2 1 0 0 [ ] ( ) ( ) [ ] H AA H AA H AAH A H A AA rnxrn rxr AAH A H A A H A VVVVVV V V VV I I VV V V VV =+=      =      =         = −− 2211 2 1 2121 2 1 0 0
  • 65. 65 SOLO Matrices General Solution of Amxn Xnxp = Bmxp Let multiply by and using: mxpnxpmxn BXA =         H H U U 2 1 we obtain: [ ]         =              Σ         BU BU X V V UU U U H A H A H A H AA I AAH A H A m 2 1 2 11 21 2 1 00 0     or:         =              Σ BU BU X V V H A H A H A H AA 2 1 2 11 00 0 or: ( ) ( )xprmmxp H A BU xmrm −=− 02 (m-r)xp - constraints equivalent to condition Bmxp∈ℜ (Amxn) mxp H Anxp H AA BUXV rxmrxnrxr 111 =Σ rxp - independent equations nxp – unknowns since r ≤ n → # Eq. ≤ # Unknown ( )H AR ( )AN ( )H AN ( )ARBXA = 11 yAx H = ( )AdomX ∈ ( )AcodomY ∈ Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 B ( ) [ ] ( ) ( ) ( ) ( ) ( )                Σ =Σ= − − −−− − H A H A rnxrmxrrm rnrxA AA H AAAmxn xnrn rxnrxr rmmxmxrnxnmxnmxm V V UUVUA 2 11 21 00 0 
  • 66. 66 ( ) ( )xprmmxp H A BU xmrm − =− 02 (m-r)xp - constraints equivalent to condition Bmxp∈ℜ (Amxn) SOLO Matrices General Solution of Amxn Xnxp = Bmxp mxp H Anxp H AA BUXV rxmrxnrxr 111 =Σ rxp - independent equations nxp – unknowns since r ≤ n → # Eq. ≤ # Unknown This equation is a Necessary and Sufficient Condition for any solutions of equation Amxn Xnxp = Bmxp. It is equivalent to Bmxp∈ℜ (Amxn) or Bmxp ∩ N (AT ) = ∅. If this condition is fulfilled, then from we have nxp unknowns ≥ rxp independent equations, that means (n-r)xp degrees of freedom. mxp H Anxp H AA BUXV rxmrxnrxr 111 =Σ mxp H AAnxp H A BUXV rxmrxrrxn 1 1 11 − Σ= Since VA1 T VA1=Ir & VA1 T VA2 = 0 the General Solution of Amxn Xnxp = Bmxp is: ( ) ( ) ( ) ( )    AN xprnA AR mxp H AAAnxp YVBUVX rnnx T rxmrxrnxr ∈ − ∈ − − +Σ= 21 1 11 where Y(n-r)xp is any (n-r)xp matrix, i.e. we used all (n-r)xp degrees of freedom. ( )H AR ( )AN ( )H AN ( )ARBXA = ( )AdomX ∈ ( )AcodomY ∈ Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 BB has to be in the column space of A ( ) ( ) =∩∈ ANBorARB
  • 67. 67 SOLO Matrices General Solution of Amxn Xnxp = Bmxp Check: ( ) [ ] ( ) ( ) ( ) ( ) ( ) ( ) ( )( ) ( ) [ ] ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) [ ] ( ) ( ) ( ) ( ) ( ) ( ) [ ] ( ) ( ) [ ] ( ) ( ) [ ] ( ) mxpmxp xmrm H A H A AA mxpxmrm H A mxp H A AA xprm mxp H A AA xprn mxp H AA rnxrmxrrm rnrxA AA xprn I A H Amxp H AAA H A xprnA H Amxp H AA I A H A rnxrmxrrm rnrxA AA xprnAmxp H AAAH A H A rnxrmxrrm rnrxA AAnxp H AAAnxpmxn BB U U UUBU BU UU BU UU Y BU UU YVVBUVV YVVBUVV UU YVBUV V V UUXVUXA rxm rmmxmxr rxm rmmxmxr rxm rmmxmxr rxmrxrrxr rmmxmxr rnnxxnrnrxmrxrnxrxnrn rnnxrxnrxmrxrnxrrxnrxr rmmxmxr rnnxrxmrxrnxr xnrn rxnrxr rmmxmxrnxnmxnmxm =         =           =         =        Σ        Σ =             +Σ +Σ        Σ = +Σ                Σ =Σ= − − −− − −−− − − − − − −−− − − − −−− − −− −− −−− − − − − − 2 1 21 0 2 1 21 1 21 1 1 11 21 221 1 1 0 12 0 211 1 1111 21 21 1 11 2 11 21 000 0 00 0 00 0        
  • 68. 68 SOLO Matrices where r is such that: General Solution of Amxn Xnxp = Bmxp Algorithm to solve Amxn Xnxp = Bmxp: (1) Compute s.v.d. of Amxn and partition according to: ( ) 0,,, 21211 >≥≥≥=Σ rrA diagrxr σσσσσσ  (2) Check if: ( ) ( )xprmmxp H A BU xmrm − =− 02 (3) If (2) is not true → no solution for (1) ( ) ( )  any xprnAmxp H AAAnxp YVBUVX rnnxrxmrxrnxr − − − +Σ= 21 1 11 ( ) [ ] ( ) ( ) ( ) ( ) ( )                Σ =Σ= − − −−− − H A H A rnxrmxrrm rnrxA AA H AAAmxn xnrn rxnrxr rmmxmxrnxnmxnmxm V V UUVUA 2 11 21 00 0  If (2) is true → (n-r)xp solutions:
  • 69. 69 SOLO Matrices General Solution of Amxn Xnxp = Bmxp Moore-Penrose Pseudoinverse of A: ( ) [ ] ( ) ( ) ( ) ( ) ( ) H AAAH A H A rmxrnxrrn rmrxA AnA † nxm rxmrxrnxr xmrm rxmrxr rnnxmxr UV U U VVA 1 1 11 2 1 1 1 21 00 0 : − −−− − − Σ=                Σ = − −  then ( ) [ ] ( ) ( ) ( ) ( ) ( ) ( ) [ ] ( ) ( ) ( ) ( ) ( ) H AAH A H A rmxrnxrrn rmrxA I AAH A H A rnxrmxrrm rnrxA AA † mxn rxnnxr xmrm rxmrxr nxn rnnxnxr xnrn rxnrxr rmmxmxrnxm UU U U VV V V UUAA 11 2 1 1 1 21 2 11 21 00 0 00 0 =                Σ                Σ = − − − − −−− − − −−− −     ( ) ( ) ( ) ( ) ( ) H AA H AA H AA H AA † nxmmxnmxm xmrmrmmxrxmmxrxmrmrmmxrxmmxr UUUUUUUUAAI −−−− =−+=− 22112211 also ( ) [ ] ( ) ( ) ( ) ( ) ( ) ( ) [ ] ( ) ( ) ( ) ( ) ( ) H AAH A H A rnxrmxrrm rnrxA I AAH A H A rmxrnxrrn rmrxA AAnxm † mxn rxnnxr xnrn rxnrxr mxm rmmxmxr xmrm rxmrxr rnnxnxr VV V V UU U U VVAA 11 2 11 21 2 1 1 1 21 00 0 00 0 : =                Σ                Σ = − − − − −−− − −−− − −     ( ) ( ) ( ) ( ) ( ) H AA H AA H AA H AAmxn † nxmnxn xnrnrmnxrxnnxrxnrnrmnxrxnnxr VVVVVVVVAAI −−−− =−+=− 22112211
  • 70. 70 SOLO Matrices General Solution of Amxn Xnxp = Bmxp Moore-Penrose Pseudoinverse of A (continue - ): Define also Znxp such that: ( ) ( ) nxp H Axprn ZVY xrrn− =− 2 : ( ) ( )  any xprnAmxp H AAAnxp YVBUVX rnnxrxmrxrnxr − − − +Σ= 21 1 11 Since, if ( ) ( )xprmmxp H A BU xmrm − =− 02 The solution of Amxn Xnxp = Bmxp is ( ) ( ) ( ) ( ) ( ) H AA H AA H AA H AAmxn † nxmnxn xnrnrmnxrxnnxrxnrnrmnxrxnnxr VVVVVVVVAAI −−−− =−+=− 22112211 ( ) [ ] ( ) ( ) ( ) ( ) ( ) H AAAH A H A rmxrnxrrn rmrxA AnA † nxm rxmrxrnxr xmrm rxmrxr rnnxmxr UV U U VVA 1 1 11 2 1 1 1 21 00 0 : − −−− − − Σ=                Σ = − −  Therefore: ( )  any nxpmxn † nxmnxnmxp † nxmnxp ZAAIBAX −+= Note: By writing the solution this way we lose the fact that we have only (n-r)xp different solutions as we have seen. Check: ( ) ( ) ( )  xprn xnrnrnnxrxmrxrnxr Y nxp H AAmxp H AAAnxp ZVVBUVX − −− +Σ= − 221 1 11
  • 71. 71 SOLO Matrices General Solution of Amxn Xnxp = Bmxp Moore-Penrose Pseudoinverse of A (continue - ): ( ) ( ) ( )  any nxpmxn † nxmnxnmxp † nxm any xprnAmxp H AAAnxp ZAAIBAYVBUVX rnnxrxmrxrnxr −+=+Σ= − − − 21 1 11 ( ) ( ) ( ) ( ) ( ) ( ) ( )xprmmxp H AA ANonBofprojection mxp H BUU BAAI orANB orARB xmrmrmmx T − + == − Ο/=∩ ∈ −− 0 0 22     Solutions exists iff: ( )H AR ( )AN ( )H AN ( )AR BXA = ( ) YVZAAI A † nxn 2=− BA† Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 B BA†Z i.e. the projection (Imxm – Amxn Anxm † ) of B on N (AH ) is zero. ( ) ( ) ( ) mxpmxp H AAmxp † nxmmxnmxm BUUBAAI mnrmrmmx 0 0 22 ==− −−  Table of Contents
  • 72. 72 SOLO Matrices General Solution of Amxn Xnxp = Bmxp Particular case (1) r = m ≤ n: solutions always exist ( )H AR ( )AN ( ) ( )ARBAN H ∈≡ &0 ( )AR BXA = ( ) YVZAAI A † nxn 2=− BA† Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 B BA†Z Since ( )[ ] ( )         Σ=Σ= − − H A H A mnmxAA H AAAmxn xnmn rxn mxmmxmnxnmxnmxm V V UVUA 2 1 1 0 ( ) ( ) ( ) ( )xprmmxp H ArmmxA BUU xmrmrmmx −− ≡⇒≡ −− 00 22 nxp unknowns ≥ mxp equations, meaning (n-m)xp degrees of freedom ( ) ( ) ( )  any nxpmxn † nxmnxnmxp † nxm any xprnAmxp H AAAnxp ZAAIBA YVBUVX rnnxrxmrxrnxr −+= +Σ= − − − 21 1 11 ( ) [ ] ( ) ( ) 11 11 1 1 21 0 : −− − − =Σ=        Σ = − H nxmmxn H nxm H AAA H A xmmn A AnA † nxm AAAUVUVVA mxmrxrnxrmxm mxm mnnxnxm  ( )[ ] ( ) ( ) [ ] ( ) mxm H A xmmn A AAH A H A mnmxAA † nxmmxn IUVV V V UAA mxm mxm mnnxnxm xnmn mxn mxmmxm =        Σ         Σ= − − − − − 0 0 1 1 21 2 1  Table of Contents
  • 73. 73 SOLO Matrices General Solution of Amxn Xnxp = Bmxp Particular case (2) r = n ≤ n: mxp equations ≥ nxp unknowns, meaning (n-m)xp constraints Only if solutions exist. ( ) ( )xprmmxp H A BU xmrm − =− 02 In this case we have nxp unknowns and mxp equations - (m-p)xp constraints = nxp independent equations, i.e. a unique solution: mxpnxmmxp H AAAnxp BABUVX nxmnxnnxn +− =Σ= 1 1 11 ( ) [ ] ( ) H A xnnm A AA H AAAmxn nxn nxn nmmxmxnnxnmxnmxm VUUVUA        Σ =Σ= − − 0 1 21  ( )[ ] ( ) H AAAH A H A nmnxAA † nxm nxmnxnnxn xmnm nxm nxnnxn UV U U VA 1 1 2 11 0: − − − Σ=         Σ= − ( )[ ] ( ) ( ) [ ] ( ) nxn H A xnnm A AAH A H A nmnxAAmxn † nxm IVUU U U VAA nxn nxn nmmxmxn xmnm nxm nxnnxn =        Σ         Σ= − − − − − 0 0 1 21 2 11  ( )H AR ( ) 0=AN ( )H AN ( )ARBXA = BA† 1nx y Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 BB has to be in the column space of A ( ) ( ) =∩∈ ANBorARB Table of Contents
  • 74. 74 SOLO Matrices General Solution of YpxmAmxn = Cpxn Y - pxm unknowns with pxn equations pxnmxnpxm CAY = Perform Singular Value Decomposition (S.V.D.) of Amxn: where ( ) 0,,, 21211 >≥≥≥=Σ rrA diagrxr σσσσσσ  UAmxm and VAnxn are unitary matrices, i.e.: [ ] ( ) ( ) [ ] H AA H AA H AAH A H A AA rmxrm rxr AAH A H A A H A UUUUUU U U UU I I UU U U UU =+=      =      =      = −− 2211 2 1 2121 2 1 0 0 [ ] ( ) ( ) [ ] H AA H AA H AAH A H A AA rnxrn rxr AAH A H A A H A VVVVVV V V VV I I VV V V VV =+=      =      =         = −− 2211 2 1 2121 2 1 0 0 ( )H AR ( )AN ( )H AN ( )ARCAY = 11 yAx H = C Y Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 ( ) [ ] ( ) ( ) ( ) ( ) ( )                Σ =Σ= − − −−− − H A H A rnxrmxrrm rnrxA AA H AAAmxn xnrn rxnrxr rmmxmxrnxnmxnmxm V V UUVUA 2 11 21 00 0 
  • 75. 75 SOLO Matrices we obtain: [ ] [ ] [ ]2121 2 11 21 00 0 AAAAH A H AA AA VCVCVV V V UUY  =              Σ or: ( ) ( )rnpxApxn rnnx VC −=− 02 px(n-r) - constraints equivalent to condition Cpxn∈ℜ (Amxn H ) nxrrxrmxr ApxnAApxm VCUY 111 =Σ pxr - independent equations pxm – unknowns since r ≤ m → # Eq. ≤ # Unknown ( ) [ ] ( ) ( ) ( ) ( ) ( )                Σ =Σ= − − −−− − H A H A rnxrmxrrm rnrxA AA H AAAmxn xnrn rxnrxr rmmxmxrnxnmxnmxm V V UUVUA 2 11 21 00 0  General Solution of YpxmAmxn = Cpxn ( )H AR ( )AN ( )H AN ( )ARCAY = C Y Null Space of A Ker (A) span by VA2 T Row Space of A span by VA1 T Column Space of A span by UA1 Left Null Space of A span by UA2 C has to be in the Row Space of A H AC ( ) ( ) Ο/=∩ ∈ ANC orARC HLet post-multiply by and using: [ ]21 VVpxnmxnpxm CAY = or: [ ] [ ]21 1 21 00 0 AA A AA VCVCUUY  =     Σ
  • 76. 76 SOLO Matrices Since UA1 T UA1=Ir & UA1 T UA2 = 0 the General Solution of YpxmAmxn = Cpxn is: where Xpx(m-r) is any px(m-r) matrix, i.e. we used all px(m-r) degrees of freedom. General Solution of YpxmAmxn = Cpxn nxrrxrmxr ApxnAApxm VCUY 111 =Σ pxr - independent equations pxm – unknowns since r ≤ m → # Eq. ≤ # Unknown This equation is a Necessary and Sufficient Condition for any solutions of equation YpxmAmxn = Cpxn. It is equivalent to Cpxn∈ℜ (Amxn) or Cpxn ∩ N (AT ) = ∅. If this condition is fulfilled, then from we have nxp unknowns ≥ rxp independent equations, that means (n-r)xp degrees of freedom. H ApxnAApxm nxrrxrmxr VCUY 111 =Σ 1 111 − Σ= rxrnxrmxr AApxnApxm VCUY px(n-r) - constraints equivalent to condition Cpxn∈ℜ (Amxn H )( ) ( )rnpxApxn rnnx VC −=− 02 ( ) ( ) H A any rmpx H AAApxnpxm xmrmrxmrxrnxr UXUVCY −− − +Σ= 21 1 11  ( )H AR ( )AN ( )H AN ( )ARCAY = C Y Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 C has to be in the Row Space of A H AC ( ) ( ) Ο/=∩ ∈ ANC orARC H H AC
  • 77. 77 SOLO Matrices Check: ( ) ( ) ( ) ( ) [ ] ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )[ ] ( ) ( ) ( ) ( ) ( ) ( )[ ] ( ) ( ) ( ) ( ) ( ) [ ] ( ) pxnH A H A AApxnH A H A ApxnApxn H A H A rnpxApxnH A H A rnxrmxrrm rnrxA rmpxAApxn H A H A rnxrmxrrm rnrxA I A H ArmpxA H AAApxnA H Armpx I A H AAApxn H A H A rnxrmxrrm rnrxA AA H Armpx H AAApxnmxnpxm C V V VVC V V VCVC V V VC V V XVC V V UUXUUVCUUXUUVC V V UUUXUVCAY xnrn rxn rnpxnxr xnrn rxn rnpx rnpxnxr xnrn rxn nxr xnrn rxnrxr rxrnxr xnrn rxnrxr rmmxxmrmrmmxrxmrxrnxrmxrxmrmmxrrxmrxrnxr xnrn rxnrxr rmmxmxrxmrmrxmrxrnxr =         =                   =         =                Σ Σ=                Σ         +Σ+Σ=                Σ +Σ= − − − − − −− − −−−− − −− − −−− − − − −−− − − − − − −−− − − − 2 1 21 2 1 0 21 2 1 1 2 111 11 2 11 22 0 21 1 11 0 1211 1 11 2 11 2121 1 11 0 00 0 00 0 00 0           General Solution of YpxmAmxn = Cpxn
  • 78. 78 SOLO Matrices where r is such that: Algorithm to solve YpxmAmxn = Cpxn: (1) Compute s.v.d. of Amxn and partition according to: ( ) 0,,, 21211 >≥≥≥=Σ rrA diagrxr σσσσσσ  (2) Check if: (3) If (2) is not true → no solution for (1) ( ) [ ] ( ) ( ) ( ) ( ) ( )                Σ =Σ= − − −−− − H A H A rnxrmxrrm rnrxA AA H AAAmxn xnrn rxnrxr rmmxmxrnxnmxnmxm V V UUVUA 2 11 21 00 0  General Solution of YpxmAmxn = Cpxn ( ) ( )rnpxApxn rnnx VC −=− 02 ( ) ( ) H A any rmpx H AAApxnpxm xmrmrxmrxrnxr UXUVCY −− − +Σ= 21 1 11  If (2) is true → px(m-r) solutions:
  • 79. 79 SOLO Matrices Moore-Penrose Pseudoinverse of A (continue - ): Define also Wpxm such that: ( ) ( )rmmxApxmrmpx UWX − =− 2: ( ) ( ) ( ) ( ) ( ) H AA H AA H AA H AAmxn † nxmnxn xnrnrmnxrxnnxrxnrnrmnxrxnnxr VVVVVVVVAAI −−−− =−+=− 22112211 ( ) [ ] ( ) ( ) ( ) ( ) ( ) H AAAH A H A rmxrnxrrn rmrxA AnA † nxm rxmrxrnxr xmrm rxmrxr rnnxmxr UV U U VVA 1 1 11 2 1 1 1 21 00 0 : − −−− − − Σ=                Σ = − −  Therefore:  ( )† nxmmxnmxm any pxm † nxmpxnpxm AAIWACY −+= Note: By writing the solution this way we lose the fact that we have only px(m-r) different solutions as we have seen. If ( ) ( )rnpxApxn rnnx VC − =− 02 General Solution of YpxmAmxn = Cpxn the solution of YpxmAmxn = Cpxn is ( ) ( ) H A any rmpx H AAApxnpxm xmrmrxmrxrnxr UXUVCY −− − +Σ= 21 1 11  Check: ( ) ( ) ( ) H A X Apxm H AAApxnpxm xmrm rmpx rmmxrxmrxrnxr UUWUVCY − − − +Σ= − 221 1 11 
  • 80. 80 SOLO Matrices Moore-Penrose Pseudoinverse of A (continue - ): Solutions exists iff: i.e. the projection (Inxn – Anxm † Amxn) of C on N (A) is zero. General Solution of YpxmAmxn = Cpxn ( ) ( )  ( )† nxmmxnmxm any pxm † nxmpxn H A any rmpx H AAApxnpxm AAIWAC UXUVCY xmrmrxmrxrnxr −+= +Σ= −− − 21 1 11  ( ) ( ) ( ) ( ) pxn H AApxnmxn † nxmnxnpxn xnrn rnpx rnnx VVCAAIC 02 0 2 ==− − − −  ( )H AR ( )AN ( )H AN ( )ARCAY = C pxm Y Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 C has to be in the Row Space of A H AC ( ) ( ) orANC orARC H Ο/=∩ ∈ ( ) ( ) ( ) ( ) ( ) pxn H AApxn ANonprojection mxnnxmnxnpxn xnrn rnpx rnnx VVCAAIC 02 0 2 ==− − − − +    ( ) ( ) ( ) ( ) H Armpx ANonWanyofprojection nxmmxnmxmpxm xrrm H pxm UXAAIW −− + =− 2    pxmW H AC ( ) ( ) ( ) ( ) ( ) ( ) ( ) pxn H AApxn ANonprojection mxn † nxmnxnpxn H xnrn rnpx rnnx VVC AAIC orANC orARC 02 0 2 == − Ο/=∩ ∈ − − −     ††† Table of Contents
  • 81. 81 SOLO Matrices Particular case (1) r = m ≤ n: ( )[ ] ( )         Σ=Σ= − − H A H A mnmxAA H AAAmxn xnmn rxn mxmmxmnxnmxnmxm V V UVUA 2 1 1 0 ( ) [ ] ( ) ( ) 11 11 1 1 21 0 : −− − − =Σ=        Σ = − H nxmmxn H nxm H AAA H A xmmn A AnA † nxm AAAUVUVVA mxmrxrnxrmxm mxm mnnxnxm  ( )[ ] ( ) ( ) [ ] ( ) mxm H A xmmn A AAH A H A mnmxAA † nxmmxn IUVV V V UAA mxm mxm mnnxnxm xnmn mxn mxmmxm =        Σ         Σ= − − − − − 0 0 1 1 21 2 1  General Solution of YpxmAmxn = Cpxn Only if solutions exist.( ) ( )rnpxApxn rnnx VC − =− 02 In this case we have pxm unknowns and pxn equations – px(n-m) constraints = pxm independent equations, i.e. a unique solution: † nxmpxn H AAApxnpxm AC UVCY rxmrxrnxr = Σ= − 1 1 11 ( )H AR ( )AN ( ) 0=H AN ( )ARCAY = C Null Space of A Ker (A) span by VA2 H Row Space of A span by VA1 H Column Space of A span by UA1 C has to be in the Row Space of A H AC ( ) ( ) ( ) ( ) ( ) ( ) pxn H AApxnmxn † nxmnxnpxn H xnrn rnpx rnnx VVCAAIC orANC orARC 02 0 2 ==− Ο/=∩ ∈ − − −  Table of Contents
  • 82. 82 SOLO Matrices Particular case (2) r = n ≤ n: pxn equations ≥ pxm unknowns, meaning px(m-n) constraints ( ) [ ] ( ) H A xnnm A AA H AAAmxn nxn nxn nmmxmxnnxnmxnmxm VUUVUA        Σ =Σ= − − 0 1 21  ( )[ ] ( ) H AAAH A H A nmnxAA † nxm nxmnxnnxn xmnm nxm nxnnxn UV U U VA 1 1 2 11 0: − − − Σ=         Σ= − ( )[ ] ( ) ( ) [ ] ( ) nxn H A xnnm A AAH A H A nmnxAAmxn † nxm IVUU U U VAA nxn nxn nmmxmxn xmnm nxm nxnnxn =        Σ         Σ= − − − − − 0 0 1 21 2 11  Since solutions always exist( ) ( )rnpxApxn rnnx VC − ≡− 02 pxm unknowns ≥ pxn equations, meaning px(m-n) degrees of freedom ( )H AR ( ) 0=AN ( )H AN ( )ARCAY = C Y Row Space of A span by VA1 H Column Space of A span by UA1 Left Null Space of A span by UA2 C has to be in the Row Space of A H AC ( ) CAAC † = ( ) H A UXAAIW 2 =− + ( ) ( ) ( ) ( ) ( ) ( ) pxn H AApxnmxn † nxmnxnpxn H xnrn rnpx rnnx VVCAAIC orANC orARC 02 0 2 ==− Ο/=∩ ∈ − − −  ( ) ( )  ( )† nxmmxnmxm any pxm † nxmpxn H A any rmpx H AAApxnpxm AAIWAC UXUVCY xmrmrxmrxrnxr −+= +Σ= −− − 21 1 11  General Solution of YpxmAmxn = Cpxn Table of Contents
  • 83. 83 SOLO Matrices Inverse of Partitioned Matrices ( ) ( ) ( ) ( )         −−− −−− =      − × − ××× − × − ×××× − × − × − ×××× − × − × − ××× − ×× ×× 11111 111111 mnnnnmmmnmmmmnnnnmmm mnnnnmmmmnnnnmmmmnnn mmnm mnnn BADCDCBADC BADCBADCBA CD BA if and exist. 1− ×nnA 1− ×nnC Let find the inverse of such that:      ×× ×× mmnm mnnn PN ML       ×× ×× mmnm mnnn CD BA       =            ×× ×× ×× ×× ×× ×× mmnm mnnn mmnm mnnn mmnm mnnn I I PN ML CD BA 0 0 Proof: nnnmmnnnnn INBLA ××××× =+1 ( ) nnnnnmmmmnnn ILDCBA ××× − ××× =−→ 1 2 nmnmmmnnnm NCLD ××××× =+ 0 nnnmmmnm LDCN ×× − ×× −=→ 1 3 mnmmmnmnnn PBMA ××××× =+ 0 mmmnnnmn PBAM ×× − ×× −=→ 1 4 mmmmmmmnnm IPCMD ××××× =+ ( ) mmmmmnnnnmmm IPBADC ××× − ××× =−→ 1
  • 84. 84 SOLO Matrices Inverse of Partitioned Matrices ( ) ( ) ( ) ( )         −−− −−− =      − × − ××× − × − ×××× − × − × − ×××× − × − × − ××× − ×× ×× 11111 111111 mnnnnmmmnmmmmnnnnmmm mnnnnmmmmnnnnmmmmnnn mmnm mnnn BADCDCBADC BADCBADCBA CD BA if and exist. 1− ×nnA 1− ×nnC Let find the inverse of such that:      ×× ×× mmnm mnnn PN ML       ×× ×× mmnm mnnn CD BA       =            ×× ×× ×× ×× ×× ×× mmnm mnnn mmnm mnnn mmnm mnnn I I PN ML CD BA 0 0 Proof (continue – 1): 1 ( ) nnnnnmmmmnnn ILDCBA ××× − ××× =− 1 2 nnnmmmnm LDCN ×× − ×× −= 1 3 4 mmmnnnmn PBAM ×× − ×× −= 1 ( ) mmmmmnnnnmmm IPBADC ××× − ××× =− 1 ( ) 11 − × − ×××× −= nmmmmnnnnn DCBAL ( ) 111 − × − ×××× − ×× −−= nmmmmnnnnmmmnm DCBADCN ( ) 11 − × − ×××× −= mnnnnmmmmm BADCP ( ) 111 − × − ×××× − ×× −−= mnnnnmmmmnnnmn BADCBAM q.e.d.
  • 85. 85 SOLO Matrices Inverse of Partitioned Matrices       =            ×× ×× ×× ×× ×× ×× mmnm mnnn mmnm mnnn mmnm mnnn I I CD BA PN ML 0 0 From: ( ) 1− ××××× −=→ mmmnnmmmmm CBNIPmmmmmmmnnm ICPBN ××××× =+we get: ( ) 11 − × − ×××× −= mnnnnmmmmm BADCPSubstitute: and: ( ) 111 − × − ×××× − ×× −−= nmmmmnnnnmmmnm DCBADCN ( ) ( ) 1111111 − ×× − × − ×××× − × − × − × − ××× −+=− mmmnnmmmmnnnnmmmmmmnnnnmmm CBDCBADCCBADC to obtain: By inter-changing , in this identity, we obtain:nmmnnnmm DBAC ×××× ↔↔ , ( ) ( ) 1111111 − ×× − × − ×××× − × − × − × − ××× −+=− nnnmmnnnnmmmmnnnnnnmmmmnnn ADBADCBAADCBA
  • 86. 86 SOLO Pre-multiplying this identity by we obtain Matrices Inverse of Partitioned Matrices Let prove the identity: ( ) ( ) 111111 − ×× − × − ××× − × − ×××× − × −=− mmmnnmmmmnnnmnnnnmmmmnnn CBDCBABADCBA ( ) ( ) 1111 − × − ×××× − ×× − ××× −− mnnnnmmmmnnnnmmmmnnn BADCBADCBA Proof: ( )( ) 1111 − × − ×××× − ×× − ××× −−≡ mnnnnmmmmnnnnmmmmnmn BADCBADCBB ( )( ) 11111 − ×× − × − ×××× − ××× − ×× ≡−−≡ mmmnmnnnnmmmmnnnnmmmmmmn CBBADCBADCCB ( ) 11 − × − ××× − nmmmmnnn DCBA ( ) ( ) ( ) ( ) 111 111111 − ×× − × − ××× − × − ×××× − ×× − ××× − × − ××× −≡ −−− mmmnnmmmmnnn mnnnnmmmmnnnnmmmmnnnnmmmmnnn CBDCBA BADCBADCBADCBA q.e.d. ( ) ( ) 111111 − × − ×××× − × − ×× − × − ××× −=− nmmmmnnnnmmmnnnmmnnnnmmm DCBADCADBADC By inter-changing ,in the first identity, we obtain:nmmnnnmm DBAC ×××× ↔↔ , ( ) ( ) 111111 − ×× − × − ××× − × − ×××× − × −=− nnnmmnnnnmmmnmmmmnnnnmmm ADBADCDCBADC q.e.d.
  • 87. 87 SOLO Matrices Inverse of Partitioned Matrices By using the identities: ( ) ( ) 111111 − × − ×××× − × − ×× − × − ××× −=− nmmmmnnnnmmmnnnmmnnnnmmm DCBADCADBADC We obtain: ( ) ( ) ( ) ( )         −−− −−− =      − × − ××× − × − ×××× − × − × − ×××× − × − × − ××× − ×× ×× 11111 111111 mnnnnmmmnmmmmnnnnmmm mnnnnmmmmnnnnmmmmnnn mmnm mnnn BADCDCBADC BADCBADCBA CD BA if and exist. 1− ×nnA 1− ×nnC ( ) ( ) ( ) ( )         −−− −−−+ =      − × − ××× − ×× − × − ××× − × − ×××× − × − ×× − × − ×××× − × − × − ×× ×× 11111 111111111 mnnnnmmmnnnmmnnnnmmm mnnnnmmmmnnnnnnmmnnnnmmmmnnnnn mmnm mnnn BADCADBADC BADCBAADBADCBAA CD BA ( ) ( ) 1111111 − ×× − × − ×××× − × − × − × − ××× −+=− nnnmmnnnnmmmmnnnnnnmmmmnnn ADBADCBAADCBA
  • 88. 88 SOLO Matrices Inverse of Partitioned Matrices If and exist, performing the computation M-1 M, we can prove: 1− ×nnA 1− ×nnC         − =      − × − ×× − × × − × − ×× ×× 111 11 00 mmnnnmmm mnnn mmnm mnnn CADC A CD A 1 2         − =      − ×× − ×× − × − × − ×× ×× 1 1111 00 mmnm mmmnnnnn mmnm mnnn C CBAA C BA 3         =      − ×× × − × − ×× ×× 1 11 0 0 0 0 mmnm mnnn mmnm mnnn C A C A 4 If and : T nnnn AA ×× = T nnnn CC ×× = ( ) ( ) ( ) ( )         −−− −−− =      − × − ××× − × − ×××× − × − × − ×××× − × − × − ××× − ×× ×× 11111 111111 mnnnnm T mmnm T mmmnnnnm T mm mnnnnm T mmmnnnnm T mmmnnn mmnm T mnnn BABCBCBABC BABCBABCBA CB BA Because this is a symmetric matrix ( ) ( ) 111111 − ×× − × − ××× − × − ×××× − × −=− nnnm T mnnnnm T mmnm T mmmnnnnm T mm ABBABCBCBABC Also: ( ) ( ) 1111111 − ×× − × − ×××× − × − × − × − ××× −+=− mmmnnm T mmmnnnnm T mmmmmnnnnm T mm CBBCBABCCBABC
  • 89. 89 SOLO Matrices Inverse of Partitioned Matrices If m=n and and also exist: 1− ×nnB 1− ×nn D5 ( ) ( )[ ] 111111 − × − ×× − ××× − × − ×××× − × −−=−− nnnnnnnnnnnnnnnnnnnnnnnn ABBADCBADCBA ( ) ( )[ ] 111111 − × − ×× − ××× − × − ×××× − × −−=−− nnnnnnnnnnnnnnnnnnnnnnnn CDDCBADCBADC also we obtain ( ) ( ) ( ) ( )         −− −− =      − × − ××× − × − ××× − × − ××× − × − ××× − ×× ×× 1111 11111 nnnnnnnnnnnnnnnn nnnnnnnnnnnnnnnn nnnn nnnn BADCCDAB ABCDDCBA CD BA Table of Contents
  • 90. 90 SOLO Matrices Matrix Inverse Lemmas Identities ( ) ( ) 1111111 − ×× − × − ×× − ×× − × − × − ×××× +−=+ mmmnnmmmmnnnnmmmmmmnnnnmmm CBDCBADCCBADC1 Proof: ( ) ( ) 1111111 − ×× − × − ×××× − × − × − × − ××× −+=− mmmnnmmmmnnnnmmmmmmnnnnmmm CBDCBADCCBADC In the identity: substitute by . 1− ×nnA nnA ×− Substitute by and by in (1). 1− ×nnAnnA × mmC × 1− ×mmC2 ( ) ( ) mmmnnmmmmnnnnmmmmmmnnnnmmm CBDCBADCCBADC ×× − ××××××× − × − ×× − × +−=+ 1111 Substitute in (1) nnnn IA ×× =3 ( ) ( ) 111111 − ×× − × − ×××× − × − × − ××× +−=+ mmmnnmmmmnnnnmmmmmmnnmmm CBDCBIDCCBDC Substitute in (2) nnnn IA ×× =4 ( ) ( ) mmmnnmmmmnnnnmmmmmmnnmmm CBDCBIDCCBDC ×× − ××××××× − ×× − × +−=+ 111
  • 91. 91 SOLO Matrices Matrix Inverse Lemmas Identities 5Substitute in (1), (2), (3), (4) by . (We don’t assume symmetric and )nmD × nm T B × nnA × mmC × ( ) ( ) 1111111 − ×× − × − ×× − ×× − × − × − ×××× +−=+ mmmnnm T mmmnnnnm T mmmmmnnnnm T mm CBBCBABCCBABC ( ) ( ) mmmnnm T mmmnnnnm T mmmmmnnnnm T mm CBBCBABCCBABC ×× − ××××××× − × − ×× − × +−=+ 1111 ( ) ( ) 111111 − ×× − × − ×××× − × − × − ××× +−=+ mmmnnm T mmmnnnnm T mmmmmnnm T mm CBBCBIBCCBBC ( ) ( ) mmmnnm T mmmnnnnm T mmmmmnnm T mm CBBCBIBCCBBC ×× − ××××××× − ×× − × +−=+ 111 From this we get: 6Substitute in (3) mmmm IC ×× = ( ) ( ) mnnmmnnnnmmmmnnmmm BDBIDIBDI × − ××××× − ××× +−=+ 11 ( ) ( ) ( )( ) ( ) ( ) ( ) ( ) ( ) ( ) mnnmmnnmmmmnnmmmmnnmmmmnnmmm mnnmmmmnnmmnnmmmmnnmmmmnnmmm mnnmmmmmmnnmmnnnnm BDBDIBDIBDIBDI BDIBDBDIBDIBDI BDIIBDBID ×× − ××× − ×××××× − ××× − ××××× − ××× − ×××××× − ××××× − ×××× +=+−++= +=+−++= +−=+ 111 111 11
  • 92. 92 SOLO Matrices Matrix Inverse Lemmas Identities 7Substitute in the identity ( ) ( ) 111111 − ×× − × − ××× − × − ×××× − × −=− mmmnnmmmmnnnmnnnnmmmmnnn CBDCBABADCBA nnnn IA ×× −= and mmmm IC ×× = to obtain: ( ) ( ) mnnmmnnnmnnmmmmn BDBIBDIB × − ××× − ×××× +=+ 11 Pre-multiplying this by we get (6).nmD × By using a similar path with the identity ( ) ( ) 111111 − ×× − × − ××× − × − ×××× − × −=− nnnmmnnnnmmmnmmmmnnnnmmm ADBADCDCBADC nnnn IA ×× −= and mmmm IC ×× = to obtain:with ( ) ( ) nmmnnmmmnmmnnnnm DBDIDBID × − ××× − ×××× +=+ 11 Post-multiplying this by we get (6).mnB ×
  • 93. 93 SOLO Matrices By matrix manipulation we obtain: Matrix Inverse Lemmas Identities 8In the identity ( ) ( ) 111111 − ×× − × − ××× − × − ×××× − × −=− mmmnnmmmmnnnmnnnnmmmmnnn CBDCBABADCBA pre-multiplying Anxn by and post-multiplying by Cmxm we get: ( ) ( ) mnnmmmmnnnnnmmmnnnnmmmmn BDCBAACBADCB × − × − ××××× − × − ×××× −=− 1111 ( ) ( ) mnnnnmmmmnnnmnnnnmmmmmmn BADCBIBADCIB × −− ×× − ××× − × − ×× − ××× −=− 111111 Use now the identity ( ) ( ) 111111 − ×× − × − ××× − × − ×××× − × −=− nnnmmnnnnmmmnmmmmnnnnmmm ADBADCDCBADC Pre-multiplying by Cmxm and post-multiplying by Anxn we get: ( ) ( ) nmmnnnnmmmmmnnnmmmmnnnnm DBADCCADCBAD × − × − ××××× − × − ×××× −=− 1111 By matrix manipulation we obtain ( ) ( ) nmmmmnnnnmmmnmmmmnnnnnnm DCBADIDCBAID × −− ×× − ××× − × − ×× − ××× −=− 111111 Table of Contents
  • 94. 94 SOLO Matrices Matrix Schwarz Inequality ( ) ( ) ( )QPPPQPQQ TTTTT 1− ≥ Table of Contents Hermann Amandus Schwarz 1843 - 1921 yxyx ≤>< , Let x, y be the elements of an Inner Product space X, than : This is the Schwarz Inequality. Let Pmxn and Qmxl be two matrices such that PT P is nonsingular, then: ( ) ( ) ( ) CxxQPPPQPxxQQx TTTTTTT ∈∨≥ −1 i.e.,: Furthermore equality holds if and only if exists a matrix Snxl such that Q = P S. Proof: Start from the inequality: and choose( ) ( ) 0≥−− SPQSPQ T ( ) ( )QPPPS TT 1− = ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) 0 1 1111 ≥−= +−−= +−−=−− − −−−− QPPPQPQQ QPPPPPPPPQQPPPPQQPPPPQQQ SPPSQPSSPQQQSPQSPQ TTTTT TTTTTTTTTTTT TTTTTTT The inequality becomes equality if and only if : that is equivalent with ( ) ( ) 0=−− SPQSPQ T SPQ =
  • 95. 95 SOLO Matrices Trace of a Square Matrix The trace of a square matrix is defined as ( ) ( )T nn n i iinn AtraceaAtrace × = × == ∑1 : q.e.d. ( ) ( )ABtraceBAtrace =1 Proof: ( ) ∑ ∑= =         = n i n j jiij baBAtrace 1 1 ( ) ( )BAtracebaabABtrace n i n j jiij n j n i ijji ==      = ∑∑∑ ∑ = == = 1 11 1 ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )ABtraceBAtraceBAtraceABtraceABtraceBAtrace TTTT 111 =≠===2 Proof: ( ) ( ) ( )ABtraceBAtracebabaBAtrace n i n j jiij n i n j ijij T ==        ≠        = ∑ ∑∑ ∑ = == = 1 11 1 ( ) ( )T n j n i ijij T BAtraceabABtrace =      = ∑ ∑= =1 1 q.e.d.
  • 96. 96 SOLO Matrices Trace of a Square Matrix The trace of a square matrix is defined as ( ) ( )T nn n i iinn AtraceaAtrace × = × == ∑1 : 3 Proof: q.e.d. ( ) ( ) ( )∑= − == n i i APAPtraceAtrace 1 1 λ where P is the eigenvector matrix of A related to the eigenvalue matrix Λ of A by           =Λ= n PPPA λ λ    0 01 ( ) ( ) ( ) ( )AtraceAPPtracePAPtrace == −− 1 1 1           =Λ= n PPPA λ λ    0 01           =Λ=→ − n PAP λ λ    0 01 1 ( ) ( ) ∑= − =Λ=→ n i itracePAPtace 1 1 λ
  • 97. 97 SOLO Matrices Trace of a Square Matrix The trace of a square matrix is defined as ( ) ( )T nn n i iinn AtraceaAtrace × = × == ∑1 : Proof: q.e.d. Definition 4 ( )AtraceA ee =det ( )AtraceA eeePe P PePPePe n i i ====== ∑=ΛΛΛ−Λ− 1 detdetdet det 1 detdetdetdetdet 11 λ If aij are the coefficients of the matrix Anxn and z is a scalar function of aij, i.e.: ( ) njiazz ij ,,1, == then is the matrix nxn whose coefficients i,j areA z ∂ ∂ nji a z A z ijij ,,1,: = ∂ ∂ =      ∂ ∂ (see Gelb “Applied Optimal Estimation”, pg.23)
  • 98. 98 SOLO Matrices Trace of a Square Matrix The trace of a square matrix is defined as ( ) ( )T nn n i iinn AtraceaAtrace × = × == ∑1 : Proof: q.e.d. 5 ( ) ( ) ( ) A Atrace I A Atrace T n ∂ ∂ == ∂ ∂ 1 ( )    = ≠ == ∂ ∂ =      ∂ ∂ ∑= ji ji a aA Atrace ij n i ii ijij 1 0 1 δ 6 ( ) ( ) ( ) ( ) nmmnTTT RBRCCBBC A BCAtrace A ABCtrace ×× ∈∈== ∂ ∂ = ∂ ∂ 1 Proof: ( ) ( ) ( )[ ]ij T ji m p pijp ik jl n l m p n k klpklp ijij BCBCbcabc aA ABCtrace === ∂ ∂ =      ∂ ∂ ∑∑∑∑ = = = = = = 11 1 1 q.e.d. 7 If A, B, C ∈ Rnxn ,i.e. square matrices, then ( ) ( ) ( ) ( ) ( ) ( ) TTT CBBC A BCAtrace A CABtrace A ABCtrace == ∂ ∂ = ∂ ∂ = ∂ ∂ 11
  • 99. 99 SOLO Matrices Trace of a Square Matrix The trace of a square matrix is defined as ( ) ( )T nn n i iinn AtraceaAtrace × = × == ∑1 : Proof: q.e.d. 8 ( )( ) ( )( ) ( )( )( ) nmmn TTT RBRCBC A ABCtrace A BCAtrace A ABCtrace ×× ∈∈= ∂ ∂ = ∂ ∂ = ∂ ∂ 721 9 ( )( ) ( )( ) ( )( ) BC A BCAtrace A CABtrace A ABCtrace TTT 811 = ∂ ∂ = ∂ ∂ = ∂ ∂ If A, B, C ∈ Rnxn ,i.e. square matrices, then 10 ( ) T A A Atrace 2 2 = ∂ ∂ ( ) ( ) ( )ij T jiji n l n m mllm ijijij Aaaaa aa Atrace A Atrace 2 1 1 22 =+=      ∂ ∂ = ∂ ∂ =      ∂ ∂ ∑∑= = 11 ( ) ( ) 1− = ∂ ∂ kT k Ak A Atrace Proof: ( ) ( ) ( ) ( ) ( ) 1111 −−−− =+++= ∂         ⋅∂ = ∂ ∂ kT k kTkTkT k k AkAAA A AAAtrace A Atrace       q.e.d.
  • 100. 100 SOLO Matrices Trace of a Square Matrix The trace of a square matrix is defined as ( ) ( )T nn n i iinn AtraceaAtrace × = × == ∑1 : Proof: q.e.d. 12 ( ) T A A e A etrace = ∂ ∂ ( ) ( ) ( ) T A n k n k kT n kk kT n n k k n n k k n A eA k A k k k A trace Ak A trace AA etrace ===      ∂ ∂ =      ∂ ∂ = ∂ ∂ ∑ ∑∑∑ = = →∞ →− − →∞ = →∞ = →∞ 1 0 1 1 00 ! 1 lim ! lim ! lim ! lim 13 ( )( ) ( )( ) ( ) ( ) ( )( ) ( )( ) ( ) ( ) ( )( ) ( )( ) ( ) TT TTTTTTTTT TTTTT TTT BACBAC A ACABtrace A BACAtrace A ABACtrace A CABAtrace A BACAtrace A CABAtrace A ACABtrace A BACAtrace A ABACtrace += ∂ ∂ = ∂ ∂ = ∂ ∂ = ∂ ∂ = ∂ ∂ = ∂ ∂ = ∂ ∂ = ∂ ∂ = ∂ ∂ 111 21 11 ( ) ( ) ( ) ( ) ( ) ( ) TTTT TTT BACBACCABBAC A ABACtrace A ABACtrace A ABACtrace +=+== ∂ ∂ + ∂ ∂ = ∂ ∂ + 86 2 2 1 1 Proof: q.e.d. 14 ( ) ( )( ) A A AAtrace A AAtrace TT 2 13 = ∂ ∂ = ∂ ∂
  • 101. 101 SOLO Matrices Trace of a Square Matrix The trace of a square matrix is defined as ( ) ( )T nn n i iinn AtraceaAtrace × = × == ∑1 : Proof: 15 ( ) ( )TTTTT ABBAABBA A ABAtrace +=+= ∂ ∂ Table of Contents ( ) ( ) ( )ij TTTT n l jlli n k kijk n l n l n k klmklm ijijij ABBAbababaa aa ABAtrace A ABAtrace +=+=      ∂ ∂ = ∂ ∂ =      ∂ ∂ ∑∑∑∑∑ === = = 111 1 1 q.e.d. 16 ( ) TTTTTT CABBAC A ABACtrace += ∂ ∂ ( ) ( ) ( )ij TTTTTT n l n r lirljr n k n m mikmjk n l n r rljrli n k n m mikmjk n l n k n m n r rlmrkmlk ijijij CABBACcabbac abcbacabac aa ABACtrace A ABACtrace +=+= +=      ∂ ∂ = ∂ ∂ =      ∂ ∂ ∑∑∑∑ ∑∑∑∑∑∑∑∑ = == = = == == = = = 1 11 1 1 11 11 1 1 1 Proof: q.e.d.
  • 102. 102 SOLO References [1] Pease, “Methods of Matrix Algebra” ,Mathematics in Science and Engineering, Vol.16, Academic Press, 1965 Matrices [2] S. Hermelin, “Robustness and Sensitivity Design of Linear Time-Invariant Systems” PhD Thesis, Stanford University, 1986 Table of Contents
  • 103. January 6, 2015 103 SOLO Technion Israeli Institute of Technology 1964 – 1968 BSc EE 1968 – 1971 MSc EE Israeli Air Force 1970 – 1974 RAFAEL Israeli Armament Development Authority 1974 – 2013 Stanford University 1983 – 1986 PhD AA Matrices
  • 104. 104 SOLO Derivatives of Matrices Matrices ljik ij kl a a δδ= ∂ ∂ For vector forms j i ijii i i y x y x y x y x y x y x ∂ ∂ =      ∂ ∂ ∂ ∂ =      ∂ ∂ ∂ ∂ =      ∂ ∂ ::: We have the following expressions: ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )( ) ( ) ( ) ( ) ( ) ( )HH TT XX XX XXtraceX XXtraceXX XXXX YXYXYX YXYXYX XtraceXtrace YXYX XX constAifA ∂=∂ ∂=∂ ∂=∂ ∂=∂ ∂−=∂ ∂⊗+⊗∂=⊗∂ ∂⋅+⋅∂=⋅∂ ∂=∂ ∂+∂=+∂ =∂ ==∂ − − −−− 1 1 111 detln detdet 0 αα
  • 105. 105 SOLO Derivatives of Determinants Matrices ( ) ( )       ∂ ∂ = ∂ ∂ − x Y YtraceY x Y 1 det det ( ) ( )             ∂ ∂       ∂ ∂ −       ∂ ∂       ∂ ∂ +             ∂ ∂ ∂ ∂ = ∂ ∂ ∂ ∂ −− −− − x Y Y x Y Ytrace x Y Ytrace x Y Ytrace x x Y YtraceY x x Y 11 11 1 det det General Form
  • 106. 106 SOLO Derivatives of Determinants Matrices ( ) ( )( ) ( ) ( ) ( ) ( )( ) ( )( ) 11 1 detdet det det det det det −− − == ∂ ∂ = ∂ ∂ = ∂ ∂ ∑ TT ij k jk ik T XBXAXBXA X BXA XX X X XX X X δ Linear Form Square Forms If X is Square and Invertible, then ( ) ( ) TT T XXAX X XAX − = ∂ ∂ det2 det If X is Not Square but A is Symmetric, then ( ) ( ) ( ) 1 det2 det − = ∂ ∂ XAXXAXAX X XAX TT T If X is Not Square and A is Not Symmetric, then ( ) ( ) ( ) ( )[ ]11 det det −− += ∂ ∂ XAXXAXAXXAXAX X XAX TTTT T
  • 107. 107 SOLO Derivatives of Determinants Matrices ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) Tk k T T T T T XXk X X XX X X X X XX X X XX − − = ∂ ∂ == ∂ ∂ −= ∂ ∂ = ∂ ∂ det det 22 detln 2 detln 2 detln 1T1- † † Nonlinear Form
  • 108. 108 SOLO Derivatives of an Inverse Matrices 11 1 −− − ∂ ∂ −= ∂ ∂ Y x Y Y x Y From this it follows ( ) ( ) ( ) ( ) ( ) ( )( ) ( ) ( ) ( )( ) ( ) ( )( )T T T TTT T jlki ij kl AXAX X AXtrace XABX X BXAtrace XX X X XbaX X bXa XX X X 11 1 11 1 11 1 1 11 1 det det −− − −− − −− − −− − −− − ++−= ∂ +∂ −= ∂ ∂ −= ∂ ∂ −= ∂ ∂ −= ∂ ∂
  • 109. 109 SOLO Derivatives of Matrices, Vectors and Scalar Forms Matrices First Order ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )ij nm mjin mn ij T ij mn njim mn ij ij ij T TTT T TT T T TT AJA X AX AJA X AX J X X aa X aXa X aXa ab X bXa ba X bXa a x xa x ax == ∂ ∂ == ∂ ∂ = ∂ ∂ = ∂ ∂ = ∂ ∂ = ∂ ∂ = ∂ ∂ = ∂ ∂ = ∂ ∂ δ δ n mJ mn ↑ ←                 = 000 010 000     
  • 110. 110 SOLO Derivatives of Matrices, Vectors and Scalar Forms Matrices Second Order ( ) ( ) ( ) ( )[ ] ( ) ( ) ( ) ( ) ( ) ( ) ( ) jlikkl ijijijT ij T ilkjkl T lj ij kl T TTT T TT TT kl kl klmn mnkl ij JXBJJBX X XBX XBBX X XBX bxBCDdxDCB x dxDCbxB bccbX X cXXb XXX X δδ δδ =+= ∂ ∂ += ∂ ∂ +++= ∂ ++∂ += ∂ ∂ = ∂ ∂ ∑∑ 2 n mJ mn ↑ ←                 = 000 010 000     
  • 111. 111 SOLO Derivatives of Matrices, Vectors and Scalar Forms Matrices Second Order (continue) ( ) ( ) ( ) ( ) ( )[ ] ( )( ) TTT TTT TT T T bcbXDDcbXDcbX X bcXDcbXD X cXDXb xBB x xBx ++=++ ∂ ∂ += ∂ ∂ += ∂ ∂ Assume W is symmetric ( ) ( )[ ] ( ) ( ) ( )[ ] ( ) ( ) ( )[ ] ( ) ( ) ( )[ ] ( ) ( ) ( )[ ] ( ) TT T T T TT ssAxWsAxWsAx A sAxWsAxWsAx x sxWsxWsx s sxWsxWsx x sAxWAsAxWsAx s −−=−− ∂ ∂ −=−− ∂ ∂ −−=−− ∂ ∂ −=−− ∂ ∂ −−=−− ∂ ∂ 2 2 2 2 2
  • 112. 112 SOLO Derivatives of Matrices, Vectors and Scalar Forms Matrices Higher Order and Nonlinear ( ) ( )∑ − = −− = ∂ ∂ 1 0 1 n r kl rnijr ij kl n XJX X X [ ] ( ) ( ) ( )[ ] ( ) ( ) ( )[ ]∑ ∑ − = −−−− −− − = += ∂ ∂ = ∂ ∂ 1 0 11 1 1 0 n r TrnTnTrrTnTrnnTnT Trn n r TTrnT XbaXXXXbaXbXXa X XbaXbXa X

Editor's Notes

  • #44: http://en.wikipedia.org/wiki/Householder_transformation http://www-history.mcs.st-andrews.ac.uk/Biographies/Householder.html
  • #49: http://en.wikipedia.org/wiki/Moore-Penrose_inverse
  • #95: C.K. Chui, G. Chen, “Kalman Filtering with Real Time Applications”. Springer Verlag, 1987, pp. 2-3
  • #105: Petersen &amp; Pedersen, “Matrix Cookbook, http://matrixcookbook.com
  • #106: Petersen &amp; Pedersen, “Matrix Cookbook, http://matrixcookbook.com
  • #107: Petersen &amp; Pedersen, “Matrix Cookbook, http://matrixcookbook.com
  • #108: Petersen &amp; Pedersen, “Matrix Cookbook, http://matrixcookbook.com
  • #109: Petersen &amp; Pedersen, “Matrix Cookbook, http://matrixcookbook.com
  • #110: Petersen &amp; Pedersen, “Matrix Cookbook, http://matrixcookbook.com
  • #111: Petersen &amp; Pedersen, “Matrix Cookbook, http://matrixcookbook.com
  • #112: Petersen &amp; Pedersen, “Matrix Cookbook, http://matrixcookbook.com
  • #113: Petersen &amp; Pedersen, “Matrix Cookbook, http://matrixcookbook.com