Matrices+ 03 - Eigenvalues in 3D: UNIZOR.COM - Math+ & Problems - Matrices

Notes to a video lecture on http://www.unizor.com

Matrices+ 02
Eigenvalues in 3D

Problem A

Find all eigenvalues and eigenvectors of this 3⨯3 matrix, if it's known that one of the eigenvalues is 10.

4	8	−4
4	−2	−4
−8	−4	−10

Note A
We specify one of the eigenvalues because general calculation of all eigenvalues in 3D leads to a polynomial equation of the 3rd degree.
Since we want to avoid the necessity to solve it, we specify one eigenvalue, which leads to finding the other two by solving a quadratic equation, which should not present any problem.

Solution A

If matrix A transforms vector v to a collinear one with the magnitude of the original one multiplied by a factor λ, the following matrix equation must hold
A·v = λ·v
or in coordinate form for 3⨯3 matrix

a1,1 a1,2 a1,3 a2,1 a2,2 a2,3 a3,1 a3,2 a3,3

·

v1 v2 v3

=

λ

·

v1 v2 v3

which is equivalent to
(a_1,1−λ)·v₁+a_1,2·v₂+a_1,3·v₃ = 0
a_2,1·v₁+(a_2,2−λ)·v₂+a_2,3·v₃ = 0
a_3,1·v₁+a_3,2·v₂+(a_3,3−λ)·v₃ = 0

This is a system of three linear equations with four unknowns λ, v₁, v₂ and v₃.

One trivial solution would be v₁=0, v₂=0 and v₃=0, in which case λ can take any value.
This is not a case worthy of analyzing.

If the matrix of coefficients of this system has a non-zero determinant, this trivial solution would be the only one.
Therefore, if we are looking for a non-trivial solution, the matrix's determinant must be zero, which gives a specific condition on the value of λ.

Therefore, a necessary condition for existence of eigenvectors v other than null-vector is
det(A) =
= (a_1,1−λ)·(a_2,2−λ)·(a_3,3−λ)+
+a_2,1·a_3,2·a_1,3+a_1,2·a_2,3·a_3,1−
−a_1,3·(a_2,2−λ)·a_3,1−
−a_1,2·a_2,1·(a_3,3−λ)−
−(a_1,1−λ)·a_2,3·a_3,2 = 0
Using values of a matrix of this problem, this equation is
(4−λ)·(−2−λ)·(−10−λ)+
+8·(−4)·(−8)+4·(−4)·(−4)−
−(−4)·(−2−λ)·(−8)−
−8·4·(−10−λ)−
−(4−λ)·(−4)·(−4) = 0
Simplifying this equation, we get
−λ³−8·λ²+108·λ+720=0

First of all, we can check if the eigenvalue 10 given in the problem is the root of this equation.
Indeed, the following is true.
−10³−8·10²+108·10+720=0

Since we know one root λ₁=10 of cubic equation, we can represent the left side of this equation as a product of (λ−10) and a quadratic polynomial with easily calculated coefficients, getting equation
(λ−10)·(−λ²−18·λ−72) = 0

To get all the roots of the original cubic equation, we have to solve a quadratic equation
−λ²−18·λ−72 = 0
or, using a canonical form,
λ²+18·λ+72 = 0
Its roots are
λ_2,3 = −9±√81−72 = −9±3

So, we have three eigenvalues for our matrix: −12, −6 and 10.

Consider now that we have determined eigenvalue λ and would like to find eigenvector v=||v₁,v₂,v₃|| transformed into a collinear one by matrix A with this exact factor of change in magnitude.

If some vector v=||v₁,v₂,v₃|| that is transformed into a collinear one with a factor λ exists, vector s·v, where s is any real non-zero number, would have exactly the same quality because of associativity and commutativity of multiplication by a scalar.
A·(s·v) = (A·s)·v = (s·A)·v =
= s·(A·v) = s·(λ·v)= λ·(s·v)

Therefore, we don't need to determine exact values v₁, v₂ and v₃, we just need to determine only the direction of vector v=||v₁,v₂,v₃||.

Let's start with the first eigenvalue λ₁=−12.
Using it in the equation A·v=λ·v, we have the following system of equations
4·v₁+8·v₂−4·v₃=−12·v₁
4·v₁−2·v₂−4·v₃=−12·v₂
−8·v₁−4·v₂−10·v₃=−12·v₃
Bringing everything to the left side, get this system
16·v₁+8·v₂−4·v₃=0
4·v₁+10·v₂−4·v₃=0
−8·v₁−4·v₂+2·v₃=0

As expected, the determinant of the coefficients of this system is zero and this system of equations is linearly dependent (third equation multiplied by −2 gives the first), as it should be, because otherwise the only solution would be a null-vector.
So, we drop the first equation and resolve the remaining two equations for v₂ and v₃ in terms of v₁.
Here is what remains
4·v₁+10·v₂−4·v₃=0
−8·v₁−4·v₂+2·v₃=0
Divide by 2 all of them to simplify and find v₃ from the second equation
2·v₁+5·v₂−2·v₃=0
v₃ = 4·v₁+2·v₂
Substituting v₃ into the first equation:
2·v₁+5·v₂−2·(4·v₁+2·v₂)=0
or
−6·v₁+v₂=0
Therefore,
v₂=6·v₁ and v₃ = 16·v₁

Regardless of the value of v₁, vector ||v₁,6·v₁,16·v₁|| is an eigenvector.
Set v₁=1 for simplicity, and vector ||1,6,16|| should be an eigenvector.

Let's check it out.

4 8 −4 4 −2 −4 −8 −4 −10

·

1 6 16

=

=

4+48−64 4−12−64 −8−24−160

=

−12 −72 −192

=

=

−12·

1 6 16

Which confirms that vector ||1,6,16|| (and any collinear to it) is an eigenvector with −12 as its eigenvalue.

Let's do the same calculations with the second eigenvalue λ₂=−6.
Using it in the equation A·v=λ·v, we have the following system of equations
4·v₁+8·v₂−4·v₃=−6·v₁
4·v₁−2·v₂−4·v₃=−6·v₂
−8·v₁−4·v₂−10·v₃=−6·v₃
Bringing everything to the left side, get this system
10·v₁+8·v₂−4·v₃=0
4·v₁+4·v₂−4·v₃=0
−8·v₁−4·v₂−4·v₃=0
Dividing the first equation by 2, the second - by 4 and the third - by −4, all these equations yield a simpler system
5·v₁+4·v₂−2·v₃=0
v₁+v₂−v₃=0
2·v₁+v₂+v₃=0

As expected, the determinant of the coefficients of this system is zero and the equations are linearly dependent (the third equation equals the first minus triple the second one), as it should be, because otherwise the only solution would be a null-vector.
So, we drop the first equation and resolve the remaining two equations for v₂ and v₃ in terms of v₁.
Here is what remains
v₁+v₂−v₃=0
2·v₁+v₂+v₃=0

Find v₃ from the first equation
v₃ = v₁+v₂
Substituting v₃ into the second equation:
2·v₁+v₂+(v₁+v₂)=0
or
3·v₁+2·v₂=0

Therefore,
v₂=−(3/2)·v₁ and v₃ = −(1/2)·v₁

Regardless of the value of v₁, vector ||v₁,−(3/2)·v₁,−(1/2)·v₁|| is an eigenvector.
Set v₁=2 for simplicity, and vector ||2,−3,−1|| should be an eigenvector.

Let's check it out.

4 8 −4 4 −2 −4 −8 −4 −10

·

2 −3 −1

=

=

8−24+4 8+6+4 −16+12+10

=

−12 18 6

=

=

−6·

2 −3 −1

Which confirms that vector ||2,−3,−1|| (and any collinear to it) is an eigenvector with −6 as its eigenvalue.

Finally, let's do the same for the third eigenvalue 10.
Using it in the equation A·v=λ·v, we have the following system of equations
4·v₁+8·v₂−4·v₃=10·v₁
4·v₁−2·v₂−4·v₃=10·v₂
−8·v₁−4·v₂−10·v₃=10·v₃
Bringing everything to the left side, get this system
−6·v₁+8·v₂−4·v₃=0
4·v₁−12·v₂−4·v₃=0
−8·v₁−4·v₂−20·v₃=0
Dividing the first equation by −2, the second - by 4 and the third - by −4, all these equations yield a simpler system
3·v₁−4·v₂+2·v₃=0
v₁−3·v₂−v₃=0
2·v₁+v₂+5·v₃=0

As expected, the determinant of the coefficients of this system is zero and the equations are linearly dependent (the first equation multiplied by 7 equals the first multiplied by 11 plus the third multiplied by 5), as it should be, because otherwise the only solution would be a null-vector.
So, we drop the first equation and resolve the remaining two equations for v₂ and v₃ in terms of v₁.
Here is what remains
v₁−3·v₂−v₃=0
2·v₁+v₂+5·v₃=0

Find v₃ from the first equation
v₃ = v₁−3·v₂
Substituting v₃ into the second equation:
2·v₁+v₂+5·(v₁−3·v₂)=0
or
7·v₁−14·v₂=0
or
v₁−2·v₂=0

Therefore,
v₂=(1/2)·v₁ and v₃ = −(1/2)·v₁

Regardless of the value of v₁, vector ||v₁,(1/2)·v₁,−(1/2)·v₁|| is an eigenvector.
Set v₁=2 for simplicity, and vector ||2,1,−1|| should be an eigenvector.

Let's check it out.

4 8 −4 4 −2 −4 −8 −4 −10

·

2 1 −1

=

=

8+8+4 8−2+4 −16−4+10

=

20 10 −10

=

=

10·

2 1 −1

Which confirms that vector ||2,1,−1|| (and any collinear to it) is an eigenvector with 10 as its eigenvalue.

Answer A

Matrix

4 8 −4 4 −2 −4 −8 −4 −10

has three eigenvalues:
−12, −6 and 10.
Their corresponding eigenvectors are:
||1,6,16||, ||2,−3,−1|| and ||2,1,−1||.
Of cause, any vector collinear to a particular eigenvector would also be an eigenvector with the same eigenvalue.

Matrices+ 02 - Eigenvalues: UNIZOR.COM - Math+ & Problems - Matrices

Notes to a video lecture on http://www.unizor.com

Matrices+ 02
Matrix Eigenvalues

The concepts addressed in this lecture for two-dimensional real case are as well applicable to N-dimensional spaces and even to real or complex abstract vector spaces with linear transformations defined there.
Presentation in a two-dimensional real space is chosen for its relative simplicity and easy exemplification.

Let's consider a 2⨯2 matrix A as a linear operator in the two-dimensional Euclidean vector space. In other words, multiplication of any vector v on a coordinate plane by this 2⨯2 matrix A linearly transforms it into another vector on the plane w=A·v.
Assume, matrix A is

5	6
6	10

Let's see how this linear operator works, if applied to different vectors.

We will use a row-vector notation in the text for compactness, but column-vector notation in the transformation examples below.
The coordinates of our vectors we will enclose into double bars, like matrices, because a row-vector is a matrix with only one row, and a column-vector is a matrix with only one column.

Our first example of a vector to apply this linear transformation is v=||1,1||.

5 6 6 10

·

1 1

=

11 16

Obviously, the resulting vector w=||11,16|| and the original one v=||1,1|| are not collinear.

Applied to a different vector v=||3,−2||, we obtain somewhat unexpected result

5 6 6 10

·

3 −2

=

3 −2

Interestingly, the resulting vector w=||3,−2|| and the original one are the same. So, this operator leaves this particular vector in place. In other words, it retains the direction of this vector and multiplies its magnitude by a factor of 1.

Finally, let's applied our operator to a vector v=||2,3||.

5 6 6 10

·

2 3

=

28 42

Notice, the resulting vector w=||28,42|| is the original one v=||2,3|| multiplied by 14. So, this operator transforms this particular vector to a collinear one, just longer in magnitude by a factor of 14.

As we see, for this particular matrix we found two vectors that, if transformed by this matrix as by a linear operator, retain their direction, while change the magnitude by some factor.
These vectors are called eigenvectors. For each eigenvector there is a factor that characterizes the change in its magnitude if this matrix acts on it as an operator. This factor is called eigenvalue. This eigenvalue in the example above was 1 for v=||3,−2||, and 14 for v=||2,3||.

There are some questions one might ask.
1. Are there always some particular vectors that retain the direction if transformed by some particular matrix?
2. If yes, how to find them and how to find the corresponding multiplication factors?
3. How many such vectors exist, if any?
4. How to find all the multiplication factors for a particular matrix transformation?

Let's analyze the linear transformation by a matrix that leaves the direction of a vector without change, just changes the magnitude by some factor λ.

Assume, we have a matrix A=||a_i,j||, where i,j∈{1,2}, in our two-dimensional Euclidean space.
This matrix converts any vector v=||v₁,v₂|| into some other vector, but we are looking for such vector v that is converted by this matrix into a collinear one.
If matrix A transforms vector v to a collinear one with the magnitude of the original one multiplied by a factor λ, the following matrix equation must hold
A·v = λ·v
or in coordinate form

a1,1 a1,2 a2,1 a2,2

·

v1 v2

=

λ

·

v1 v2

which is equivalent to
(a_1,1−λ)·v₁ + a_1,2·v₂ = 0
a_2,1·v₁ + (a_2,2−λ)·v₂ = 0

This is a system of two linear equations with three unknowns λ, v₁ and v₂.

One trivial solution would be v₁=0 and v₂=0, in which case λ can take any value.
This is not a case worthy of analyzing.

If the matrix of coefficients of this system has a non-zero determinant, this trivial solution would be the only one.
Therefore, if we are looking for a non-trivial solution, the matrix's determinant must be zero, which gives a specific condition on the value of λ.

Therefore, a necessary condition for existence of other than null-vector v is
(a_1,1−λ)·(a_2,2−λ) − a_1,2· a_2,1 = 0
or
λ² − (a_1,1+a_2,2)·λ +
+ a_1,1·a_2,2−a_1,2·a_2,1 = 0

Since we are looking for real values of λ, we have to examine a discriminant D of his quadratic equation.
D =
=(a_1,1+a_2,2)²−4·(a_1,1·a_2,2−a_1,2·a_2,1)
=(a_1,1−a_2,2)²+4·a_1,2·a_2,1
If D is negative, there are no real solutions for λ.
If D is zero, there is one real solutions for λ.
If D is positive, there are two real solutions for λ.

Consider now that we have determined λ and would like to find vectors transformed into collinear ones by matrix A with this exact factor of change in magnitude.

If some vector v=||v₁,v₂|| that is transformed into a collinear one with a factor λ exists, vector s·v, where s is any real non-zero number, would have exactly the same quality because of associativity and commutativity of multiplication by a scalar.
A·(s·v) = (A·s)·v = (s·A)·v =
= s·(A·v) = s·(λ·v)= λ·(s·v)

Therefore, we don't need to determine exact values v₁ and v₂, we just need to determine only the direction of vector v=||v₁,v₂||, and this direction is determined by the factor v₁/v₂ or v₂/v₁ (to cover all cases, when one of them might be zero).

If v₂≠0, the directions of a vector v and that of vector ||v₁/v₂,1|| are the same.
If v₁≠0, the directions of a vector v and that of vector ||1,v₂/v₁|| are the same.

From this follows that, firstly, we can search for eigenvectors among those with v₂≠0, restricting our search to vectors ||x=v₁/v₂,1||.
Then we can search for eigenvectors among those with v₁≠0, restricting our search to vectors ||1,x=v₁/v₂||.
In both cases we will have to solve a system of two linear equations with two unknowns λ and x.

Searching for vectors ||x,1||
In this case the matrix equation that might deliver the required vector looks like this

a1,1 a1,2 a2,1 a2,2

·

x 1

=

λ

·

x 1

Performing the matrix by vector multiplication on the left side and scalar by vector on the right side and equating each component, we obtain a system of two equations with two unknowns - λ and x:
a_1,1·x+a_1,2·1 = λ·x
a_2,1·x+a_2,2·1 = λ·1

Take the right side of the second equation λ and substitute into the right side of the first equation, obtaining a quadratic equation for x:
a_1,1·x+a_1,2 = (a_2,1·x+a_2,2)·x
or
a_2,1·x² + (a_2,2−a_1,1)·x − a_1,2 = 0
Two solutions for this equations x_1,2, assuming they are real values, produce two vectors ||x₁,1|| and ||x₂,1||, each of which satisfy the condition of collinearity after the matrix transformation.
Generally speaking, the factor λ will be different for each such vector.

Searching for vectors ||1,x||
In this case the matrix equation that might deliver the required vector looks like this

a1,1 a1,2 a2,1 a2,2

·

1 x

=

λ

·

1 x

Performing the matrix by vector multiplication on the left side and scalar by vector on the right side and equating each component, we obtain a system of two equations with two unknowns - λ and x:
a_1,1·1+a_1,2·x = λ·1
a_2,1·1+a_2,2·x = λ·x

Take the right side of the first equation λ and substitute into the right side of the second equation, obtaining a quadratic equation for x:
a_2,1·1+a_2,2·x = (a_1,1·1+a_1,2·x)·x
or
a_1,2·x² + (a_1,1−a_2,2)·x − a_2,1 = 0
Two solutions for this equations x_1,2, assuming they are real values, produce two vectors ||1,x₁|| and ||1,x₂||, each of which satisfy the condition of collinearity after the matrix transformation.
Generally speaking, the factor λ will be different for each such vector.

Once again, let's emphasize important definitions.
Vectors transformed into collinear ones by a matrix of transformation are called eigenvectors or characteristic vectors for this matrix.
The factor λ corresponding to some eigenvector is called eigenvalue or characteristic value of the matrix and this eigenvector.

Let's determine eigenvectors and eigenvalues for a matrix A

5	6
6	10

used as an example above.

The quadratic equation to determine the multiplier λ for this matrix is
λ² − (a_1,1+a_2,2)·λ +
+ a_1,1·a_2,2−a_1,2·a_2,1 = 0
which amounts to
λ² − 15λ + 14 = 0
with solutions
λ₁ = 1 and λ₂ = 14

Let's find the eigenvectors of this matrix.
The quadratic equation for eigenvectors of type ||x,1|| is
6x² + (10−5)x − 6 = 0 or
6x² + 5x − 6 = 0 or
Solutions are
x_1,2 = (1/12)·(−5±√25+4·36) =
= (1/12)·(−5±13)
Therefore,
x₁ = 2/3
x₂ = −3/2
Two eigenvectors are:
v₁ = ||2/3,1|| which is collinear to vector ||2,3|| used in the example above and
v₂ = ||−3/2,1|| which is collinear to vector ||3,−2|| used in the example above.

The matrix transformation of these eigenvectors are

5 6 6 10

·

2/3 1

=

28/3 14

But the resulting vector ||28/3,14|| equals to 14·||2/3,1||, which means that eigenvector ||2/3,1|| has eigenvalue 14.

5 6 6 10

·

−3/2 1

=

−3/2 1

But the resulting vector ||−3/2,1|| equals to eigenvector ||−3/2,1||, which means that eigenvector ||−3/2,1|| has eigenvalue 1.

Not surprisingly, both eigenvectors found above have eigenvalues already found (1 and 14).

The quadratic equation for eigenvectors of type ||1,x|| is
6x² + (5−10)x − 6 = 0 or
6x² − 5x − 6 = 0 or
Solutions are
x_1,2 = (1/12)·(5±√25+4·36) =
= (1/12)·(5±13)
Therefore,
x₁ = 3/2
x₂ = −2/3
Two eigenvectors are:
v₁ = ||1,3/2|| which is collinear to vector ||2,3|| used in the example above and
v₂ = ||1,−2/3|| which is collinear to vector ||3,−2|| used in the example above.
So, we did not gain any new eigenvalues by searching for vectors of a form ||1,x||.

The above calculations showed that for a given matrix we have two eigenvectors, each with its own eigenvalue.

Based on these calculations, we can now answer the questions presented before.

Q1. Are there always some particular vectors that retain the direction if transformed by some particular matrix?
A1. Not always, but only if the quadratic equations for x
a_2,1·x² + (a_2,2−a_1,1)·x − a_1,2 = 0
and
a_1,2·x² + (a_1,1−a_2,2)·x − a_2,1 = 0
where ||a_i,j|| (i,j∈{1,2}) is a matrix of transformation, have real solutions.

Q2. If yes, how to find them?
A2. Solve the quadratic equations above and, for each real solutions x of the first equation, vector ||x,1|| is an eigenvector and, for each real solutions x of the second equation, vector ||1,x|| is an eigenvector. Then apply the matrix of transformation to each eigenvector ||x,1|| or ||1,x|| and compare the result with this vector. It should be equal to some eigenvalue λ multiplied by this eigenvector.

Q3. How many such vectors exist, if any?
A3. As many as real solutions have quadratic equations above, but no more than two.
Incidentally, in three-dimensional case our equations will be polynomial of the 3rd degree, and the number of solutions will be restricted to three.
In N-dimensional case this maximum number will be N.

Q4. How to find all the multiplication factors for a particular matrix transformation?
A4. Quadratic equation for eigenvalues
λ² − (a_1,1+a_2,2)·λ +
+ a_1,1·a_2,2−a_1,2·a_2,1 = 0
can have 0, 1 or 2 real solutions.

The concept of eigenvectors and eigenvalues (characteristic vectors and characteristic values) can be extended to N-dimensional Euclidean vector spaces and even to abstract vector spaces, like, for example, a set of all real functions integrable on a segment ||0,1||.
The detail analysis of these cases is, however, beyond the current course, which aimed, primarily, to introduce advance concepts.


Problem A
Research conditions when a diagonal matrix (only elements along the main diagonal are not zero) has eigenvalues.

Solution A
Matrix of transformation A=||a_i,j|| has zeros for i≠j.
So, it looks like this

a1,1	0
0	a2,2

The equation for eigenvalues in this (a_1,2=a_2,1=0) case is
λ² − (a_1,1+a_2,2)·λ + a_1,1·a_2,2 = 0
with immediately obvious solutions
λ₁=a_1,1 and λ₂=a_2,2
So, the values along the main diagonal of a diagonal matrix are the eigenvalues of this matrix.

Determine the eigenvectors now among vectors ||x,1||.
Original quadratic equation for this case is
a_2,1·x² + (a_2,2−a_1,1)·x − a_1,2 = 0

With a_2,1=a_1,2=0 it looks simpler:
(a_2,2−a_1,1)·x = 0
From this we conclude that, if a_2,2≠a_1,1, the only solution is x=0, so our eigenvector is ||0,1||.
The eigenvalue for this eigenvector is a_2,2.
If a_2,2=a_1,1, any x is good enough, so any vector is an eigenvector.

Determine the eigenvectors now among vectors ||1,x||.
Original quadratic equation for this case is
a_1,2·x² + (a_1,1−a_2,2)·x − a_2,1 = 0

With a_2,1=a_1,2=0 it looks simpler:
(a_1,1−a_2,2)·x = 0
From this we conclude that, if a_2,2≠a_1,1, the only solution is x=0, so our eigenvector is ||1,0||.
The eigenvalue for this eigenvector is a_1,1.
If a_2,2=a_1,1, any x is good enough, so any vector is an eigenvector.

Answer A
If matrix of transformation is diagonal

a1,1	0
0	a2,2

and a_2,2≠a_1,1,
the two eigenvectors are base unit vectors and the eigenvalues are a_1,1 for base unit vector ||1,0|| and a_2,2 for base unit vector ||0,1||.
In the case of a_1,1=a_2,2 any vector is an eigenvector with eigenvalue a_1,1.


Problem B
Prove that symmetrical matrix always has real eigenvectors.

Solution B

Matrix of transformation A=||a_i,j|| is symmetrical, which means a_1,2=a_2,1.

Recall that a necessary condition for existence of real eigenvalues λ is
(a_1,1−λ)·(a_2,2−λ) − a_1,2· a_2,1 = 0
or
λ² − (a_1,1+a_2,2)·λ +
+ a_1,1·a_2,2−a_1,2·a_2,1 = 0

Since we are looking for real values of λ, we have to examine a discriminant D of his quadratic equation.
D = (a_1,1−a_2,2)²+4·a_1,2·a_2,1
Since a_1,2=a_2,1, their product is non-negative, which makes the whole discriminant non-negative.
If D is zero, there is one real solutions for λ.
If D is positive, there are two real solutions for λ.
So, one or two solutions always exist.

Matrices+ 01 - Matrix as Operator: UNIZOR.COM - Math+ & Problems - Matrices

Notes to a video lecture on http://www.unizor.com

Matrices+ 01
Matrix as Operator

In the beginning matrices were just tables of real numbers with m rows and n columns, we called them m⨯n matrices.
We knew how to add two matrices with the same numbers of rows and columns, how to multiply a matrix by a scalar number and how to multiply an m⨯n matrix by an n⨯k matrix.

Here is a short recap.

Addition of matrices
Let A_m⨯n be an m⨯n matrix with elements a_i,j, where the first index signifies the row, where this element is located and the second index is a column.
Let B_m⨯n be another m⨯n matrix with elements b_i,j.
Then a new matrix C=A+B is an m⨯n matrix with elements c_i,j=a_i,j+b_i,j.

Multiplication of a matrix by a scalar
Let s be any real number (scalar) and A_m⨯n be an m⨯n matrix with elements a_i,j.
Multiplication of s by A_m⨯n produces a new matrix B_m⨯n with elements b_i,j=s·a_i,j.
Obviously, since the product of numbers is commutative and associative, the product of a matrix by a scalar is commutative, and the product of a matrix by two scalars is associative.

Product of two matrices
Let A_m⨯n be an m⨯n matrix with elements a_i,j.
Let B_n⨯k be an n⨯k matrix with elements b_i,j.
It's important for a definition of a product of matrices that the number of columns in the first matrix A_m⨯n equals to a number of rows in the second B_n⨯k (in our case this number is n.)
The product C_m⨯k of A_m⨯n and B_n⨯k is m⨯k matrix with elements c_p,q calculated as sums of products of n elements of p^th row of matrix A_m⨯n by n elements of q^th column of matrix B_n⨯k
∀p∈[1,m], ∀q∈[1,k]:
c_p,q = Σ_[1≤t≤n]a_p,t·b_t,q
It's important to note that, generally speaking, multiplication of two matrices IS NOT commutative, see Problem C below.
The product of three matrices, however, is associative, see Problem B below.

For our purposes we will only consider "square" matrices with the same number of rows and columns n and n-dimensional Euclidean vector space of sequences of n real numbers organized in a row (row-vector, which can be viewed as 1⨯n matrix) or in a column (column-vector, which can be viewed as n⨯1 matrix).

What happens if we multiply an n⨯n matrix by a column-vector, which we can consider as an n⨯1 matrix, according to the rules of multiplication of two matrices?
In theory, the multiplication is possible because the number of columns in the first matrix n equals to the number of rows in the second one (the same n) and, according to the rules of multiplication, the resulting matrix should have the number of rows of the first matrix n and the number of columns of the second one, that is 1. So, the result is an n⨯1 matrix, that is a column-vector.

As we see, multiplication of our n⨯n matrix by a column-vector with n rows results in another column-vector with n row.
In other word, multiplication by n⨯n matrix on the left transforms one column-vector into another, that is this multiplication represents an operation in n-dimensional vector space of column-vectors. The n⨯n matrix itself, therefore, acts as an operator in the vector space of column-vectors of n components.

Similar operations can be considered with row-vectors and their multiplication by matrices. The only difference is the order of this multiplication.
In a case of column-vectors we multiplied n⨯n matrix by n⨯1 column-vector getting another n⨯1 column-vector.
In case of row-vectors, we change the order and multiply a 1⨯n row-vector by n⨯n matrix on the right, getting another 1⨯n row-vector.
Therefore, an n⨯n matrix can be considered an operator in a vector space of row-vectors, if we apply the multiplication by matrix from the right.

Let's examine the properties of the transformation of n⨯1 column-vectors by multiplying them by n⨯n matrices.

Problem A
Prove that multiplication by an n⨯n matrix is a linear transformation in the n-dimensional Euclidean space ℝⁿ of n⨯1 column-vectors.
(a) ∀u∈ℝⁿ, ∀ K∈ℝ, ∀A_n⨯n:
A·(K·u) = K·(A·u) = (K·A)·u
(b) ∀u,v∈ℝⁿ, ∀A_n⨯n:
A·(u+v) = A·u + A·v
(c) ∀u∈ℝⁿ, ∀A_n⨯n,B_n⨯n:
(A+B)·u = A·u + B·u

Hint A
The proof is easily obtained straight from the definition of matrix multiplications.


Problem B
Prove that consecutive multiplication by two n⨯n matrices is associative in the n-dimensional Euclidean space ℝⁿ of n⨯1 column-vectors.
∀u∈ℝⁿ, ∀A_n⨯n,B_n⨯n:
A·(B·u) = (A·B)·u

Hint B
It's all about changing of an order of summation.
Let's demonstrate it for n=2.

Given two matrices A and B and a column-vector u.

Matrix A

a1,1	a1,2
a2,1	a2,2

Matrix B

b1,1	b1,2
b2,1	b2,2

Column-vector u

u1

u2

Let v=B·u and w=A·(B·u)=A·v
Then column-vector v's components are
v₁ = b_1,1·u₁ + b_1,2·u₂
v₂ = b_2,1·u₁ + b_2,2·u₂

The components of vector w are
w₁ = a_1,1·v₁ + a_1,2·v₂ =
= a_1,1·(b_1,1·u₁ + b_1,2·u₂) +
+ a_1,2·(b_2,1·u₁ + b_2,2·u₂) =
= a_1,1·b_1,1·u₁ + a_1,1·b_1,2·u₂ +
+ a_1,2·b_2,1·u₁ + a_1,2·b_2,2·u₂ =
= (a_1,1·b_1,1 + a_1,2·b_2,1)·u₁ +
+ (a_1,1·b_1,2 + a_1,2·b_2,2)·u₂

w₂ = a_2,1·v₁ + a_2,2·v₂ =
= a_2,1·(b_1,1·u₁ + b_1,2·u₂) +
+ a_2,2·(b_2,1·u₁ + b_2,2·u₂) =
= a_2,1·b_1,1·u₁ + a_2,1·b_1,2·u₂ +
+ a_2,2·b_2,1·u₁ + a_2,2·b_2,2·u₂ =
= (a_2,1·b_1,1 + a_2,2·b_2,1)·u₁ +
+ (a_2,1·b_1,2 + a_2,2·b_2,2)·u₂

Let's calculate the product of two matrices A·B
Matrix A·B:

a1,1·b1,1+a1,2·b2,1	a1,1·b1,2+a1,2·b2,2
a2,1·b1,1+a2,2·b2,1	a2,1·b1,2+a2,2·b2,2

Notice that coefficients at u₁ and u₂ in expression for w₁ above are the same as elements of the table (A·B) of the first row.
Analogously, coefficients at u₁ and u₂ in expression for w₂ above are the same as elements of the table (A·B) of the second row.
That means that, if we multiply matrix (A·B) by a column-vector u, we will get the same vector w as above.
That proves the associativity for a two-dimensional case.

Problem C
Prove that consecutive multiplication by two n⨯n matrices, generally speaking, IS NOT commutative.
That is, in general,
(A·B)·u) ≠ (B·A)·u

Proof C
To prove it, it is sufficient to present a particular case when a product of two matrices is not commutative.

Consider matrix A

1	2
3	4

Matrix B

−1	2
−3	4

Then matrix A·B will be

−7	10
−15	22

Reverse multiplication B·A will be

5	6
9	10

As you see, matrices A·B and B·A are completely different, and, obviously, their product with most vectors will produce different results.
Take a vector with components (1,0), for example. Matrix A·B will transform it into (−7,−15), while matrix B·A will transform it into (5,9).
End of Proof.

Vectors+ 11 - 2D Vector · Complex scalar: UNIZOR.COM - Math+ & Problems ...

Notes to a video lecture on http://www.unizor.com

Vectors+ 11
2D Vector · Complex scalar

We know that a two-dimensional vector multiplied by a real scalar changes its length proportionally, but the direction remains the same for a positive multiplier or changes to opposite for a negative multiplier.
Let's examine how vectors on a two-dimensional plane are affected if multiplied by a non-zero complex scalar.

Our first task is to define a multiplication operation of a vector by a complex number.
To accomplish this, let's recall that any complex number a+i·b, where both a and b are real numbers and i²=−1, can be represented by a vector in a two-dimensional Euclidean space (that is, on the coordinate plane) with abscissa a and ordinate b.
This representation establishes one-to-one correspondence between vectors on a plane and complex numbers.

Using this representation, let's define an operation of multiplication of a vector on a coordinate plane {a,b} by a complex number z=x+i·y as follows:
1. Find a complex number that corresponds to our vector. So, if a vector has coordinates {a,b}, consider a complex number a+i·b.
2. Multiply this complex number by a multiplier z=x+i·y using the rules of multiplication of complex numbers. This means
(a+i·b)·z = (a+i·b)·(x+i·y) =
= a·x+a·i·y+i·b·x+i·b·i·y =
= (a·x−b·y) + i·(a·y+b·x)
3. Find the vector that corresponds to a result of multiplication of two complex number in a previous step. This vector should have abscissa a·x−b·y and ordinate a·y+b·x.
4. Declare the vector in step 3 as a result of an operation of multiplication of the original vector {a,b} by a complex multiplier z=x+i·y. So, the result of multiplication of vector {a,b} by a complex multiplier z=x+i·y is a vector {a·x−b·y,a·y+b·x}

Now we have to examine the geometric aspect of this operation.
For this, let's represent a multiplier z=x+i·y as
z = √x²+y²·(x/√x²+y² + i·y/√x²+y²)

Two numbers, x/√x²+y² and y/√x²+y², are both in the range from −1 to 1 and a sum of their squares equals to 1.
Find an angle φ such that
cos(φ)=x/√x²+y² and
sin(φ)=y/√x²+y².
Now the multiplier looks like
z = |z|·[cos(φ) + i·sin(φ)]
where |z| = √x²+y²

Using this representation, the product of a vector {a,b} by multiplier z=x+i·y looks like
{a·x−b·y,a·y+b·x} = {a',b'}
where
a' = |z|·[a·cos(φ)−b·sin(φ)] and
b' = |z|·[a·sin(φ)+b·cos(φ)]

The geometric meaning of the transformation from vector {a,b} to vector {a',b'} is a rotation of the vector by angle φ.
Here is why.
The length of a vector {a,b} is L=√a²+b².
If the angle this vector makes with an X-axis is α, the abscissa and ordinate of our vector can be expressed as
a = L·cos(α)
b = L·sin(α)

Using this representation, let's express the coordinates of a vector {a',b'} obtained as a result of multiplication of the original vector {a,b} by a complex
z=x+i·y=|z|·[cos(φ) + i·sin(φ)]
in terms of L and α.

a'=|z|·[L·cos(α)·cos(φ)−L·sin(α)·sin(φ)]
and
b'=|z|·[L·cos(α)·sin(φ)+L·sin(α)·cos(φ)]

Recall from Trigonometry:
cos(α+φ)=cos(α)·cos(φ)−sin(α)·sin(φ)
sin(α+φ)=cos(α)·sin(φ)+sin(α)·cos(φ)

Now you see that
a'=|z|·L·cos(α+φ)
b'=|z|·L·sin(α+φ)

So, the multiplication of a vector {a=L·cos(α),b=L·sin(α)} by a complex number z=x+i·y=|z|·[cos(φ)+i·sin(φ)], can be interpreted as changing the length of an original vector by a factor |z| and a rotation by angle φ.

Vectors+ 10 Complex Hilbert Space: UNIZOR.COM - Math+ & Problems - Vectors

Notes to a video lecture on http://www.unizor.com

Vectors+ 10
Complex Hilbert Spaces

As we know, a Hilbert space consists of
(a) an abstract vector space V with operation of addition
∀v₁,v₂∈V: v₁+v₂∈V
(b) a scalar space S of numbers (we consider only two cases: S=ℝ - a set of all real numbers or S=ℂ - a set of all complex numbers) with operation of multiplication of a scalar from this set S by a vector from vector space V
∀λ∈S and ∀v∈V: λ·v∈V
(c) an operation of a scalar product of two vectors from V denoted as ⟨v₁,v₂⟩, resulting in a scalar from S
∀v₁,v₂∈V: ⟨v₁,v₂⟩∈S

Operations of addition of two vectors, resulting in a vector, multiplication of a vector by a scalar, resulting in a vector, and scalar product of two vectors, resulting in a scalar, must satisfy certain set of axioms presented in lecture Vectors 08 of this part of a course, including commutative, associative and distributive laws.

In all the previously considered cases our scalar space S was a set of all real numbers ℝ.
Now we will consider a set of all complex numbers ℂ as a scalar space S and examine the validity of our axioms as applied to a simple vector space.
By all accounts this should not present any problems since arithmetic operations with complex numbers are similar to those with real numbers.
There is, however, one complication.

One of the axioms of a scalar product was:
For any vector a from a vector space V, which is not a null-vector, its scalar product with itself is a positive real number
∀a∈V, a≠0: ⟨a,a⟩ > 0
This axiom is needed to introduce a concept of a length (or a magnitude, or a norm) of any vector as
||a|| = √⟨a,a⟩

Consider now a simple case of a vector space V - a set of all complex numbers ℂ with a scalar space S being a set of complex numbers ℂ as well.
We will use the word vector as an element of a vector space V and the word scalar as an element of a scalar space S, but both spaces are sets of all complex numbers ℂ.

Addition of two vectors (complex numbers) and multiplication of a vector (a complex number) by a scalar (also a complex number) are defined, as regular operations with complex numbers.
Scalar product of two vectors (two complex numbers) is defined as their regular operation of multiplication.
Let's check how valid our definition of a length of a vector is in this case with all the axioms previously introduced for a set of all real numbers ℝ being the scalar space S.

Vector a from V (a complex number) can be expressed as a₁+i·a₂, where a₁ and a₂ are also vectors from V, but belong to a subset of only real numbers, while i is an imaginary unit √−1, an element of a scalar space S (also a set of all complex numbers).

Let's calculate a scalar product of vector a with itself using the old laws of operations previously defined
⟨a₁+i·a₂,a₁+i·a₂⟩ =
[since a₁ and a₂ are real numbers]
= a₁²+2i·a₁·a₂+i²·a₂² =
= a₁²+2i·a₁·a₂−a₂²
The problem is, this is not a positive number since it contains an imaginary part.
So, the set of axioms we introduced before for a scalar space being a set of all real numbers ℝ is contradictory in case of complex numbers ℂ as a scalar space for a simple case above.

To overcome this problem, we have to revise our axioms in such a way that they will hold for examples like above for complex scalars, while being held as well for real numbers as scalars, since real numbers are a subset of complex ones.

There are two small changes in the axioms for a scalar product that we introduce to solve this problem.
Let a and b be two vectors and λ - a complex number as a scalar.
(1) λ·⟨a,b⟩ = ⟨λ·a,b⟩ = ⟨a,λ·b⟩
(2) ⟨a,b⟩ = ⟨b,a⟩
where horizontal bar above a complex number means complex conjugate number, that is a number x−i·y for a number x+i·y and a number x+i·y for a number x−i·y.
Incidentally, a conjugate to imaginary unit i (that is, 0+i·1) is −i (that is, 0−i·1).

First of all, if a complex number x+i·y is, actually, a real one (that is, if y=0), its conjugate number is the same as the original. So, in case of a scalar space being a set of all real numbers ℝ these axioms are exactly the same as the old previously accepted ones.

Secondly, for complex scalars these modified axioms solve the problem in a simple case of V=ℂ and S=ℂ mentioned before for a vector a=a₁+i·a₂, where a₁ and a₂ are from a subset of only real numbers.

Let's calculate a scalar product using the modified rules.
⟨a₁+i·a₂,a₁+i·a₂⟩ =
[using distributive law]
= ⟨a₁,a₁⟩ + ⟨a₁,i·a₂⟩ +
+ ⟨i·a₂,a₁⟩ + ⟨i·a₂,i·a₂⟩ =
= a₁² + ⟨a₁,i·a₂⟩ +
+ ⟨i·a₂,a₁⟩ + ⟨a₂·−i·i·a₂⟩ =
[since −i²=1 and addition is commutative]
= a₁² + a₂² + ⟨a₁,i·a₂⟩ + ⟨i·a₂,a₁⟩ =
[using a newly modified rule]
λ·⟨a·b⟩ = ⟨λ·a,b⟩ = ⟨a,λ·b⟩]
= a₁² + a₂² + ⟨−i·a₁,a₂⟩ + ⟨i·a₂,a₁⟩ =
= a₁² + a₂² + (−i)·⟨a₁,a₂⟩ + i·⟨a₂,a₁⟩ =
[since a₁ and a₂ are vectors from a subset of real numbers, their scalar product is commutative, and the last two terms cancel each other]
= a₁² + a₂²
which is a positive real number for any non-zero complex number.

This expression fully corresponds to a concept of an absolute value of a complex number a₁+i·a₂ and to a concept of a length of a vector in two-dimensional Euclidean space with abscissa a₁ and ordinate a₂ used as graphical representation of a complex number a₁+i·a₂.

Moreover, similar calculations can be applied to N-vector of complex numbers as a vector space with a set of all complex numbers as a scalar space.
Indeed, consider for brevity a case of N=2 since the more general case of any N is totally analogous.
The element of our two-dimensional vector space is a pair {a,b}, where both components are complex numbers.
Its scalar product with itself is defined as
⟨{a,b},{a,b}⟩ = (a·a) + (b·b)
and each term on the right of this equation, as we determined, is a positive real number.

As we see, a small modification of the axioms of scalar product in case of complex scalars solves the problem and, at the same time, does not change anything we knew about scalar product with real scalars.

As we did before, we can say that in case of a complex scalar space a scalar product of a non-null vector by itself is positive and the square root of it is its length or magnitude or norm
||a||² = ⟨a,a⟩

Problem A
Prove the Cauchy-Schwarz-Bunyakovsky inequality for an abstract vector space V and a complex scalar space S=ℂ
∀a,b∈V: |⟨a·b⟩|² ≤ ⟨a·a⟩·⟨b·b⟩
where absolute value of any complex number Z=X+i·Y is defined as
|Z|² = X² + Y² = Z·Z

Proof A
Consider any non-zero scalar (complex number) x and non-negative (by axiom) scalar product of vector a+xb by itself
0 ≤ ⟨a+x·b,a+x·b⟩
Use distributive properties to open all parenthesis
0 ≤ ⟨a,a⟩+⟨a,x·b⟩+⟨x·b,a⟩+⟨x·b,x·b⟩
Using the modified commutative rules introduced in this lecture, we can transform individual members of this expression as follows
⟨a,x·b⟩ = x·⟨a,b⟩
⟨x·b,a⟩ = x·⟨b,a⟩
⟨x·b,x·b⟩ = x·x·⟨b,b⟩

Now our inequality looks like
0 ≤ x·x·⟨b,b⟩+x·⟨a,b⟩+x·⟨b,a⟩+⟨a,a⟩

This inequality is true for any complex number x.
If vector b equals a null-vector, the inequality is held because ⟨b,b⟩=0 and ⟨a,b⟩=0 (see Problem A of lecture Vectors 08 of this part of a course).
Assume, vector b is not a null-vector, and let's see how our inequality looks for specific value of x:
x = −⟨a,b⟩/⟨b,b⟩

Let's evaluate each member of the inequality above.
x·x·⟨b,b⟩ = ⟨a,b⟩·⟨a,b⟩/⟨b,b⟩
x·⟨a,b⟩ = −⟨a,b⟩·⟨a,b⟩/⟨b,b⟩
x·⟨a·b⟩ = −⟨a,b⟩·⟨a,b⟩/⟨b,b⟩

Putting these values into our inequality and cancelling plus and minus of the same numbers, we obtain
0 ≤ −⟨a,b⟩·⟨a,b⟩/⟨b,b⟩ + ⟨a,a⟩
Multiplying by positive ⟨b,b⟩ and separating members into different sides of an inequality, we obtain
⟨a,b⟩·⟨a,b⟩ ≤ ⟨a,a⟩·⟨b,b⟩
or, since |Z|²=Z·Z for any complex Z,
|⟨a,b⟩|² ≤ ⟨a,a⟩·⟨b,b⟩
End of proof.