Linear Algebra - Distance,Hyperplanes and Halfspaces,Eigenvalues,Eigenvectors ( Continued 3 )
Summary
TLDRThis lecture concludes a series on linear algebra for data science, focusing on the relationships between eigenvectors and fundamental subspaces. The instructor explains the significance of symmetric matrices, highlighting that they always have real eigenvalues and linearly independent eigenvectors. These concepts are crucial in data science, particularly for covariance matrices and algorithms like principal component analysis (PCA). The lecture connects eigenvectors to null space and column space, providing foundational knowledge for further study in regression analysis and machine learning.
Takeaways
- 🧮 Symmetric matrices are frequently used in data science, especially in algorithms and covariance matrices.
- 🔢 The eigenvalues of symmetric matrices are always real, and the corresponding eigenvectors are also real.
- ♻️ For symmetric matrices, we are guaranteed to have n linearly independent eigenvectors, even if some eigenvalues are repeated.
- 🔗 Eigenvectors corresponding to zero eigenvalues are found in the null space of the matrix, while those corresponding to non-zero eigenvalues span the column space.
- 🚫 If a matrix is full rank (none of the eigenvalues are zero), there will be no vectors in the null space.
- 🧩 The eigenvectors of symmetric matrices that correspond to non-zero eigenvalues form a basis for the column space.
- 📐 The connection between eigenvectors, null space, and column space is important for data science algorithms like principal component analysis (PCA).
- 🔍 Eigenvectors of symmetric matrices are linear combinations of the matrix's columns.
- 📊 Symmetric matrices of the form A^T A or A A^T are frequently encountered in data science computations and always have non-negative eigenvalues.
- 📚 The lecture series covers essential linear algebra concepts for data science, laying the foundation for further topics in regression analysis and machine learning.
Q & A
What happens when a matrix is symmetric?
-When a matrix is symmetric, its eigenvalues are always real, and it guarantees that there are n linearly independent eigenvectors, even if eigenvalues are repeated.
Why are symmetric matrices important in data science?
-Symmetric matrices are important in data science because they frequently occur in computations, such as the covariance matrix, and they have useful properties like real eigenvalues and guaranteed linearly independent eigenvectors.
What is the significance of eigenvalues being real for symmetric matrices?
-For symmetric matrices, real eigenvalues imply that the corresponding eigenvectors are also real, making the matrix easier to work with in practical applications like data science and machine learning.
How are eigenvectors related to the null space when the eigenvalue is zero?
-Eigenvectors corresponding to eigenvalue zero are in the null space of the matrix. If an eigenvalue is zero, the corresponding eigenvector lies in the null space.
What is the connection between eigenvectors and the column space for symmetric matrices?
-For symmetric matrices, the eigenvectors corresponding to nonzero eigenvalues form a basis for the column space. This means that the column space can be described using these eigenvectors.
What role do repeated eigenvalues play in the context of eigenvectors?
-When eigenvalues are repeated, there may be fewer linearly independent eigenvectors for a general matrix. However, for symmetric matrices, even with repeated eigenvalues, there will still be n linearly independent eigenvectors.
How do a transpose A and A transpose matrices relate to symmetric matrices in data science?
-Both A transpose A and A A transpose are symmetric matrices, which frequently occur in data science computations, such as covariance matrices. Their symmetry guarantees real, non-negative eigenvalues and linearly independent eigenvectors.
What does it mean if a matrix has no eigenvalues equal to zero?
-If a matrix has no eigenvalues equal to zero, it is full rank, meaning there are no vectors in the null space. This implies that all eigenvectors are outside the null space.
How are eigenvectors computed for a symmetric matrix with repeated eigenvalues?
-For a symmetric matrix with repeated eigenvalues, the eigenvectors can still be computed to be linearly independent, ensuring that the matrix has the full set of n independent eigenvectors.
What is the importance of the relationship between eigenvalues, null space, and column space in linear algebra?
-The relationship between eigenvalues, null space, and column space is critical in linear algebra because it helps define the structure of a matrix. Eigenvectors corresponding to zero eigenvalues belong to the null space, while eigenvectors corresponding to nonzero eigenvalues define the column space. These concepts are foundational in data science and machine learning algorithms like PCA.
Outlines
📊 The Role of Eigenvalues and Eigenvectors in Data Science
This section discusses the importance of eigenvalues and eigenvectors in data science, specifically how they relate to linear algebra concepts. The eigenvalue equation (A - λI = 0) is explained, emphasizing how eigenvalues can be real or complex. However, for symmetric matrices, eigenvalues and their corresponding eigenvectors are always real. Symmetric matrices, like the covariance matrix, play a crucial role in data science. This section also notes that for matrices with distinct eigenvalues, there are guaranteed to be linearly independent eigenvectors, but this is not always true for repeated eigenvalues, unless the matrix is symmetric.
🔄 Symmetric Matrices and their Special Properties
This paragraph delves deeper into the properties of symmetric matrices, which have real and non-negative eigenvalues. These matrices appear frequently in data science computations, such as AᵀA or AAᵀ. Because these matrices are symmetric, they guarantee the existence of n linearly independent eigenvectors. The relationship between these eigenvectors and the matrix's column and null spaces is introduced, providing a foundation for understanding eigenvalues’ role in determining matrix rank and space interactions.
🧮 Eigenvalues, Null Spaces, and Full-Rank Matrices
Here, the focus shifts to the relationship between eigenvectors and the null space of a matrix. If an eigenvalue is zero, its corresponding eigenvector lies in the null space. Conversely, eigenvectors with non-zero eigenvalues cannot belong to the null space. This section explains how a matrix with no zero eigenvalues is full rank, implying no vectors exist in its null space. The connection between eigenvalues and the full rank of a matrix is a key point in this analysis.
📐 Column Space and Eigenvectors of Symmetric Matrices
This paragraph elaborates on the relationship between eigenvectors and the column space in symmetric matrices. If a matrix has r zero eigenvalues, the corresponding eigenvectors span the null space, while the remaining n-r eigenvectors span the column space. Through the rank-nullity theorem, the text explains that the rank of the matrix is n-r, meaning there are n-r independent column vectors. The eigenvectors corresponding to non-zero eigenvalues form a basis for the matrix's column space.
🔗 Linear Combinations of Eigenvectors and Column Space
In this section, the text explores how eigenvectors corresponding to non-zero eigenvalues are linear combinations of the matrix's columns. It shows that these eigenvectors form a basis for the column space, and each eigenvector can be expressed as a weighted combination of the matrix’s columns. The paragraph demonstrates how these eigenvectors span the column space and help reduce the complexity of the matrix representation by focusing on the independent vectors.
📝 Example of Eigenvalues and Eigenvectors in Action
An example of a 3x3 symmetric matrix is presented to illustrate the theoretical concepts discussed earlier. The example matrix has eigenvalues of 0, 1, and 2, and corresponding eigenvectors are calculated. The vector corresponding to the zero eigenvalue lies in the null space, while the others span the column space. The text verifies these relationships using matrix multiplication, demonstrating how the eigenvectors represent relationships between the matrix’s variables and span the column space.
📚 Conclusion: Key Takeaways on Symmetric Matrices
The final section summarizes the lecture series on linear algebra for data science. It revisits the critical points, such as the fact that symmetric matrices always have real eigenvalues and n linearly independent eigenvectors. The connection between eigenvectors and the null space and column space of a matrix is reiterated, particularly in symmetric matrices. The importance of these concepts in algorithms like principal component analysis (PCA) is highlighted, as well as their broader applications in data science. The lecture concludes by mentioning upcoming modules on statistics and machine learning.
Mindmap
Keywords
💡Eigenvalue
💡Eigenvector
💡Symmetric Matrix
💡Null Space
💡Column Space
💡Rank
💡Covariance Matrix
💡Full Rank Matrix
💡Principal Component Analysis (PCA)
💡Rank-Nullity Theorem
Highlights
Introduction to the connection between eigenvectors and fundamental subspaces in linear algebra for data science.
Eigenvalue-eigenvector equation: A - λI = 0, showcasing the polynomial nature and potential for real or complex eigenvalues in general matrices.
Symmetric matrices have special properties in data science, including real eigenvalues and guaranteed real eigenvectors.
Symmetric matrices are common in data science, e.g., covariance matrices, and ensure n linearly independent eigenvectors.
For symmetric matrices, repeated eigenvalues still guarantee independent eigenvectors, unlike general matrices.
The identity matrix serves as an example of a symmetric matrix with repeated eigenvalues but independent eigenvectors.
Matrices of the form AᵀA or AAᵀ, commonly encountered in data science, are always symmetric and have non-negative eigenvalues.
Eigenvectors corresponding to zero eigenvalues are in the null space of matrix A, establishing a key relationship.
When none of the eigenvalues are zero, the matrix is full rank, indicating no non-trivial solutions to Ax = 0.
For symmetric matrices, eigenvectors corresponding to non-zero eigenvalues span the column space of the matrix.
The rank-nullity theorem links the number of non-zero eigenvalues to the rank and nullity of the matrix.
Eigenvectors are linear combinations of the columns of matrix A, and for symmetric matrices, these eigenvectors form a basis for the column space.
In symmetric matrices, the connection between null space, column space, and eigenvectors forms the foundation of data science algorithms.
Practical example: A symmetric 3x3 matrix is used to compute eigenvalues and eigenvectors, verifying theoretical results.
Eigenvectors corresponding to zero eigenvalues can identify relationships between variables in data science problems.
Transcripts
[Music]
this is the last lecture in the series
of lectures on linear algebra for data
science and as I mentioned in the last
class today I'm going to talk to you
about the connections between
eigenvectors on the fundamental
subspaces that we have described earlier
we saw in the last lecture that the
eigenvalue eigenvector equation this
else in this equation having to be
satisfied which is a minus lambda I
equals 0 in general
we also saw that this would turn out to
be a polynomial of degree n in lambda
which basically means that even if this
matrix a is real because the solutions
to a polynomial equation could be either
real or complex you could have
eigenvalues that are complex so for a
general matrix
you could have eigenvalues which are
either real or complex and notice that
since we write the equation ax equals
lambda X whenever this eigen values
become complex then the eigen vectors
are also complex vectors so this is true
in general however if the matrix is
symmetric and symmetric matrices are of
the form equal to a transpose then there
are certain nice properties for these
matrices which are very useful for us in
data science we also encounter symmetric
matrices quite a bit in data science for
example the covariance matrix turns out
to be a symmetric matrix and there are
several other cases where we deal with
symmetric matrices so these properties
of symmetric matrices are very useful
for us when we look at algorithms in
data science now the first property of
symmetric matrices that is very useful
to us is if the matrix is symmetric then
the eigenvalues are always real so
irrespective of what the symmetric
matrix is this polynomial would
give real solutions for symmetric
matrices and as I mentioned before if
this turns out to be real then the
eigenvectors are also real now there is
another aspect of an eigen values and
eigenvectors that is important if I have
a matrix a and I have n different
eigenvalues lambda 1 to lambda and all
of them are distinct then I'll
definitely have n linearly independent
eigenvectors corresponding to them which
could be nu 1 nu 2 all the way up to nu
n however if there are certain
eigenvalues which are repeated so for
example if you take a case where I can
value lambda 1 is repeated then I could
have some polynomial which is like this
so the polynomial original polynomial
has eigen value lambda 1 repeated twice
and then there's another n minus tooth
order polynomial which will give you n
minus 2 other solutions now in this case
when I have lambda 1 repeated like this
then it could turn out that this eigen
value either has to I can victors which
are independent or it might have just
one eigenvector so finding n linearly
independent eigenvectors is not always
guaranteed for any general matrix and we
already know that eigenvectors could be
complex for any general matrix however
when we talk about symmetric matrices we
can say for sure that the eigenvalues
would be real the eigenvectors would be
real further we are always guaranteed
that we'll have n linearly independent
eigenvectors for symmetric matrices it
doesn't matter how many times the
eigenvalues get repeated one classic
example of a symmetric matrix where
eigen values are repeated many times so
take identity matrix something like this
here
this identity matrix has eigen value
lambda equal to one which is repeated
thrice but it would have three
independent eigen vectors 1 0 0 0 1 0
and 0 0 so this is a case where I can
values repeated tries but there are
three independent eigenvectors so this
is also an important result that we
should keep in mind and as I mentioned
in the last slide symmetric matrices
have a very important role in data
sciences in fact symmetric matrices of
the type a transpose a or EA a transpose
are often encountered in data sense
computations and notice that both of
these matrices are symmetric so for
example if I take a transpose a
transpose this will be a transpose a
transpose transpose which will be a
transpose a so the transpose of the
matrix is the same you can verify that a
a transpose is also symmetric through
the same idea so we know matrices of the
form a transpose a or a a a transpose
are both symmetric and they are often
encountered when we do computations in
data science and we know from the
previous slide I had mentioned that the
eigenvalues of symmetric matrices are
real
if the symmetric matrix also takes this
form or this form we can also say that
while the eigenvalues are real they are
also non-negative that is they can they
will be either 0 or positive but none of
the eigenvalues will be negative so this
is another important idea that we will
use when we do data science when we look
at covariance matrices and so on also
the fact that this a transpose a and a
transpose are symmetric matrices
guarantees that there will be n linearly
independent eigenvectors for matrices of
this form also so what we are going to
do right now is because of the
importance of symmetric matrices in data
sense computations we are going to look
at the connection between the
eigenvectors and the column space a null
space
for a symmetric matrix some of these
results translate to non symmetric
matrices also but for symmetric matrices
all of these are results that we can use
so we go back to the eigenvalue
eigenvector equation a new is lambda nu
and this result that we are going to
talk about right now it's true whether
the matrix a is symmetric or not
if a nu equals lambda nu we ask the
question what happens when lambda is
zero that is one of the eigenvalues
becomes zero so when one of the
eigenvalues becomes zero then we have
this equation which is a nu equals zero
so we can interpret new as an
eigenvector corresponding to eigenvalue
zero we have also seen this equation
before when we talked about different
subspaces for matrices
we saw that null space vectors are of
the form a beta is zero from one of our
initial lectures you notice that this
and this form are the same so that
basically means that nu which is an
eigenvector corresponding like
corresponding to eigenvalue lambda
equals 0 is a null space vector because
it is just of the form that we have here
so we could say the eigenvectors
corresponding to 0 eigen values are in
the null space of the original matrix a
conversely if the eigenvalue
corresponding to an eigen vector is not
0 then that eigenvector cannot be in the
null space of a so these are important
results that we need to know so this is
how heigen vectors are connected to null
space if none of the eigen values are 0
that basically means that the matrix a
is full rank and that means that I can
never solve
a new equal to 0 and get non-trivial new
so it's not possible if a is full right
so if a is full blank I cannot solve for
this and get non-trivial new so whenever
lambda is lambda such that there are
there is no eigen value that is 0 that
means a is full rank matrix that means
there is no eigen vector such that a new
is 0 which basically means that there
are no vectors in the null space now
let's see the connection between
eigenvectors and column space in this
case i'm going to show you the result
and this result is valid for symmetric
matrices let us assume that i have a
symmetric matrix a and the symmetric
matrix a we know will have n real
eigenvalues let's assume that all of
these eigen values are 0 so this are
could be 0 also that means there is no
eigenvalue which is 0 so even then all
of this discussion is valid but as a
general case let us assume that our i
gain values of 0 so there are our 0
eigen values and since we are assuming
this matrix is n by n there will be n
real eigen values of which are are 0 so
there will be n minus R nonzero eigen
values and from the previous slide we
know that the our eigen vectors
corresponding to this are 0 eigen values
are all in the null space okay so since
I have all 0 eigen values I will have
our eigen vectors corresponding to this
so all of these are eigen vectors are in
the null space which basically means
that the dimension of the null space is
are because there are our vectors in the
null space and from rank nullity theorem
we know that rank plus nullity is equal
to number of columns in this case n
since there are our eigen vectors in the
null space nullities are so the rank of
the matrix has to be equal to n minus r
so that is what we are saying here and
further we know that column rank is
equal to row rank and since the rank of
the matrix is n minus R the column rank
also has to be n minus R this basically
means that there are n minus R
independent vectors in the columns of
the matrix so one question that we might
ask is the following we could ask what
could be a basis set for this column
space or what could be the n minus r
independent vectors that we can use as
the columns subspace so there are a few
things that we can notice based on what
we have discussed till now first notice
that the n minus r eigenvectors that we
talked about in the last slide the ones
that are not i ghen vectors
corresponding to lambda equal to 0 they
cannot be in the null space because
lambda is a number which is different
from 0 so these n minus r eigenvectors
cannot be in the null space of the
matrix a so let me write again we are
discussing all of this for symmetric
matrices we know that all of this n
minus I are Reagan vectors are also
independent because we said irrespective
of what the symmetric matrix is we will
always get n linearly independent eigen
vectors so that means these n minus r
eigenvectors are also independent we
also know that each of these independent
eigenvectors are going to be linear
combinations of columns of a to see this
let us look at this equation so let me
write this out so I could call this as a
I am going to expand this nu into
new one new - all the way up to new
again notice that these are components
of new we are just taking one
eigenvector new and then these are the
end components in that eigenvector I can
write this as lambda mu and from the
previous lecture of how to do this
column multiplication and how to
interpret this column multiplication I
said you could think of this as nu 1
times the first column of a + nu 2 times
the second column of a all the way up to
Nu n types nth column of a equal to
lambda nu now in this equation let me be
very clear
these are scalars which are components
in the eigenvector nu these are column
vectors is the first column of a second
column of a this is n column of a this
is again a scalar lambda which is the
eigen value corresponding to Nu so this
could be true for any of the N minus R
eigen vectors which are not in the null
space of this matrix a now take lambda
to the other side so you will have this
equation as nu is nu 1 by lambda a 1 and
so on plus nu n by lambda a again new
one is a scalar lambda is a scalar so
these are all constants that we are
using to multiply these columns now you
will clearly see that each of this eigen
vectors n minus R eigen vectors are
linear combinations of the columns of a
so there are n minus R linear in
linearly independent eigen vectors like
this and each of this are combinations
of columns of a and we also know that
the dimension of the column space is n
minus R in other words if you take all
of this columns a 1 to a n these can be
represented using just n minus R
linearly independent vectors now when we
put all of these facts together which is
the N minus R eigen vectors are linearly
independent
there are combinations of columns of a
and the number of independent columns of
a can be only n minus R so this implies
that the eigenvectors corresponding to
the nonzero eigenvalues for a symmetric
matrix form a basis for the column space
so this is the important result that I
wanted to show you with all of these
ideas now again these results we will
see and use as we look at some of the
data finds algorithms later so let's
take a simple example to understand how
all of this works let's consider a
matrix which is of this form here it's a
three by three matrix first thing that I
want you to notice that this is a
symmetric matrix so if you do a
transpose equals saying
and we said symmetric matrices will
always have real eigenvalues and when
you do the eigen value computation for
this the way you do the eigen value
computation is you take determinant a
minus lambda I equal to zero then you're
going to get a third order polynomial
you set it equal to zero and then you
calculate the three solutions to this
polynomial and these would turn out to
be the solution 0 1 2 and you take each
of these solutions and then substitute
it back in and then solve for ax equals
lambda X then you would get the three
eigen vectors corresponding to this
which are given by this this and this
notice from our discussion before since
this is an eigen vector corresponding to
lambda equal to 0 so this is going to be
in the null space of this matrix a and
these are the remaining two how do I get
this 2 which is 3 minus 1 n n by n so
it's a 3 by 3 matrix and nullity is 1
because there is only one eigen vector
corresponding to lambda equal to 0 so I
get two other linearly independent
vectors and in the last slide when we
were discussing the connections
claim that these two eigenvectors will
be in the column space or in other words
what we are claiming is that these three
columns can simply be written as a
linear combination of these two columns
and we are also sure that when we do a
times new one this will go to zero so
let us verify all of this in the next
slide
so let's first check eight times new one
so this is a matrix I have eight times
new one here and you can quite easily
see that when you do this computation
we'll get this zero zero zero which
basically shows that this is the eigen
vector corresponding to zero eigen value
interestingly in our initial lectures we
talked about null space and then we said
the null space vector identifies a
relationship between variables now since
this eigenvector is in the null space
the eigen vector corresponding to the
eigen vector corresponding to zero eigen
value or eigen vectors corresponding to
zero eigen values identify the
relationships between the variables
because these eigen vectors are in the
null space of the matrix so it's an
interesting connection that we can make
so the eigenvectors corresponding to
zero eigenvalue can be used to identify
relationships among variables now let's
do the last thing that we discuss let us
now check if the other two eigen vectors
shown below so this is for the other two
eigen values span the column space so
what I've done here is I've taken each
one of these columns from matrix a so
this is column one so this is a 1 this
is a 2 and this is a 3 column 1 is 6
times nu 2 column 2 is 8 times nu 2 and
column 3 is 2 times nu 3 so we can say
that a 1 a 1 a 2 and a 3
linear combinations of new to and new T
so new to a new tree form a basis for
this column space of matrix a so to
summarize we have ax equal to lambda X
and we largely focused on symmetric
matrices in this lecture so we saw that
if we have symmetric matrices they have
real eigenvalues we also saw that
symmetric matrices have n linearly
independent eigenvectors we saw the
daigon vectors corresponding to zero
eigenvalues span the null space of the
matrix a and eigenvectors corresponding
to nonzero eigenvalues span the column
space of a for symmetric matrices that
we described in this lecture so with
this we have described most of the
important fundamental ideas from linear
algebra that we will use quite a bit in
the material that follows the linear
algebra parts will be used in regression
analysis which you will see as part of
this course and many of these ideas are
also useful in algorithms that do
classification for example we talked
about how spaces and so on and the
notion of eigenvalues and eigenvectors
are used pretty much in almost every
data science algorithm of particular
node is one algorithm which is called
the principal component analysis where
these ideas of connections between null
space column space and so on are used
quite heavily so I hope that we have
given you reasonable understanding of
some of the important concepts that you
need to learn to understand some of the
material that we are going to teach in
this course and as I mentioned before
linear algebra is a vast topic there are
several ideas how do these ideas
translate which ones of these are
applicable are not applicable for non
symmetric matrices and so on and from
the previous lectures how do we develop
some of those concepts more can be found
in many good linear algebra books
however our aim here has been to really
call out the most important concepts
that we are going to use again and again
in this first course on data science for
engineers more advanced topics in linear
algebra will be covered when we teach
the next course on machine learning
where those concepts might be more
useful in advanced machine learning
techniques that we will pitch so with
this we close the series of lectures on
linear algebra and the next set of
lectures would be on the use of
statistics in data science I thank you
and I hope to see you back after you go
through the module on statistics which
will be taught by my colleague
precession connor seaman thank you
[Music]
[Music]
関連動画をさらに表示
1 Principal Component Analysis | PCA | Dimensionality Reduction in Machine Learning by Mahesh Huddar
Eigenvectors and eigenvalues | Chapter 14, Essence of linear algebra
Week 3 Lecture 13 Principal Components Regression
Dear linear algebra students, This is what matrices (and matrix manipulation) really look like
Eigen values and Eigen Vectors in Tamil | Unit 1 | Matrices | Matrices and Calculus | MA3151
Lecture 05 : Basis and Dimension
5.0 / 5 (0 votes)