Lecture 05 : Basis and Dimension

Summary

This lecture delves into the fundamental concepts of basis and dimension in vector spaces, crucial for machine learning algorithms and applications like dimension reduction and dictionary learning. It explains the criteria for a set of vectors to form a basis, emphasizing linear independence and the ability to span the space. The lecture also distinguishes between finite and infinite dimensional spaces, provides examples of bases in various spaces, and introduces the concept of dimension as the number of vectors in a basis. Important results related to basis and dimension, including the implications for linear dependence and the extension of linearly independent sets, are highlighted.

Takeaways

• 📚 The lecture discusses the concept of basis and dimension in vector spaces, which are fundamental to machine learning algorithms.
• 🔍 A basis for a vector space is a set of linearly independent vectors that span the entire space, allowing any vector to be expressed as a linear combination of these basis vectors.
• 📏 There are two types of vector spaces: finite-dimensional and infinite-dimensional, differentiated by the number of vectors in their basis.
• 🧩 The dimension of a vector space is the number of vectors in its basis, which is a key characteristic of the space.
• 📉 In machine learning, basis and dimension concepts are crucial for algorithms such as dimension reduction and dictionary learning.
• 🌐 The standard basis for R2, R3, and RN are sets of vectors with elements that are zero except for a single one, representing the standard orientation in space.
• 🔢 The dimension of spaces like R2, RN, and the space of all 2x2 real matrices are given by simple formulas related to the number of elements in the basis.
• 🔑 If a set of vectors contains more than the dimension of the vector space, it is linearly dependent, emphasizing the importance of the basis size.
• 📈 Any linearly independent subset of a vector space can be extended to form a basis for that space, highlighting the flexibility in choosing a basis.
• 📊 For finite-dimensional vector spaces, all bases contain the same number of vectors, which equals the space's dimension.
• 📘 The lecture provides examples of finding the basis and dimension of subspaces in R3 and R4, illustrating the process of determining these properties.

Q & A

What is the main topic of the lecture?

The main topic of the lecture is the concept of basis and dimension in vector spaces, which are important in the context of machine learning algorithms.

Why are basis and dimension important in machine learning?

Basis and dimension are important in machine learning because they help in representing input data as a linear combination of certain vectors, which is essential for algorithms like dimension reduction and dictionary learning.

What is the definition of a basis in vector spaces?

A basis for a vector space V is a set of linearly independent vectors in V that spans the vector space V. This means every vector in the space can be written as a linear combination of the basis vectors.

What are the two main properties a set of vectors must have to be considered a basis?

A set of vectors must be linearly independent and must span the entire vector space to be considered a basis.

What is the difference between finite and infinite dimensional vector spaces?

A finite dimensional vector space has a basis containing a finite number of vectors, while an infinite dimensional vector space has a basis with an infinite number of vectors.

Can you provide an example of a basis for R2 over the real numbers?

An example of a basis for R2 over the real numbers is the set of vectors {(1, 0), (0, 1)}.

What is the dimension of a vector space?

The dimension of a vector space is the number of elements in a basis of the vector space.

What is the relationship between the dimension of a vector space and the number of vectors in its basis?

The dimension of a vector space is equal to the number of vectors in any of its bases.

How can you determine if a set of vectors is linearly dependent?

A set of vectors is linearly dependent if it contains more vectors than the dimension of the vector space it is supposed to span.

Can a linearly independent set of vectors be extended to form a basis of a vector space?

Yes, if a linearly independent set of vectors does not span the entire vector space, it can be extended by adding more linearly independent vectors until it becomes a basis.

What is the significance of the dimension of the intersection of two subspaces in relation to the dimensions of the subspaces themselves?

The dimension of the intersection of two subspaces, along with the dimensions of the subspaces themselves, follows the formula: dimension of S1 + dimension of S2 = dimension of (S1 + S2) + dimension of (S1 ∩ S2).

Outlines

00:00

📚 Introduction to Basis and Dimension in Vector Spaces

The script begins with a refresher on vector subspaces and introduces the concept of basis and dimension of these spaces, crucial for machine learning algorithms. It explains that feature vectors can be represented as linear combinations of certain vectors, which may be linearly independent or orthogonal. The lecture aims to identify the specific set of vectors that form a basis for the feature space, which is essential in algorithms like dimension reduction and dictionary learning. The mathematical definition of a basis is presented, emphasizing that it consists of linearly independent vectors that span the vector space. The script also distinguishes between finite and infinite dimensional vector spaces.

05:02

🔍 Examples of Vector Space Bases and Their Properties

This paragraph provides examples of vector space bases, such as the standard basis for R2, R3, and RN, which are sets of vectors with elements of one and zeros elsewhere. It further explores the basis of spaces of 2x2 matrices with real entries and 2x2 symmetric matrices, highlighting the difference in the number of basis vectors required for each. The dimension of a vector space is defined as the number of elements in its basis, with examples given for R2, RN, and spaces of matrices and polynomials, emphasizing that the dimension is a key characteristic of the space.

10:04

📉 Important Results on Basis and Dimension

The script discusses important results related to bases and dimensions of vector spaces. It states that if a set of vectors exceeds the dimension of the space, it must be linearly dependent. It also mentions that any linearly independent subset can be extended to form a basis of the vector space. Additionally, it asserts that all bases of a finite-dimensional vector space contain the same number of vectors, which equals the space's dimension, and introduces the concept of the sum and intersection of subspaces and their dimensions.

15:07

📚 Finding the Basis and Dimension of Subspaces

The paragraph demonstrates how to find the basis and dimension of subspaces using the example of a subspace S of R3, defined by a linear equation. It shows the process of simplifying the equation to express the components of vectors in S in terms of a smaller set of variables, leading to the identification of the basis vectors and the calculation of the subspace's dimension. The example illustrates the method of determining the basis and dimension for subspaces within a larger vector space.

20:12

🔍 Further Examples of Subspace Analysis

This section continues with more examples of finding the basis and dimension of subspaces within R3 and R4. It explains how to determine the basis for a subspace defined by a set of equal components and how to find the intersection of two subspaces. The script uses these examples to show that the dimension of the intersection of two subspaces can be zero, indicating that their only common element is the zero vector, and that the sum of dimensions of two subspaces equals the dimension of their union plus the dimension of their intersection.

25:14

🚀 Conclusion and Preview of Upcoming Topics

The script concludes with a summary of the concepts of basis and dimension learned in the lecture and previews the next topic, which will be linear transformations, another fundamental concept in mathematics and machine learning. It also provides references for further study and expresses gratitude for the audience's attention, ending with applause and music, signaling the end of the lecture.

Keywords

💡Vector Subspaces

Vector subspaces are subsets of a vector space that are themselves vector spaces, adhering to the rules of vector addition and scalar multiplication. In the context of the video, vector subspaces are foundational to understanding the concept of basis and dimension, as they allow for the exploration of subsets within a larger vector space. The script discusses how feature vectors in machine learning can be represented as linear combinations within these subspaces.

💡Basis

A basis in a vector space is a set of linearly independent vectors that span the entire space. It serves as the fundamental framework for expressing any vector within the space as a linear combination of the basis vectors. The video emphasizes the importance of basis in various machine learning algorithms, such as dimension reduction and dictionary learning, where it is crucial to identify a set of vectors that can represent the feature space efficiently.

💡Dimension

The dimension of a vector space is the number of vectors in its basis, representing the minimum number of components needed to describe any vector within that space. The script explains that the dimension is key to understanding the complexity of the space and its applications in machine learning, such as in wavelet-based algorithms where functions are approximated as linear combinations of orthonormal functions.

💡Linearly Independent

Linear independence is a property of vectors where no vector in the set can be written as a linear combination of the others. This concept is critical for defining a basis, as all vectors in the basis must be linearly independent. The script uses this term to describe the first qualification for a set of vectors to form a basis of a vector space.

💡Linear Combination

A linear combination is an expression constructed from a set of vectors by multiplying each vector by a scalar (a constant) and adding the results. In the script, the ability to express all feature vectors as linear combinations of a basis set is central to the discussion of how data can be represented in machine learning algorithms.

💡Finite Dimensional Vector Spaces

Finite dimensional vector spaces are those with a basis consisting of a finite number of vectors. The script contrasts these with infinite dimensional vector spaces and uses the concept to explain that the number of basis vectors equates to the dimension of the space, as seen in examples like R2 and R3.

💡Infinite Dimensional Vector Spaces

Infinite dimensional vector spaces have a basis with an infinite number of vectors, allowing for the representation of an endless variety of vectors. The script mentions these in the context of function approximation in wavelet-based algorithms, where an infinite set of orthonormal functions can be used to approximate any function.

💡Standard Basis

The standard basis of a vector space consists of vectors with a single element being one and all other elements being zero. For example, in R2, the standard basis vectors are [1, 0] and [0, 1]. The script uses the standard basis to illustrate simple examples of bases for R2, R3, and RN.

💡Orthonormal

Orthonormal vectors are orthogonal unit vectors that are often used in function approximation and signal processing. The script refers to orthonormal functions in the context of wavelet-based algorithms, where functions are approximated as linear combinations of such functions derived from a mother wavelet.

💡Linear Dependence

Linear dependence occurs in a set of vectors when at least one vector can be expressed as a linear combination of the others, indicating redundancy in the set. The script explains that any set with more vectors than the dimension of the vector space will be linearly dependent, emphasizing the importance of this concept in determining the basis of a space.

💡Subspace

A subspace is a vector space that is a subset of another vector space, maintaining the properties of vector addition and scalar multiplication. The script discusses finding the basis and dimension of subspaces, such as those defined by certain conditions within R3 and R4, to illustrate the application of these concepts in more complex scenarios.

Highlights

Introduction to the concept of basis and dimension in vector subspaces, crucial for machine learning algorithms.

Basis vectors must be linearly independent and span the vector space.

Explanation of finite and infinite dimensional vector spaces.

The importance of basis in dimension reduction and dictionary learning algorithms.

Wavelet-based algorithms use orthonormal functions as a basis to approximate functions.

Mathematical definition of a basis for a vector space V over a field F.

Qualifications for a set to be a basis: linear independence and spanning the vector space.

Examples of bases for R2, R3, and RN with standard basis vectors.

Basis for 2x2 matrices and symmetric matrices as vector spaces.

Dimension of a vector space is the number of elements in its basis.

Dimension examples for R2, RN, and spaces of 2x2 matrices and polynomials.

Result: A basis of an n-dimensional vector space cannot have more than n vectors.

Result: Any linearly independent subset can be extended to form a basis.

Result: All bases of a finite-dimensional vector space have the same number of vectors.

Result: Dimension of the sum of two subspaces equals the sum of their dimensions minus the dimension of their intersection.

Finding the basis and dimension of a subspace defined by a linear equation in R3.

Example of finding the basis and dimension for subspaces S1 and S2 in R4.

Conclusion on the importance of understanding basis and dimension for further study in linear transformations.

Transcripts

[Music]

[Music]

if you remember in the last lecture we

have talked about uh Vector sub

spaces so today we are again going to

talk about a very important concept from

the vector spaces that is basis and

dimension of vector sub

spaces so in most of the machine

learning algorithm we need to

represent our input data that is in

terms of feature vectors as a linear

combination of certain

vectors that all the feature vectors we

want to write as linear combination of a

set of vectors those set of vectors may

be linearly independent or they may be

orthogonal so to write all the feature

vectors of our data set in terms of

linear combination of these vectors we

have to find out that particular set of

vectors so these vectors form a basis

for the feature space today we will

learn about the basis of the vector

spaces the concept of basis is really

important especially in the case of

Dimension

reduction or the recent algorithm like

dictionary learning based algorithm

further in wavelet based algorithms we

approximate any function as the linear

combination of set of orthonormal

functions those we have generated from

the

mother wavelength so let us come to the

definition of basis so

mathematically it's speaking the

definition of bases is given as let VF

be a vector space so we are having a

vector space V over the field F A basis

for V is a set of linearly independent

vectors in V which spans the vector

space V now so what

there should what should be the

qualification to be a basis the first

all the vectors of that set should be

linearly

independent number

two you take any Vector from the vector

space V that Vector can be written as

the linear combination of the vectors of

the basis set there are two types of

vector spaces finite dimensional Vector

spaces and infinite dimensional Vector

spaces so if the basis of a vector space

V contains the finite number of vectors

then we say that it is a finite

dimensional Vector space otherwise we

say that the vector space is an infinite

dimensional Vector

space a vector in basis is called the

basis Vector so if we talk this

definition

mathematically so what we are having a

set

B having vectors let us say V1

V2 up to

VN is a

basis

of a n dimensional Vector space

we if the set of

vectors V1 V2

VN is linearly

independent

that is the first thing we

need and the second thing

is for any

vector v belongs to the vector space

V we

have V =

to Alpha 1 V1 + Alpha 2

V2 plus alpha n

VN

where alpha

1 Alpha

2 alpha

n

are

scalars from the field

f

so this is uh mathematically we can

Define basis in this way also so if you

see some example of the

bases

so if you take V equals

to R2 over the field r

then if you take vectors like 1

Z and 01 in

R2 then this set forms a basis 4 V the

first thing both of these vectors are

linearly

independent and the second thing you

take any vector v from R2 we can write

that Vector let us say you are taking

some vector Alpha Beta arbitrary

Vector belongs to

R2

then we can write Alpha Beta as Alpha

* 1 + beta *

01 so what I want to say that this set

spans whole R2 space similarly if you

take V = to

R3 over the field of real

numbers then one of the possible basis

is 1 0 0 0 1 0 and 0 0

1 if you take V equals to

RNR then one of the possible basis is 1

0 0

0 0

1

0 0

0

1 so all these are nles vector having

one of the element is one and rest of

the elements are zero so all these three

are called standard basis for R2 R3 and

RN respectively if we

talk if we take a vector space like

this so R 2 by two matrices having real

entri

so

so certainly this V over the field of

real number forms a vector

space we have seen it in previous

lectures now what will be the basis of

this so basis of this will

be

so this is one of the possible basis for

this Vector space V here you can notice

all the vectors those are 2x2 matrices

are linearly

independent and any 2x2 matrix can be

written as the linear combination of

these four matrices

if I change this uh Vector space let us

say V equals

to m2x

2 set of

all

2x2

real symmetric

Matrix then what will be the basis

certainly this set over the field of

real numbers will form a vector space

and what will be the basis one of the

possible bases will be given

as 1 0

1 sorry 1 0 0

0 0 1 1

0 and 0 0 0

1 because this is space of symmetric

matrices so these two elements will be

equal to have symmetric to be the matric

symmetric so this basis will be having

three vectors while this is having four

vectors so these are some of the

examples of basis of different Vector

spaces my next definition is dimension

so what is the dimension of a vector

space

the dimension of a vector space is

nothing just the number of elements in

the

bases or number of vectors in the

basis so formally I can say the number

of vectors in a basis of VF is called

the dimension of the vector space VF so

as you have seen example uh the

dimension of vector space R2 over the

field of real number is two the

dimension of vector space RN over the

field of real number is n the dimension

of the vector space of all 2 by two real

matrices over the field of real number

is four if you are taking the vector

space of all M by n real matrices over

the field of real number then Dimension

will become M * n if you take the vector

space of all the pols having degree n or

less then the dimension of that Vector

space will be n + 1 because we will be

having n + 1 elements in the

basis similarly dimension of vector

space of all pols is

infinite why because it is a infinite

dimensional Vector

space we can write any Pol omal from

this Vector space as a linear

combination of finite number of vectors

from

that basis however basis will be

containing the infinite number of

elements now come to some important

results related to the basis and

dimension so my first result is let V1

V2 VN be a basis of a n dimensional

Vector space VF if s be a subset of

vectors having more than n

vectors then the set s is linearly

dependent so what I want to convey that

if the dimension of vector space is n

any set containing more than n vectors

will be linearly dependent because in a

set you can have at most and linearly

independent vector v

s and if you are having such a set where

you are having an linearly independent

Vector then that set will be a basis of

that particular Vector space so what in

other words I can say that the basis is

a maximal linearly independent set of a

vector space if you are having any other

vector in that set that Vector can be

written as a linear combination of rest

of the vectors my second result is let

V1 V2 VK be a linearly independent

subset of a n dimensional Vector space V

over the field F where K is less than

n then s can be extended to a basis of

VF for example suppose I am having a

Vector space R5 and I am having three

linearly independent vectors of R5 so

what I can do I can include two more

Alli vectors to that set of three

vectors and then I will be having five

linearly independent

vectors in

R5 and then this set will be a basis of

R5 another important result is like let

VF be a finite dimensional Vector space

then any two bases of V have the same

number of

vectors we can have multiple bases for a

vector space

however each of the bases will be having

the same number of

vectors that is equals to the dimension

of the vector space so for example if we

talk

about

r3r you have seen

that one of the set of vector 1 0

0 0 1

0 and 0 0

1 this particular set forms a basis for

R3 over the field of real numbers

if I take another set of three vectors

in R3 let us say 1 0

0 1 1

0 and 1 one

1 so again these are three vectors in

R3 and all this is a linearly

independent set

so B2 also a basis for R3 so we can have

many other basis where we are having

three Ali vectors from the set R3 from

the vector space R3 as a basis but the

common thing is all these sets will be

having three

vectors my next result is let V be a

vector space over the field F and it is

finite

dimensional if you take two sub species

of V let us say S1 and S2 then dimension

of S1 + dimension of S2 equal to

dimension of S1 + S2 Plus dimension of

S1 intersection S2 so we have written S1

here it will be S2 so dimension of S1

intersection S2 we will see this result

by this

example suppose we need to find out the

basis and dimension of the Subspace s of

R3 given Ed S = to X1 X2

X3 such that X1 + X2 - X3 = to 0 and we

are having another Subspace of R3 that

is W given by vectors X1 X2 X3 belongs

to R3 such that

X1 = to X2 = to X3 so how to find basis

of the so my Vector space

is R3 over the field of real

numbers and I am defining my set S

as the space of all the vectors X1 X2

X3 belongs to R3 such

that such that X1 +

X2 - X3 = to

0

so in the previous lecture we have

learned that how to prove that s is a

Subspace of

R3 now we need to find out basis of s so

here I can

write X1 X2

X3 belongs to

R3 such that X1 +

X2 =

to

X3 so I can write it

X1

X2 and since X3 = to X1 + X2 so I can

replace X3 as X1 +

X2

this I can write

as

X1

1 Z so one is coming from the first

component in second component we don't

have any X1 so X1 is zero and then

1

plus

X2 0 1

1 so here I can write the basis of

s is containing two vectors 1

1 and 0 1

1 hence dimension of

s equals to 2 so in that way we can find

out the dimension basis and dimension of

a vectory space take another example

there again we equals

to

R3 over the field of real number and we

are having w

h X1 X2

X3 such that X1 = to

X2 equals

to

X3 so what kind of Vector we are having

in W where all the three

components are equal so for example 1 1

1 2 2 2 -1 -1 -1 all these kind of

vectors so certainly the basis of I can

write it

X1

X1 X1 because X2 equals to X1 and X3 is

also

equal to

X1 belongs to

R3 so what is this it is

X1 1 1

1 so the basis of w

is having only one vector 1 one 1 so any

Vector of w can be written as some

scalar times this 1 one one so dimension

of w

is

1 if you need to find out basis

of s intersection W then what is s

intersection

W it is all vectors X1 X2

X3 belongs to

R3 such that at X1 +

X2 - X3 = 0 this condition is coming

from the Subspace

s

and the condition from Subspace W is X1

= to X2 = to

X3 so this I can

write X1

X2 X1 + X2 such that this I have written

by this condition such that X1 = to X2 =

to

X3 so if I take X1 = to

X2 then I can write it X1 now X2 = to X1

so

X1 and then X1 + X1

is

2x1 however what I need I need all the

three component should be

equal so in this case what we can have

the only possibility

is that it should have the zero

Vector because we need X1 X2 and X1 + X2

all three are equal it will be only

where then X1 is 0 X2 is0 and so that X1

+ X2 also become Z so this is the

basis of s intersection W hence

dimension

of s intersection W

is zero why zero because I told you

earlier

that the vector space containing the

zero Vector can be span by the empty set

by the basis Pi empty basis so there is

no element in the basis hence basis is

dimension of s intersection W is zero so

this is the way for finding the

basis and then dimension of a vector

space let us take one more example find

the basis and dimension of S1 S2 S1

intersection

S2 where S1 S2 are the subspaces of R4

over the field of real

numbers as I told you the intersection

of two subspaces also a Subspace so

again S1 intersection S2 is also a

Subspace of

R4 so let us find out the basis of S1 so

S1 is given as X X1

X2 X3

X4 belongs to R4 such that the first

constant is X1 + X2 -

X3 + X4 =

0 the second condition is X1 +

X2 + X3 + X4 =

to0 so what we are having here by adding

these two conditions what I can write X4

=

to - X1 -

X2 now from the first condition I can

write X3 = to X1 +

X2 plus

X4 if I substitute the value of X4 from

here that is - X1 - X2 I can get X3 = to

0 so let me write it here

Now

using these result so

X1 X2 X3 0 and X4 is - X1 -

X2 so it

is if I take X1 it is 1 0 0 - 1

+ X2 0 1 0 - 1

so here basis of S1

is 1 0 0

-1 and then 0 1 0 -

1 hence the dimension of S1 is 2

similarly you can find the dimension of

S2 using the same procedure and you will

find that dimension of S2 is coming 1 2

1 and 0 1 1 2

so hence basis of S2

is given by these two vectors and

dimension of S2 is 2 so now dimension of

S1 is 2 dimension of S2 is 2 so

dimension of S1 plus dimension of S2 is

4 which is equals to the vector space

dimension of the vector space R4 hence

dimension of S1 intersection S2 is zero

so the basis of S1 intersection S2

contains only zero

element which says that if you put all

these four conditions together the

solution will be X1 = to X2 = to X3 = to

X4 = to 0 all four components are zero

so in this lecture we have learned the

concept of basis and

dimension in the next lecture we will we

learn another very important concept of

mathematics related to the machine

learning that is linear

Transformations these are the

references for this

lecture hope you have enjoyed the

lecture thank you thank you very

[Applause]

[Music]

much

[Applause]

[Music]

[Laughter]

n