Section 1.2 - Vectors · 2 = v u u t nX 1 i=0 ˜2 i = q ˜2 0 + +˜2 n 1: Clearly kxk 2 = p xTx....
Transcript of Section 1.2 - Vectors · 2 = v u u t nX 1 i=0 ˜2 i = q ˜2 0 + +˜2 n 1: Clearly kxk 2 = p xTx....
Section 1.2 - Vectors
Maggie MyersRobert A. van de Geijn
The University of Texas at Austin
Practical Linear Algebra – Fall 2009
http://z.cs.utexas.edu/wiki/pla.wiki/ 1
Vectors
We will call a one-dimensional array of numbers a (real-valued)(column) vector:
x =
χ0
χ1...
χn−1
,
where χi ∈ R for 0 ≤ i < n.
The set of all such vectors is denoted by Rn.
http://z.cs.utexas.edu/wiki/pla.wiki/ 2
A vector has a direction and a length:
Its direction can be visualized by an arrow from the origin tothe point (χ0, χ1, . . . , χn−1).
Its length is given by the Euclidean length of this arrow:
‖x‖2 =√χ2
0 + χ21 + · · ·+ χ2
n−1, which is often called thetwo-norm.
There are other norms:
The “taxi-cab norm” or one-norm:‖x‖1 = |χ0|+ |χ1|+ · · ·+ |χn−1|.The “Infinity norm” or inf-norm:‖x‖∞ = max(|χ0|, |χ1|, · · · , |χn−1|).The “p-norm”, for p ≥ 1:
‖x‖p = (|χ0|p + |χ1|p + · · ·+ |χn−1|p)1/p.
http://z.cs.utexas.edu/wiki/pla.wiki/ 3
Warning
When we talk about the length of a vector, sometimes we mean n,the number of components in the vector, and sometimes we meanthe Euclidean length. Usually, it is clear from the context what wemean...
We will try to adhere to the following convention:
The length of a vector will refer to the Euclidean length(two-norm).
The size of the vector will refer to the number of componentsin the vector.
http://z.cs.utexas.edu/wiki/pla.wiki/ 4
Exercise
Let x ∈ R2 equal x =(
34
). Draw this vector, starting at the
origin. What is its length?
http://z.cs.utexas.edu/wiki/pla.wiki/ 5
Example
Let S be the set of all points in R2 with the property that a pointis in S iff (if and only if) the vector from the origin to that pointhas two-norm equal to one. Describe the set S.
Answer
S is the set of all point on the circle with center at the origin andradius one:
#" !�
��
(Why?)
http://z.cs.utexas.edu/wiki/pla.wiki/ 6
Exercise
Let S be the set of all points in R2 with the property that a pointis in S iff the vector from the origin to that point has one-normequal to one. Describe the set S.
http://z.cs.utexas.edu/wiki/pla.wiki/ 7
Exercise
Let S be the set of all points in R2 with the property that a pointis in S iff the vector from the origin to that point has one-normequal to one. Describe the set S.
Answer
@@
@@�
�
��
(Why?)
http://z.cs.utexas.edu/wiki/pla.wiki/ 8
Exercise
Let S be the set of all points in R2 with the property that a pointis in S iff the vector from the origin to that point has inf-normequal to one. Describe the set S.
http://z.cs.utexas.edu/wiki/pla.wiki/ 9
Exercise
Let S be the set of all points in R2 with the property that a pointis in S iff the vector from the origin to that point has inf-normequal to one. Describe the set S.
Answer
(Why?)
http://z.cs.utexas.edu/wiki/pla.wiki/ 10
Operations with vectors
In our subsequent discussion, let x, y ∈ Rn α ∈ R, with
x =
χ0
χ1...
χn−1
and y =
ψ0
ψ1...
ψn−1
.
http://z.cs.utexas.edu/wiki/pla.wiki/ 11
Equality
Two vectors are equal if all their components are element-wiseequal: x = y if and only if, for 0 ≤ i < n, χi = ψi.
http://z.cs.utexas.edu/wiki/pla.wiki/ 12
Copy (assignment)
The copy operation assigns the content of one vector to another:
y := x
(pronounce: y becomes x).
http://z.cs.utexas.edu/wiki/pla.wiki/ 13
Copy (assignment)
The copy operation assigns the content of one vector to another:
y := x
(pronounce: y becomes x).
http://z.cs.utexas.edu/wiki/pla.wiki/ 14
Copy
Algorithm Mscript
for i = 0, . . . , n− 1ψi := χi
endfor
for i=1:ny(i) = x( i );
end
http://z.cs.utexas.edu/wiki/pla.wiki/ 15
Warning
Mscript starts indexing at 1 (as mathematicians usually do).
We start indexing at 0 (as computer scientists usually do).
Get used to it!
http://z.cs.utexas.edu/wiki/pla.wiki/ 16
Copy as a routine
function [ yout ] = SLAP_Copy( x, y )
% Figure out the size (length) of vector x
[ m_x, n_x ] = size( x );
if ( m_x == 1 )
n = n_x;
else
n = m_x;
end
% Create an output vector of the same shape as y
yout = zeros( size( y ) );
for i=1:n
yout( i ) = x( i );
end
return
http://z.cs.utexas.edu/wiki/pla.wiki/ 17
Comments on copy as a routine
This routine seems REALLY convoluted.
The routine can also handle copying to and from (what wewill later call) a row vector.
The reason for why the routine is written in the given way willbecome clear later.
YES, we realize that this exact same operation can essentiallybe implemented as yout = x.
Trust us: things will start falling in place later.
http://z.cs.utexas.edu/wiki/pla.wiki/ 18
Copy - Cost
Copying one vector to another requires 2n memory operations(memops):
The vector x of length n must be read, requiring n memops.
The vector y must be written, which accounts for the other nmemops.
Note
We realize that there is also a cost with initializing yout to avector with zeros (yout = zeros( size( y ) );).
This is an artifact of Mscript. In a “real” language like C, thiswould not be the case (if implemented appropriately).
http://z.cs.utexas.edu/wiki/pla.wiki/ 19
Scaling (scal)
Multiplying vector x by scalar α yields a new vector, y = αx.
y has the same direction as x.
y is “stretched” (scaled) by a factor α.
Scaling a vector by α means each of its components, χi, isscaled by α:
αx = α
χ0
χ1...
χn−1
=
αχ0
αχ1...
αχn−1
.
http://z.cs.utexas.edu/wiki/pla.wiki/ 20
Scal
Algorithm Mscript
for i = 0, . . . , n− 1ψi := αχi
endfor
for i=1:ny(i) = alpha * x( i );
end
http://z.cs.utexas.edu/wiki/pla.wiki/ 21
Scal as a routine
function [ yout ] = SLAP_Scal( alpha, x, y )
% Figure out the size (length) of vector x
[ m_x, n_x ] = size( x );
if ( m_x == 1 )
n = n_x;
else
n = m_x;
end
% Create an output vector of the same shape as y
yout = zeros( size( y ) );
for i=1:n
yout( i ) = alpha * x( i );
end
return
http://z.cs.utexas.edu/wiki/pla.wiki/ 22
Comments on copy as a routine
This routine seems REALLY convoluted (this is the last timewe will say this).
YES, we realize that this exact same operation can essentiallybe implemented as yout = alpha * x.
http://z.cs.utexas.edu/wiki/pla.wiki/ 23
Scal - Cost
Scaling a vector requires 2n memory operations (memops) and nfloating point operations (flops):
The vector x of length n must be read, requiring n memops.
Each element must be multiplied by α, requiring n floatingpoint operations (flops).
The result vector y must be written, which accounts for theother n memops.
http://z.cs.utexas.edu/wiki/pla.wiki/ 24
Vector addition (add)
The vector addition x+ y is defined by
x+ y =
χ0
χ1...
χn−1
+
ψ0
ψ1...
ψn−1
=
χ0 + ψ0
χ1 + ψ1...
χn−1 + ψn−1
.
Note
Clearly, (in exact arithmetic), vector addition commutes:x+ y = y + x.
http://z.cs.utexas.edu/wiki/pla.wiki/ 25
Vector addition (add)
Algorithm Mscript
for i = 0, . . . , n− 1ζi := χi + ψi
endfor
for i=1:nz(i) = x( i ) + y( i );
end
Add - Cost
Adding two vectors requires 3n memory operations (memops) andn floating point operations (flops):
The vectors x and y of size n must be read, requiring 2nmemops.
Components must be added, requiring n flops.
The result vector y must be written, which accounts for theother n memops.
http://z.cs.utexas.edu/wiki/pla.wiki/ 26
Scaled vector addition (axpy)
One of the most commonly encountered operations is y := αx+ y:
αx+ y = α
χ0
χ1...
χn−1
+
ψ0
ψ1...
ψn−1
=
αχ0 + ψ0
αχ1 + ψ1...
αχn−1 + ψn−1
.
The name, axpy, stands for alpha times x plus y.
http://z.cs.utexas.edu/wiki/pla.wiki/ 27
axpy
Algorithm Mscript
for i = 0, . . . , n− 1ψi := αχi + ψi
endfor
for i=1:ny(i) = ...
alpha * x(i) + y(i);end
axpy - Cost
The cost of an axpy is 3n memory operations (memops) and 2nfloating point operations (flops):
The vectors x and y of size n must be read, requiring 2nmemops.
Components of x must be multiplied by α and added to thecorresponding component of y, requiring 2n flops.
The result vector y must be written, which accounts for theother n memops.
http://z.cs.utexas.edu/wiki/pla.wiki/ 28
Dot product (dot)
The other commonly encountered operation is the dot (inner)product defined by
xT y =n−1∑i=0
χiψi = χ0ψ0 + χ1ψ1 + · · ·+ χn−1ψn−1.
We have seen this before in Section 1.1...
http://z.cs.utexas.edu/wiki/pla.wiki/ 29
The transition from day k to day k + 1 is then written as thematrix-vector product (multiplication) χ
(k+1)s
χ(k+1)c
χ(k+1)r
=
0.4 0.3 0.10.4 0.3 0.60.2 0.4 0.3
χ
(k)s
χ(k)c
χ(k)r
,
or x(k+1) = Px(k)
This is simply a more compact representation (way of writing) theequations
χ(k+1)s = 0.4× χ(k)
s + 0.3× χ(k)c + 0.1× χ(k)
r
χ(k+1)c = 0.4× χ(k)
s + 0.3× χ(k)c + 0.6× χ(k)
r
χ(k+1)r = 0.2× χ(k)
s + 0.4× χ(k)c + 0.3× χ(k)
r
http://z.cs.utexas.edu/wiki/pla.wiki/ 30
dot
Algorithm Mscript
α = 0for i = 0, . . . , n− 1
α := α+ χi × ψi
endfor
alpha = 0;for i=1:n
alpha += x(i) * y(i);end
dot - Cost
The cost of a dot is (approximately) 2n memory operations(memops) and 2n floating point operations (flops):
The vectors x and y of size n must be read, requiring 2nmemops.
Corresponding components of x and y must be multiplied andadded to α, requiring 2n flops.
The result scalar α must be written, which accounts for theother 1 memops, which can be ignored.
http://z.cs.utexas.edu/wiki/pla.wiki/ 31
Exercise
Let x, y ∈ Rn. Show that xT y = yTx.
http://z.cs.utexas.edu/wiki/pla.wiki/ 32
Vector length (norm2)
Recall: The Euclidean length of a vector x (the two-norm) is givenby
‖x‖2 =
√√√√n−1∑i=0
χ2i =
√χ2
0 + · · ·+ χ2n−1.
Clearly ‖x‖2 =√xTx.
Cost
If computed with a dot product, it requires approximately nmemops and 2n flops.
Note
In practice the two-norm is typically not computed via the dotoperation, since it can cause overflow or underflow in floating pointarithmetic. Instead, vectors x and y are first scaled to prevent suchunpleasant side effects.
http://z.cs.utexas.edu/wiki/pla.wiki/ 33