Dot Product and Matrix Properties

Introduction to Data Science Linear Algebra Dot Product and Matrix Properties

The Dot Product

Let’s say, there is a shop inventory that lists unit prices and quantities for each of the products they carry. If the shop has $10$ small-sized storage boxes each worth $\$5$ , $7$ medium-sized storage boxes each worth $\$10$ , and $5$ small storage boxes each worth $\$15$ . Now, the price vector $p$ and the quantity vector $q$ can be written as:

$p = \begin{bmatrix}5\\10\\15\end{bmatrix}, q = \begin{bmatrix}10\\7\\5\end{bmatrix}$

The net-worth of the boxes is calculated as:

$(10)(\$5)+(7)(\$10)+(5)(\$15) = \$195$

This operation of multiplying two entries in pairs of vectors and summing arises often in applications of linear algebra and is also foundational in basic linear algebra theory.

Definition: The dot product of two vectors $x, y$ in $\mathbb{R}^n$ is defined as:

$x . y = x_1y_1+x_2y_2+x_3y_3+....+x_ny_n$

Let us solve an example using S2.

Q: Determine $x .y$ given $x = \begin{bmatrix}1\\2\\3\\4\end{bmatrix}$ and $y = \begin{bmatrix}4\\5\\6\\7\end{bmatrix}$

				
					%use s2
// define vectors x &amp; y
var x = DenseVector(arrayOf(1.0, 2.0, 3.0, 4.0))
var y = DenseVector(arrayOf(4.0, 5.0, 6.0, 7.0))

// P = x.y
val P = x.multiply(y)
println(P)

[4.000000, 10.000000, 18.000000, 28.000000]

Geometrical Connection:

The dot product $x . y$ can also be calculated with the help of the angle $\theta$ between these two vectors as follows:

$x . y =\left|x\right|\left|y\right|cos \theta$

Let us find the dot product of the vectors in the figure below:

From the figure, vector $x$ can be written as: $(3\vec{i}+5\vec{j})-(0\vec{i}+2\vec{j}) = (3\vec{i}+3\vec{j})$

$\implies x = 3\vec{i}+3\vec{j}$

Similarly, vector $y$ can be written as: $(4\vec{i}+2\vec{j})-(0\vec{i}+2\vec{j}) = (4\vec{i}+0\vec{j})$

$\implies y = 4\vec{i}$

Given, $\theta = 45^\circ$

$x . y =\left|x\right|\left|y\right|cos \theta$

$=\left|3\vec{i}+3\vec{j}\right|\left|4\vec{i}\right|cos 45^\circ$

$=(\sqrt{3^2+3^2}) (\sqrt{4^2}) (\frac{1}{\sqrt{2}})$

$=(3\sqrt{2}) (4) \frac{1}{\sqrt{2}}$

$\implies x . y = 12$

Note: $x . y = 0$ if and only if the vectors $x$ and $y$ are orthogonal (perpendicular to each other).

Real World Application of Vector Dot Product

In Natural Language Processing, one of the basic ways to compare a finite number of text documents is to use
vectorized word counts. Let’s suppose the documents have a combined total of $n$ distinct words, which are arranged in some order. Each document is then associated with a vector of length $n$ whose $i^{th}$ entry indicates the number of times the $i^{th}$ word occurs in the associated document.

One way to measure similarity between two documents is to take the dot product of the associated
unit vectors: If two documents $A$ and $B$ have associated vectors $a$ and $b$ respectively, their similarity is defined by:

Similarity, $S(A, B) = \frac{a . b}{\left|a\right| \left|b\right|}$

As defined earlier, $x . y = \left|x\right| \left|y\right| cos \theta$

$\implies cos \theta = \frac{x . y}{\left|x\right| \left|y\right|}$

We know that, $0 \leq cos \theta \leq 1$

$\implies 0 \leq \frac{x . y}{\left|x\right| \left|y\right|} \leq 1$

Similarly we have, $0 \leq S(A, B) \leq 1$ for any two documents $A$ and $B$ .

Documents with no words in common are associated with orthogonal vectors and thus have $0$ similarity.
If two documents have similarity $1$ , their associated vectors are scalar multiples of each other, meaning that they have the same words and that the words appear in the same proportions.

Let us try to find the vectorized word count similarity between the following sentences:

A: “Julie likes John more than Kelly likes John”

B: “Joel likes Jonam more than Julie likes John”

Respective word count in A & B:

Word	Count in A	Count in B
Julie	1	1
John	2	1
Kelly	1	0
Joel	0	1
Jonam	0	1
likes	2	2
more	1	1
than	1	1

Therefore, the two vectors are:

$a = [1, 2, 1, 0, 0, 2, 1, 1]$ and $b = [1, 1, 0, 1, 1, 2, 1, 1]$

$S(A, B) = \frac{a . b}{\left|a\right| \left|b\right|}$

$\implies S(A, B) = \frac{(1+2+0+0+0+4+1+1)}{\sqrt {12} \sqrt {10}}$

$\implies S(A, B) = 0.821583836$

Approximately, $S(A, B) = 0.822$

Hence, the similarity between the two given sentences $A$ and $B$ , $S(A, B) = 0.822$ .

The Transpose

The dot product gives us a compact way to express the formula for an entry of a matrix product: to obtain
the $(i, j)^{th}$ entry of a matrix product $AB$ , we dot the $i^{th}$ row of $A$ and the $j^{th}$ column of $B$ .

However, the matrix product by itself is not quite flexible enough to handle a common use case: suppose
we have two matrices $A$ and $B$ which contain tabular data stored in the same format.

For example, suppose that the columns of $A$ store the vectorized word counts for a series of emails sent from Kelly, while $B$ stores vectorized word counts for a series of emails sent from Ana. If we want to calculate the similarity of each of Kelly’s email to each of Ana’s emails, then we want to dot the columns of $A$ but not its rows with the columns of $B$ .

So we define the transpose $A^T$ of a matrix $A$ to be the matrix resulting from switching $A$ ‘s rows and columns.

Definition: If $A$ is an $m \times n$ matrix, then its transpose $A^T$ is defined to be the matrix with $n$ rows whose $i^{th}$ row is equal to the $i^{th}$ column of $A$ , for each $i$ from $1$ to $n$ .

If $A = \begin{bmatrix}1 & 2 & 3\\4 & 5 & 6\end{bmatrix}$ , then $A^T = \begin{bmatrix}1 & 4\\2 & 5\\3 & 6\end{bmatrix}$

Let us implement the same using S2.

				
					%use s2
// define a matrix
var A = DenseMatrix(arrayOf(
    doubleArrayOf(1.0, 2.0, 3.0),
    doubleArrayOf(4.0, 5.0, 6.0)))
println(A)

// B = Tranpose of A
val B = A.t()
println("\n\nTranspose of A:\n")
println(B)

2x3
	[,1] [,2] [,3] 
[1,] 1.000000, 2.000000, 3.000000, 
[2,] 4.000000, 5.000000, 6.000000, 


Transpose of A:

3x2
	[,1] [,2] 
[1,] 1.000000, 4.000000, 
[2,] 2.000000, 5.000000, 
[3,] 3.000000, 6.000000,

Note:Transpose is a linear operator, meaning that $(cA + B)^T = cA^T + B^T$ , where $c$ is a constant and $A$ & $B$ are matrices.

Matrix Properties

Now that we know what the transpose of a matrix is, let’s learn about various matrix properties and their illustrations in S2.

Symmetric Matrix

A matrix in which its $(i,j)^{th}$ entry will be necessarily equal to its $(j,i)^{th}$ entry is known as a Symmetric Matrix.

In other words, if $A$ is an $n \times n$ matrix satisfying the equation $A = A^T$ , we say that $A$ is symmetric.

				
					%use s2
// define a matrix
var A = DenseMatrix(arrayOf(
    doubleArrayOf(1.0, 2.0, 3.0),
    doubleArrayOf(2.0, 4.0, 5.0),
    doubleArrayOf(3.0, 5.0, 6.0)))
println(A)

print("\n")
//prints 'true' if A is symmetric, else prints 'false'
val precision = 1e-15
println(MatrixPropertyUtils.isSymmetric(A,precision))

3x3
	[,1] [,2] [,3] 
[1,] 1.000000, 2.000000, 3.000000, 
[2,] 2.000000, 4.000000, 5.000000, 
[3,] 3.000000, 5.000000, 6.000000, 

true

Skew-Symmetric Matrix

If $A$ is an $n \times n$ matrix satisfying the equation $A^T = -A$ , we say $A$ is Skew-Symmetric.

				
					%use s2
// define a matrix
var A = DenseMatrix(arrayOf(
    doubleArrayOf(0.0, -6.0, 4.0),
    doubleArrayOf(6.0, 0.0, -5.0),
    doubleArrayOf(-4.0, 5.0, 0.0)))
println(A)

print("\n")
//prints 'true' if A is skew-symmetric, else prints 'false'
val precision = 1e-15
println(MatrixPropertyUtils.isSkewSymmetric(A,precision))

3x3
	[,1] [,2] [,3] 
[1,] 0.000000, -6.000000, 4.000000, 
[2,] 6.000000, 0.000000, -5.000000, 
[3,] -4.000000, 5.000000, 0.000000, 

true

Idempotent Matrix

If $A$ is an $n \times n$ matrix satisfying the equation $A = A*A$ or $A = A^2$ , we say $A$ is Idempotent.

				
					%use s2
// define a matrix
var A = DenseMatrix(arrayOf(
    doubleArrayOf(2.0, -2.0, -4.0),
    doubleArrayOf(-1.0, 3.0, 4.0),
    doubleArrayOf(1.0, -2.0, -3.0)))
println(A)

//prints 'true' if A is idempotent, else prints 'false'
val precision = 6.0
println("\n")
println(MatrixPropertyUtils.isIdempotent(A,precision))

3x3
	[,1] [,2] [,3] 
[1,] 2.000000, -2.000000, -4.000000, 
[2,] -1.000000, 3.000000, 4.000000, 
[3,] 1.000000, -2.000000, -3.000000, 


true

Orthogonal Matrix

If $A$ is an $n \times n$ matrix satisfying the equation $A.A^T = A^T.A = I$ where $I$ is an Identity Matrix of order $n$ , we say $A$ is Orthogonal.

				
					%use s2
// define a matrix
var A = DenseMatrix(arrayOf(
    doubleArrayOf(1.0, 0.0, 0.0),
    doubleArrayOf(0.0, 0.0, -1.0),
    doubleArrayOf(0.0, -1.0, 0.0)))
println(A)

//prints 'true' if A is orthogonal, else prints 'false'
val precision = 1e-15
println("\n")
println(MatrixPropertyUtils.isOrthogonal(A,precision))

3x3
	[,1] [,2] [,3] 
[1,] 1.000000, 0.000000, 0.000000, 
[2,] 0.000000, 0.000000, -1.000000, 
[3,] 0.000000, -1.000000, 0.000000, 


true

Previous Topic

Back to Lesson

Next Topic