Matrix Algebra

Introduction to Data Science Linear Algebra Matrix Algebra

A matrix is a rectangular array of numbers. The size of a matrix is described as the number of rows by the number of columns. In other words, a matrix with $m$ rows and $n$ columns is said to be an $m \times n$ matrix. The entry in the $i^{th}$ row and $j^{th}$ column of a matrix $A$ is referred to as $A$ ’s $(i, j)^{th}$ entry, and it is denoted by $A_{i,j}$ .

Matrices are versatile structures with a variety of problem-solving uses. For example,
1. A matrix can be thought of as a list of column vectors, so we can use a matrix to package many column
vectors into a single mathematical object.
2. An $m \times n$ matrix can be thought of as a linear transformation from $\mathbb{R}^n$ to $\mathbb{R}^m$ .

In this topic, we are going to develop both of these perspectives and define some operations which facilitate common manipulations that arise while handling matrices.

Matrix Operations

Matrix Addition

Matrix Addition for two $m \times n$ matrices $A$ and $B$ is the sum $A+B$ that results in another $m \times n$ matrix whose each entry is the sum of the corresponding entries in $A$ and $B$ .

If $A = \begin{bmatrix}1 & 2\\4 & 5\end{bmatrix}$ and $B = \begin{bmatrix}3 & 2\\8 & 6\end{bmatrix}$ , then compute $A+B$ .

				
					%use s2
// define the given matrices
var A = DenseMatrix(arrayOf(
    doubleArrayOf(1.0, 2.0),
    doubleArrayOf(4.0, 5.0)))
var B = DenseMatrix(arrayOf(
    doubleArrayOf(3.0, 2.0),
    doubleArrayOf(8.0, 6.0)))

//Perform Addition
// C = A+B
val C = A.add(B)
println(C)

2x2
	[,1] [,2] 
[1,] 4.000000, 4.000000, 
[2,] 12.000000, 11.000000,

Scalar Multiplication

The product of a number $c$ and an $m \times n$ matrix $A$ is defined to be the $m \times n$ matrix each of whose entries is $c$ times the corresponding entry of $A$ .

Compute the value of $2\begin{bmatrix}1 & 2\\4 & 5\end{bmatrix}$ .

				
					%use s2
// define matrix A
var A = DenseMatrix(arrayOf(
    doubleArrayOf(1.0, 2.0),
    doubleArrayOf(4.0, 5.0)))
//define C
val C = 2.0
// M = AC
val M = A.scaled(C)
println(M)

2x2
	[,1] [,2] 
[1,] 2.000000, 4.000000, 
[2,] 8.000000, 10.000000,

Matrix-Vector Multiplication

Let’s say $A$ is an $m \times n$ matrix and $x$ is a column vector in $\mathbb{R}^n$ , then $Ax$ is defined as the linear combination of the columns of $A$ with weights given by the entries of $x$ .

For a clear understanding, refer to the example below:

If matrix $A = \begin{bmatrix}1 & 2 & 3\\4 & 5 & 6\end{bmatrix}$ and vector $v=\begin{bmatrix}1\\2\\3\end{bmatrix}$ , then $Av = 1\begin{bmatrix}1\\4\end{bmatrix}+2\begin{bmatrix}2\\5\end{bmatrix}+3\begin{bmatrix}3\\6\end{bmatrix} = \begin{bmatrix}14\\32\end{bmatrix}$ .

Let us illustrate the same using S2.

				
					%use s2
// define matrix A
var A = DenseMatrix(arrayOf(
    doubleArrayOf(1.0, 2.0, 3.0),
    doubleArrayOf(4.0, 5.0, 6.0)))
// define a vector
var v = DenseVector(arrayOf(1.0, 2.0, 3.0))

// B = Av
val B = A.multiply(v)
println(B)

[14.000000, 32.000000]

Matrix Product

If $A$ is an $m \times n$ matrix and $B$ is an $n \times p$ matrix, then the matrix product of these two matrices $AB$ is an $m \times p$ matrix $C$ whose $k^{th}$ column is defined to be the product of $A$ and the $k^{th}$ column of $B$ .

For example, if $A = \begin{bmatrix}1 & 2\\4 & 5\end{bmatrix}$ and $B = \begin{bmatrix}3 & 4\\6 & 7\end{bmatrix}$ , then the matrix product of $A, B$ is:

$AB = C = \begin{bmatrix}\begin{bmatrix}1 & 2\\4 & 5\end{bmatrix}\begin{bmatrix}3\\6\end{bmatrix} & \begin{bmatrix}1 & 2\\4 & 5\end{bmatrix}\begin{bmatrix}4\\7\end{bmatrix}\end{bmatrix}=\begin{bmatrix}3\begin{bmatrix}1\\4\end{bmatrix}+6\begin{bmatrix}2\\5\end{bmatrix} & 4\begin{bmatrix}1\\4\end{bmatrix}+7\begin{bmatrix}2\\5\end{bmatrix}\end{bmatrix}$

Therefore, $C = \begin{bmatrix}15 & 18\\42 & 21\end{bmatrix}$

Implementing using S2 is much easier as you can see below.

				
					%use s2
// define matrices A &amp; B
var A = DenseMatrix(arrayOf(
    doubleArrayOf(1.0, 2.0),
    doubleArrayOf(4.0, 5.0)))
var B = DenseMatrix(arrayOf(
    doubleArrayOf(3.0, 4.0),
    doubleArrayOf(6.0, 7.0)))

// C = AB
val C = A.multiply(B)
println(C)

2x2
	[,1] [,2] 
[1,] 15.000000, 18.000000, 
[2,] 42.000000, 51.000000,

The Inverse of a Matrix

The inverse of an invertible matrix is another matrix which on multiplication with the given matrix results in an identity matrix.

For a matrix $A$ , $A^{-1}$ is its inverse and these two matrices $A$ and $A^{-1}$ satisfy the equation: $AA^{-1} = A^{-1}A = I$ where $I$ denotes the $n \times n$ identity matrix which has ones along the diagonal starting at the top left entry and zeros elsewhere.

Invertible Matrix Theorem

Statement: Suppose that $A$ is an $n \times n$ matrix, then the following are equivalent(that is, for a given matrix they are either all true or all false).

The transformation $x \rightarrow Ax$ from $\mathbb{R}^n$ to $\mathbb{R}^n$ is bijective.
The range of $A$ is $\mathbb{R}^n$ .
The null space of $A$ is $\left\{0\right\}$ .

Proof:

Let us begin by showing that (2) and (3) are equivalent.
If the columns of $A$ are linearly dependent, then the range of $A$ is spanned by fewer than $n$ vectors.
Therefore, if the rank of $A$ is equal to $n$ , then the columns of $A$ are linearly independent.
This implies that a linear combination of the columns is equal to the zero vector only if the weights are all zero. In other words, the only solution of the equation $Ax = 0$ is the zero vector.
In other words, the null space of $A$ is $\left\{0\right\}$ .
Conversely, if the null space of $A$ is $\left\{0\right\}$ , then the columns of $A$ are linearly independent, and the rank of $A$ is therefore equal to $n$ .
By definition of bijectivity, (2) and (3) together imply (1), and (1) implies (2) and (3). Therefore, the
three given statements are equivalent.

Computing inverse of an invertible matrix using S2:

				
					%use s2

// Create a matrix
val A = DenseMatrix(arrayOf(
    doubleArrayOf(5.0, 4.0, 4.0, 1.0, 5.0, 4.0, 2.0, 4.0, 1.0, 1.0), 
    doubleArrayOf(4.0, 5.0, 2.0, 2.0, 1.0, 2.0, 4.0, 5.0, 5.0, 2.0), 
    doubleArrayOf(5.0, 5.0, 3.0, 3.0, 5.0, 2.0, 3.0, 4.0, 1.0, 3.0), 
    doubleArrayOf(1.0, 2.0, 5.0, 5.0, 3.0, 1.0, 4.0, 3.0, 3.0, 3.0), 
    doubleArrayOf(2.0, 2.0, 4.0, 2.0, 3.0, 1.0, 3.0, 5.0, 4.0, 4.0), 
    doubleArrayOf(5.0, 4.0, 5.0, 1.0, 1.0, 3.0, 2.0, 3.0, 3.0, 4.0), 
    doubleArrayOf(3.0, 4.0, 4.0, 3.0, 4.0, 3.0, 2.0, 5.0, 5.0, 5.0), 
    doubleArrayOf(3.0, 4.0, 3.0, 3.0, 2.0, 1.0, 4.0, 2.0, 2.0, 1.0), 
    doubleArrayOf(4.0, 1.0, 1.0, 1.0, 1.0, 4.0, 4.0, 2.0, 2.0, 1.0), 
    doubleArrayOf(1.0, 5.0, 5.0, 5.0, 1.0, 1.0, 2.0, 4.0, 1.0, 4.0)))

// Compute the inverse
val Ainv = Inverse(A)
println("inverse: $Ainv\n")

//verification
val I = A.multiply(Ainv) // A*A_inverse = I, the identity matrix
println("A*Ainv = I: $I\n")
val det: Double = MatrixMeasure.det(I)
println("determinant of I: $det")

inverse: 10x10
	[,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] 
[1,] -0.053795, 0.540354, 0.510748, 0.754387, -0.463907, 0.379793, -0.429973, -0.927418, -0.199307, -0.302318, 
[2,] 0.001053, -0.302488, -0.254928, -0.616193, 0.184844, -0.173248, 0.320145, 0.765180, 0.035960, 0.192260, 
[3,] 0.147015, -0.039141, -0.159575, 0.114186, 0.055723, 0.143278, -0.125604, 0.033857, -0.110351, -0.006015, 
[4,] -0.038542, 0.511186, 0.375881, 0.827975, -0.618041, 0.145380, -0.229813, -0.892096, -0.138295, -0.131325, 
[5,] 0.043452, -0.227621, -0.015788, -0.167973, 0.124814, -0.146489, 0.205971, 0.313659, -0.016667, -0.069269, 
[6,] 0.105785, -0.243783, -0.286237, -0.295539, 0.006206, -0.156237, 0.295182, 0.273379, 0.264409, 0.178383, 
[7,] -0.086773, -0.457569, -0.250290, -0.616079, 0.523478, -0.228678, 0.128076, 0.843330, 0.287830, 0.162570, 
[8,] 0.124988, 0.342720, 0.122915, 0.217325, 0.045236, -0.040330, -0.304115, -0.571875, -0.039731, 0.070352, 
[9,] -0.012258, 0.205563, -0.049505, 0.243281, -0.199434, 0.080622, 0.113154, -0.106082, -0.144272, -0.205092, 
[10,] -0.239306, -0.394510, 0.010704, -0.450285, 0.297947, -0.029015, 0.238082, 0.378149, 0.185634, 0.129286, 

A*Ainv = I: 10x10
	[,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] 
[1,] 1.000000, -0.000000, 0.000000, -0.000000, -0.000000, 0.000000, 0.000000, -0.000000, 0.000000, 0.000000, 
[2,] 0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 0.000000, -0.000000, -0.000000, 0.000000, 0.000000, 
[3,] 0.000000, 0.000000, 1.000000, 0.000000, -0.000000, 0.000000, 0.000000, -0.000000, 0.000000, 0.000000, 
[4,] -0.000000, -0.000000, 0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, -0.000000, 
[5,] 0.000000, 0.000000, 0.000000, 0.000000, 1.000000, 0.000000, 0.000000, 0.000000, -0.000000, 0.000000, 
[6,] 0.000000, -0.000000, 0.000000, 0.000000, 0.000000, 1.000000, 0.000000, -0.000000, 0.000000, 0.000000, 
[7,] 0.000000, -0.000000, 0.000000, 0.000000, -0.000000, -0.000000, 1.000000, -0.000000, 0.000000, 0.000000, 
[8,] 0.000000, -0.000000, 0.000000, -0.000000, 0.000000, -0.000000, 0.000000, 1.000000, 0.000000, 0.000000, 
[9,] 0.000000, -0.000000, 0.000000, -0.000000, 0.000000, -0.000000, -0.000000, 0.000000, 1.000000, 0.000000, 
[10,] 0.000000, -0.000000, 0.000000, 0.000000, 0.000000, 0.000000, -0.000000, -0.000000, -0.000000, 1.000000, 

determinant of I: 1.0000000000000016

Previous Topic

Back to Lesson

Next Topic

Very helpful, I've needed this explanation for 15 years!

I believe there is a mistake in the above - most likely for the example of removing drift has been…

Great description! Thanks a lot!

I dont see the trail version of the license in 3.6.0 release. Can you please check? I also paid the…

[…] Transforming a QP Problem to an SOCP Problem [Numerical Method] […]