Abacus.AI now introduces Model Showcase to share your ML creativity with the world!

Latest in AI

A Primer on Linear Algebra – What You Need to Know Before Jumping Into Deep Learning

In this article, we will dig into some of the preliminary mathematical concepts, collectively known as linear algebra. A good foundation in Linear Algebra goes a long way for understanding Deep Learning.

Deep Learning is a subdomain of Machine Learning (ML) that simulates how our brains process the data and thus creates patterns for decision making. Underlying this sophisticated process are mathematical operations of Linear Algebra. So, without further ado, let’s dive in.

Linear Algebra

When you think of Linear Algebra, esoteric theories and findings you may have likely encountered in your math lessons may spring to your mind. In this post, we will focus on the important Linear Algebra concepts that form the bolts and nuts of Deep Learning Algorithms.

Some of these nuts and bolts include Scalers, Vectors, Matrices, and Tensors. Before we explain each of these terms, you need to understand what a Tensor is briefly.


Vectors and Matrices are examples of the general entity known as a Tensor. Going by the same token, a vehicle is a general entity whose instances can be classified into buses, trucks, and cars.

Each tensor has a rank (an order) that decides the number of dimensions in an array required to represent it.


Scalars represent single numbers and are examples of 0th– order tensors. They are expressed in mathematical notations that define the set of values to which a scalar belongs.

For example, X ∈ R denotes that X is a scalar value that belongs to a set of real-valued numbers: R. 

Likewise, in Deep Learning, there are various sets of numbers that are denoted by particular letters:

N – denotes a set of positive integers {1,2, 3…}

Z – represents positive and negative integers as well as zero

Q – represents rational numbers expressed as fractions of integers

Some of the scalar data types that Python supports are int, float, complex, bytes, and Unicode. There are 24 new fundamental data types to describe different scalars types in NumPy, a Python library. You can find more information about Python’s different scalar data types in this document.


Vectors are ordered arrays of single numbers. Unlike scalars, which are an example of a 1st-order tensor. Vectors have both, direction and magnitude. In contrast, a scalar only measures the magnitude which implies the size of an object. To quickly differentiate between a vector and a scalar, here is a common example:

You all love to travel in cars to your favorite destination. So, the distance to the destination is a scalar quantity as it is the rate of change in distance over a given period of time. However, the velocity of the car is a vector quantity as it is the speed of the car in a particular direction. 

A key factor to note on the vector is that they exist in a space. A vector space consists of the entire collection of all possible vectors of a particular length. Mathematically, our real-world definition of three-dimensional space is denoted by R3 which is the three-dimensional real-valued vector space. 

Typically, deep learning vectors represent feature vectors, with their original components determining how important a given feature is. Some examples would include the size of a set of pixels in a two-dimensional image or price values for a cross-section of financial instruments at a given time.

Defining vectors in Python

import numpy as np
#Declaring vectors
#The below statement doesn't add the vectors instead it combines both vectors
#Vectors are added using numpy
#calculating the vector cross product


<class 'list'>
[2, 8, 9, 5, 6, 7]
[ 7 14 16]
<class 'numpy.ndarray'>
[  2  31 -28]

Often you need to identify the components of a vector explicitly. The ith scalar element of a vector can be written as xi. An n-dimensional vector can be written using the following notation:

X=x1 x2 . . . x n


In mathematics, a Matrix is a rectangular array of numbers, symbols, or expressions  arranged in rows and columns. It is an example of 2nd-order tensors. If m and n are positive integers, that is m, n ∈ ℕ then the m×n matrix contains m rows and n columns. m×n matrix can be written as:

An element in the ith row and jth column can be written as follows:

There are various types of matrices:

  • Row Matrix: A Matrix having only one row is known as a Row Matrix.
  • Column Matrix: A Matrix having only one column is known as a column Matrix.
  • Zero or null Matrix: A Matrix having all elements as zero is known as a zero or null Matrix.
  • Square Matrix: A Matrix having an equal number of rows and columns is known as a Square Matrix.
  • Diagonal Matrix: A Matrix in which all elements are zero excluding the diagonal elements is known as a diagonal matrix.
  • Scalar Matrix: A Matrix where all the elements in the diagonal are equal is known as a scalar matrix.
  • Identity Matrix: A Matrix in which all the elements in the diagonal line are one is known as an Identity Matrix.

As with vectors, in Python, we use numpy libraries to create n-dimensional arrays known as matrices. We use matrix function to pass in the lists and thereby define a matrix.

Matrix Addition

It is possible to add scalars and vectors to matrices. Also, matrices can be added with each other. These various types of Matrix additions are frequently used in machine learning as well as deep learning. Thus, it is worthwhile to get familiar with them before moving into complex concepts in deep learning.

Matrix-Scalar Addition

Addition of a scalar to a Matrix could be performed as follows:

import numpy as np
x = np.matrix([[3, 1],[1, 2]])
sum = x.sum()
Output: 7

Matrix-Matrix Addition
Two matrices can be added only if they have the same dimensions, which implies they should have an equal number of rows and columns. Matrix-Matrix addition can be accomplished by adding corresponding elements in each Matrix:

Matrix Scalar Multiplication

In Matrix scalar multiplication, each element in the matrix is multiplied by the given scalar:

Matrix Multiplication
For matrix multiplication the number of columns in the first matrix must be equal to the number of rows in the 2nd matrix. The resulting matrix has the number of rows of the first matrix and the number of columns in the 2nd matrix:


As described earlier, a tensor is a general entity that encapsulates the scalar, vector, and matrix behavior. When you drill this down further, a tensor can be described as an array of numbers organized into a regular grid with a variable number of axes. Just like vectors and matrices you can perform arithmetic operations with tensors.

Below is a typical representation of a tensor:

       t111, t121, t131     t112, t122, t132     t113, t123, t133

T = (t211, t221, t231), (t212, t222, t232), (t213, t223, t233)

        t311, t321, t331     t312, t322, t332     t313, t323, t333

Creating a Tensor in Python

To perform arithmetic operations with Tensors it is much more productive and efficient to load the libraries such as TensorFlow or PyTorch.

import torch

The first line of code imports the torch library to your program. The 2nd line creates a Tensor of size (3,3) i.e., 3 rows and 3 columns. It is populated with Floating point zeros as shown below:

0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

We can also create Tensors with random numbers as you will discover in the code below.

import torch

It will produce the below (3,3) by populating with floating point numbers.

0.5413 0.1869 0.3412 0.0343 0.9839 0.8978 0.6938 0.6755 0.2258

Basic Math operations with Tensors

Here we will look into some of the basic tensor operations including:

  1. torch.add
  2. torch.mul

torch.add (input, alpha=1, other, out=None)

Break down of the function:

(input + alpha × other)


input: The first tensor

alpha: Multiplication factor

other: The second tensor

Code snippet:

import torch
x1=torch.rand(3,3) #Returns an array of ones with the same size as the given array

As the above code is self-explanatory, we will not cover the entire code.

2. torch.mul

Multiplication works similarly except for the fact that there isn’t any addition operator involved:

torch.mul (input, other, *, out=None)

Break down of the function:



Input: The first tensor

other: Multiplication factor


Now that you have a basic understanding of preliminary linear algebra concepts, we hope to see you again in the upcoming blog posts relating to core deep learning concepts.

Goodbye for now!

Related posts
AI ResearchLatest in AI

Abacus.AI at NeurIPS 2020

AI ResearchLatest in AI

A Study on Encodings for Neural Architecture Search

AI EducationLatest in AI

Beginner's Guide to Transformer Models

AI EducationAI ResearchLatest in AI

Debiasing Facial Prediction Models with Adversarial Fine-Tuning

Leave a Reply

Your email address will not be published. Required fields are marked *