Orthogonal and Orthonormal Vectors in Linear Algebra

Two vectors are said to be orthogonal if they are perpendicular to each other, meaning they form a 90-degree angle when placed in the same coordinate system. This concept is fundamental in linear algebra and has significant applications in various fields.

In two-dimensional space, if two vectors are orthogonal, they meet at a right angle (90°). In three-dimensional space, this means they lie on mutually perpendicular planes. The idea of orthogonality extends to higher dimensions as well, meaning two vectors are still perpendicular even if they exist in a space with more than three dimensions.

Notation: 

A⊥B means A·B = 0

Where: A and B are vectors, and ⋅ represents the dot product between the two.

Example

1) 0·x = 0 for any vector x, the zero vector is orthogonal to every vector in Rⁿ.(n-dimensional real space)

2) Consider two vectors: u= (3, 4) and v= (−4, 3).

Their dot product is: (3 × −4)+ (4 × 3)= −12+12=0

Since the dot product is zero, u and v are orthogonal.

How do we define the dot product?

Dot product(scalar product) of two n-dimensional vectors A and B, is given by this expression.

A . B=\sum_{i=1}^{n} a_{i} b_{i}        

Thus the vectors A and B are orthogonal to each other if and only if

A.B=\sum_{i=1}^{n} a_{i} b_{i}=A^{T} B=0        

Note: In a compact form the above expression can be written as (A^T)B.

Example:

Consider the vectors v₁ and v₂ in 3D space, v_1=\left[\begin{array}{c} 1 \\ -2 \\ 4 \end{array}\right], v_{2}=\left[\begin{array}{l} 2 \\ 5 \\ 2 \end{array}\right]        
Taking the dot product of the vectors v_{1}, v_{2}=v_1^{T} v_2=[1-24]\left[\begin{array}{l} 2 \\ 5 \\ 2 \end{array}\right]=0        
Hence the vectors are orthogonal to each other.

Code: Python program to illustrate orthogonal vectors. 

// C++ program to calculate the dot product of two vectors
#include <iostream>
#include <vector>

using namespace std;

// Function to calculate the transpose of a matrix
vector<vector<int> > transpose(vector<vector<int> > matrix)
{
    vector<vector<int> > transpose(matrix[0].size(), vector<int>(matrix.size()));
    
    for (int i = 0; i < matrix.size(); i++) {
        for (int j = 0; j < matrix[0].size(); j++) {
            transpose[j][i] = matrix[i][j];
        }
    }
    
    return transpose;
}

// Function to calculate the dot product of two vectors
int dot(vector<vector<int> > vect_A, vector<vector<int> > vect_B) 
{
    int product = 0;
    int n = vect_A[0].size();
    
    // Loop to calculate the dot product
    for (int i = 0; i < n; i++) {
        product = product + vect_A[0][i] * vect_B[i][0];
    }
    
    return product;
}

int main() 
{
    vector<vector<int> > v1 = {{1, -2, 4}};
    vector<vector<int> > v2 = {{2, 5, 2}};
    
    vector<vector<int> > transposeOfV1 = transpose(v1);
    
    int result = dot(v2, transposeOfV1);
    
    cout << "Result = " << result << endl;
    
    return 0;
}

import java.util.Arrays;

public class OrthogonalVector {
    public static void main(String[] args) {
        int[][] v1 = {{1, -2, 4}};
        int[][] v2 = {{2, 5, 2}};

        int[][] transposeOfV1 = transpose(v1);

        int result = dot(v2, transposeOfV1);
        System.out.println("Result = " + result);
    }

    public static int[][] transpose(int[][] matrix) {
        int[][] transpose = new int[(matrix[0].length)][(matrix.length)];
        for (int i = 0; i < matrix.length; i++) {
            for (int j = 0; j < matrix[0].length; j++) {
                transpose[j][i] = matrix[i][j];
            }
        }
        return transpose;
    }
    static int dot(int vect_A[][], int vect_B[][])
    {
 
        int product = 0;
         int n = vect_A[0].length;
        // Loop for calculate dot product
        for (int i = 0; i < n; i++)
            product = product + vect_A[0][i] * vect_B[i][0];
        return product;
    }
}

# A python program to illustrate orthogonal vector
 
# Import numpy module
import numpy
 
# Taking two vectors
v1 = [[1, -2, 4]]
v2 = [[2, 5, 2]]
 
# Transpose of v1
transposeOfV1 = numpy.transpose(v1)
 
# Matrix multiplication of both vectors
result = numpy.dot(v2, transposeOfV1)
print("Result  = ", result)
 
# This code is contributed by Amiya Rout

using System;

public class OrthogonalVector
{
    public static void Main()
    {
        int[][] v1 = { new[] { 1, -2, 4 } };
        int[][] v2 = { new[] { 2, 5, 2 } };

        int[][] transposeOfV1 = Transpose(v1);

        int result = Dot(v2, transposeOfV1);
        Console.WriteLine("Result = " + result);
    }

    public static int[][] Transpose(int[][] matrix)
    {
        int[][] transpose = new int[(matrix[0].Length)][];
        for (int i = 0; i < matrix[0].Length; i++)
        {
            transpose[i] = new int[(matrix.Length)];
            for (int j = 0; j < matrix.Length; j++)
            {
                transpose[i][j] = matrix[j][i];
            }
        }
        return transpose;
    }

    static int Dot(int[][] vect_A, int[][] vect_B)
    {
        int product = 0;
        int n = vect_A[0].Length;
        // Loop for calculate dot product
        for (int i = 0; i < n; i++)
        {
            product = product + vect_A[0][i] * vect_B[i][0];
        }
        return product;
    }
}

// Function to calculate the transpose of a matrix
function transpose(matrix) {
    const transpose = new Array(matrix[0].length);
    
    for (let i = 0; i < matrix[0].length; i++) {
        transpose[i] = new Array(matrix.length);
        for (let j = 0; j < matrix.length; j++) {
            transpose[i][j] = matrix[j][i];
        }
    }
    return transpose;
}

// Function to calculate the dot product of two matrices
function dotProduct(matrixA, matrixB) {
    const n = matrixA[0].length;
    let product = 0;

    for (let i = 0; i < n; i++) {
        product += matrixA[0][i] * matrixB[i][0];
    }

    return product;
}

// Driver Code
const v1 = [[1, -2, 4]];
const v2 = [[2, 5, 2]];

const transposeOfV1 = transpose(v1);

const result = dotProduct(v2, transposeOfV1);
console.log("Result = " + result);

Result = 0

Note: An orthogonal set of non-zero vectors is linearly independent.

Unit Vector:

Let's consider a vector A. The unit vector of the vector A can be defined as \hat{a}=\frac{A}{|A|}        

Example: Consider a vector A in 2D space. A=\left[\begin{array}{l} 3 \\ 4 \end{array}\right]        
The magnitude of A is given by\text { Magnitude of } \mathrm{A}:|A|=\sqrt{3^{2}+4^{2}}=5        
So the unit vector of A can be calculated as \hat{a}=\frac{A}{|A|}=\left[\begin{array}{l} 3 / 5 \\ 4 / 5 \end{array}\right]        

Properties of unit vector:

• Unit vectors are used to define directions in a coordinate system.
• Any vectors can be written as a product of a unit vector and a scalar magnitude.

Orthonormal vectors

a set of vectors A is orthonormal if every vector in A has magnitude 1 and the set of vectors are mutually orthogonal.

Now, take the same 2 vectors which are orthogonal to each other and you know that when I take a dot product between these 2 vectors it is going to 0. So If we also impose the condition that we want each of these vectors to have unit magnitude then what we could possibly do is by taking this vector and then divide this vector by the magnitude of this vector as we see in the unit vector.

Now we can write v₁ and v₂ as v_{1}=\left[\begin{array}{c} 1 \\ -2 \\ 4 \end{array}\right] / \sqrt{1^{2}+(-2)^{2}+4^{2}} \quad

v_{2}=\left[\begin{array}{l} 2 \\ 5 \\ 2 \end{array}\right] / \sqrt{2^{2}+5^{2}+2^{2}}        

So what we do is we have taken the vectors from the previous example and converted them into unit vectors by dividing them by their magnitudes. So, these vectors will still be orthogonal to each other and now individually they also have unit magnitude. Such vectors are known as orthonormal vectors.

Note: All orthonormal vectors are orthogonal by the definition itself.

Gram-Schmidt Process

The Gram-Schmidt process is a method for converting a set of linearly independent vectors into an orthonormal set, meaning that the vectors are both orthogonal to each other and have unit length. Here is how it works:

Projection Operator

First, we define the projection operator. The projection of a vector v onto u is given by:

{Proj}_{\mathbf{u}} \mathbf{v} = \frac{\mathbf{v} \cdot \mathbf{u}}{\| \mathbf{u} \|^2} \mathbf{u}

Where:

• v⋅u is the dot product of v and u
• ∥u∥ ² is the square of the norm of u

Example:

Given the vectors:

• \mathbf{v} = \begin{bmatrix} 1 \\ 1 \end{bmatrix}
• \mathbf{u} = \begin{bmatrix} 1 \\ 0 \end{bmatrix}

We compute the projection of v onto u:

\text{Proj}_{\mathbf{u}} \mathbf{v} = \frac{(1)(1) + (1)(0)}{1^2 + 0^2} \begin{bmatrix} 1 \\ 0 \end{bmatrix} = 1 \cdot \begin{bmatrix} 1 \\ 0 \end{bmatrix} = \begin{bmatrix} 1 \\ 0 \end{bmatrix}

Now, subtract the projection from v to get an orthogonal vector v₁​:

\mathbf{v_1} = \mathbf{v} - \text{Proj}_{\mathbf{u}} \mathbf{v} = \begin{bmatrix} 1 \\ 1 \end{bmatrix} - \begin{bmatrix} 1 \\ 0 \end{bmatrix} = \begin{bmatrix} 0 \\ 1 \end{bmatrix}

Thus, v₁​ is orthogonal to u.

Gram-Schmidt Algorithm

To orthonormalize a set of linearly independent vectors v₁,v₂,…,v_n we proceed as follows:

1. Step 1: Let u ₁ = v ₁ ​, then normalize it to get \mathbf{e_1} = \frac{\mathbf{u_1}}{\| \mathbf{u_1} \|}
   
2. Step 2: Let u ₂ = v ₂ - \text{Proj}_{\mathbf{u_1}} \mathbf{v_2} ​, then normalize it to get \mathbf{e_2} = \frac{\mathbf{u_2}}{\| \mathbf{u_2} \|}
   
3. Step 3: Let u ₃ =v ₃ − \text{Proj}_{\mathbf{u_1}} \mathbf{v_3} - \text{Proj}_{\mathbf{u_2}} \mathbf{v_3} ​, then normalize it to get \mathbf{e_3} = \frac{\mathbf{u_3}}{\| \mathbf{u_3} \|}

And this process continues for v₄,v₅,… subtracting projections onto all previous vectors to ensure orthogonality, and normalizing to get unit vectors. The result is an orthonormal set e_1,e₂,…,e_n​.

Example:

continue with the example of Projection Operator

v₁​ is orthogonal to u.
Normalize the Vectors to Get Orthonormal Vectors

Now, we normalize both v and v₁​ to get the orthonormal vectors:

1. Normalize u to get e₁​ : \mathbf{e_1} = \frac{\mathbf{u}}{\|\mathbf{u}\|} = \frac{1}{1} \begin{bmatrix} 1 \\ 0 \end{bmatrix} = \begin{bmatrix} 1 \\ 0 \end{bmatrix}
2. Normalize v_1​ to get e₂ ​: \mathbf{e_2} = \frac{\mathbf{v_1}}{\|\mathbf{v_1}\|} = \frac{1}{\sqrt{2}} \begin{bmatrix} 0 \\ 1 \end{bmatrix} = \begin{bmatrix} 0 \\ 1 \end{bmatrix}

Final Orthonormal Set:

• \mathbf{e_1} = \begin{bmatrix} 1 \\ 0 \end{bmatrix}
• \mathbf{e_2} = \begin{bmatrix} 0 \\ 1 \end{bmatrix}

This set of vectors is orthonormal because:

• They are orthogonal (e ₁ ⋅e ₂ =0)
• They have unit length ( \|\mathbf{e_1}\| = \|\mathbf{e_2}\| = 1 )

Practice problems

1) Given the vectors \mathbf{v_1} = \begin{bmatrix} 2 \\ 1 \end{bmatrix} and \mathbf{v_2} = \begin{bmatrix} -1 \\ 4 \end{bmatrix}, check if they are orthogonal.

2) Let \mathbf{v} = \begin{bmatrix} 4 \\ 3 \end{bmatrix} and \mathbf{u} = \begin{bmatrix} 1 \\ 2 \end{bmatrix}. Compute the projection of v onto u.

3) Given two linearly independent vectors \mathbf{v_1} = \begin{bmatrix} 1 \\ 1 \end{bmatrix} and \mathbf{v_2} = \begin{bmatrix} 1 \\ 2 \end{bmatrix}, apply the Gram-Schmidt process to find an orthonormal basis.

4) Given the set of vectors \mathbf{v_1} = \begin{bmatrix} 1 \\ 1 \end{bmatrix} and \mathbf{v_2} = \begin{bmatrix} 2 \\ -2 \end{bmatrix}, check if they form an orthonormal set. If not, make them orthonormal by applying normalization.

Orthogonal and Orthonormal Vectors - FAQs

What is the relationship between orthogonal and orthonormal vectors?

All orthonormal vectors are orthogonal, but not all orthogonal vectors are orthonormal. To be orthonormal, vectors must also have a unit length, in addition to being orthogonal.

Can two vectors be orthogonal if they are not in the same dimension?

No, two vectors must reside in the same vector space (i.e., have the same dimension) for their dot product to be defined and for them to be considered orthogonal.

Can orthogonal vectors be used to represent coordinates in a space?

Yes, orthogonal vectors are often used to represent coordinate axes in a vector space, especially in orthogonal coordinate systems (e.g., the x-axis and y-axis in 2D space).

Are orthogonal vectors always linearly independent?

Yes, if two vectors are orthogonal, they are also linearly independent. If one vector is a scalar multiple of the other, their dot product will not be zero, and they will not be orthogonal.

In machine learning, how do orthogonal vectors help with feature selection?

Orthogonal vectors, especially in techniques like PCA (Principal Component Analysis), help to select uncorrelated features, improving model performance by reducing redundancy and enhancing generalization.