02. Yet Another Way To Look At The System Of Equations

In school, you studied something called a system of linear equations. Let us begin exactly there, with a familiar example, and slowly change how we look at it.

2.1 - The Row Picture vs. The Column Picture

Consider the two equations in two variables.

In school, the standard way to solve this system was geometric. Each equation was treated as a line in the plane. The equation \(2x - y = 1\) can be rewritten as \(y = 2x - 1\). It is a line with slope \(m = 2\), passing through \((0, -1)\) and \((\tfrac12, 0)\). Similarly, the equation \(x + y = 5\) can be rewritten as \(y = 5 - x\). It is a line with slope \(m = -1\). When we draw both lines on the same coordinate plane, they intersect at the point \((2, 3)\). This point lies on both lines and therefore satisfies both equations. In school language, this point was called the unique solution of the system.

This way of thinking is called the row picture. Each equation is a row. Each row is a line. The solution is where the rows agree. Now we will look at the same problem from a different angle.

Let us write this system of linear equations in the matrix form as \(A\mathbf{x} = \mathbf{b}\). \(A\) is called the coefficient matrix, \(\mathbf{x}\) is the vector of unknowns, and \(\mathbf{b}\) is the target vector. This is the same system, written more compactly. Nothing has changed mathematically. But this form allows a new interpretation.

Instead of thinking row by row, we now think column by column. The above equation can be rewritten as,

This equation says something very concrete. Our goal is to find numbers \(x\) and \(y\) such that a linear combination of the two column vectors lands exactly on the target. In other words, How many steps along the vector \((2, 1)\), and how many steps along the vector \((-1, 1)\), do we need to reach \((1, 5)\)?

With some hit-and-trial, you can come up with the solution \(x=2\) and \(y=3\). Geometrically it says: Take \(2\) steps along the vector \((2, 1)\). This takes you to \((4, 2)\). Then take \(3\) steps along the vector \((-1, 1)\). This takes you from \((4, 2)\) to \((1, 5)\). You have reached the target.

Both viewpoints describe the same mathematics. But the column picture scales naturally to higher dimensions, where imagining intersection of lines/planes/hyperplanes to get a solution is no longer possible.

2.2 - Gaussian Elimination Method

In the previous example, we found the solution by reasoning and a bit of trial and error. That worked because the system was small and well-behaved. But this raises a natural question.: Is there a systematic method that will always tell us the values of the unknown variables? Is there an algorithm that works step by step, without guesswork? The answer is yes. Such a method was developed by the German mathematician Carl Friedrich Gauss, and it is now known as Gaussian Elimination. We will learn it through an example.

Consider the system of simultaneous linear equations.

Writing and rewriting equations quickly becomes tedious. Gaussian elimination avoids this by organizing all the information into a single object. We collect the coefficients of the variables and the constants on the right-hand side into an augmented matrix.

This matrix represents the entire system. Any operation we perform on the rows of this matrix corresponds to a valid transformation of the original equations. The advantage is simple but important. We no longer need to rewrite equations repeatedly. We manipulate the matrix directly.

Gaussian elimination is based on three allowed operations, called elementary row operations.

1. Swap two rows: \(R_i \leftrightarrow R_j\)
2. Multiply a row by a nonzero scalar: \(R_i \rightarrow c.R_i\)
3. Add a scalar multiple of one row to another row: \(R_i \rightarrow R_i + c.R_j\)

These operations do not change the solution of the system. They only simplify its appearance.

The goal is to transform the augmented matrix into an upper triangular form, also called row echelon form, where all entries below the main diagonal elements, also called pivots, are zero.

Step 1: First pivot

We begin with the first pivot, the entry in the \((1,1)\) position. Here, the pivot is \(1\). Our aim is to eliminate the entries below it. For this, we will perform two row operations: \(R_2 \rightarrow R_2 - 2R_1\) and \(R_3 \rightarrow R_3 - R_1\). After these operations, the augmented matrix becomes,

Now all entries below the first pivot are zero.

Step 2: Second pivot

We move to the second pivot, the entry in the \((2,2)\) position, which is \(2\). To eliminate the entry below it, we add half of the second row to the third row: \(R_3 \rightarrow R_3 + \tfrac{1}{2}R_2\). This gives

At this stage, the matrix is in upper triangular form. Once the matrix is in row echelon form, solving the system becomes straightforward by back-substitution.

From the last row,

Substitute \(z = 1\) into the second row,

Now substitute \(y = 2\) and \(z = 1\) into the first row,

So the solution of the system is \((x = -3, \text{ } y = 2, \text{ } z = 1)\). You know the geometric interpretation of this solution: we need to take \(3\) steps in the opposite direction of the first vector \((1, 2, 1)\), \(2\) steps along the second vector \((2, 6, 1)\) and \(1\) step along the third vector \((1, 1, 4)\), to reach the target vector \((2, 7, 3)\).

The natural question that now arises is: is it always possible to express one vector as a linear combination of others? In other words, is a solution always guaranteed? Let's answer this in the next section.

2.3 - Rank Of A Matrix

Not having a solution to the system of linear equations means it is not possible to reach to the target vector with the linear combination of vectors given in the matrix. To see if such a case is possible or not, let's try to see: given a collection of vectors, what new vectors can you form from them?

2.3.1 - Vector Space

Consider the matrix,

It has three column vectors.

All the vectors that can be reached using these three vectors can be written in the form

The set of all vectors that can be formed is called the vector space.

If a target vector lies in the span of the columns, the system has a solution. If it does not, no solution exists. This immediately connects back to the column picture. Solving \(A\mathbf{x} = \mathbf{b}\) is the same as asking: Is \(\mathbf{b}\) in the span of the columns of \(A\)?

We say, if all the \(n\) columns are linearly independent, then \(n\) vectors span the entire \(\mathbb{R}^n\) space, and the solution always exists. But, what is independence?

2.3.2 - Linear Independence and Redundant Vectors

Consider two vectors in 3D space.

What can you reach with these?

Notice the last coordinate: it's always zero. No matter what values you choose for \(x\) and \(y\), that third coordinate remains stubbornly at zero. You're trapped in the \(xy\)-plane. You can move east-west, north-south, but you cannot lift yourself off the ground. You have freedom in two dimensions, but you're imprisoned in the third.

Can you reach the vector \(\begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}\)? Absolutely not. To reach it, you'd need

which is impossible!

Now let's add a third vector. But here's the critical question: does it actually add new freedom? Suppose our third vector is

Notice anything? This vector also has zero in the third position. In fact, I can write it exactly as

This third vector is dependent: it brings nothing new. It's already reachable using \(\mathbf{v}_1\) and \(\mathbf{v}_2\). This means if \(A = \begin{bmatrix} 1 & 0 & 2 \\ 0 & 1 & 3 \\ 0 & 0 & 0 \end{bmatrix}\), then \(A.\mathbf{x} = \mathbf{b}\) is not solvable unless \(\mathbf{b}\) is of the form \(\begin{bmatrix} a \\ b \\ 0 \end{bmatrix}\).

Now consider a genuinely different third vector.

This vector points straight up, perpendicular to the \(xy\)-plane. Can you write this as a combination of \(\mathbf{v}_1\) and \(\mathbf{v}_2\)? Absolutely not. This vector is independent. It lives outside the plane and cannot be reached by any combination of \(\mathbf{v}_1\) and \(\mathbf{v}_2\). This gives us a new matrix:

Can we reach \(\begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}\) now? Yes! Simply choose \(x = 1, y = 1, z = 1\). In fact, every vector in 3D space is now reachable. Want \(\begin{bmatrix} 5 \\ -2 \\ 7 \end{bmatrix}\)? Take \(5\mathbf{v}_1 - 2\mathbf{v}_2 + 7\mathbf{v}_3^*\).

2.3.3 - The Rank

The above example was still pretty obvious. What happens when you get a matrix with the columns like the ones below?

At first glance, they might look independent, but upon careful observation, we will see \(\mathbf{u}_3 = \mathbf{u}_1 + 2.\mathbf{u}_2\). The matrix \(U = \begin{bmatrix} 1 & 2 & 5 \\ 2 & 1 & 4 \\ 1 & 3 & 7 \end{bmatrix}\) can only reach a 2D plane in 3D space, not the entire space.

This is still relatively simple. What I am trying to ask is: is there a way we can figure out if all the columns are independent or not? Yes! And luckily, the answer is our very old method: Gaussian Elimination!

That bottom row of zeros is the signature of dependence. It tells us: you only have two independent directions, not three.

The number of independent vectors is called the rank of the matrix. It tells you the dimension of the space you can actually reach. And that, ultimately, determines whether your system of equations has a solution. Equivalently, rank can also be defined as the number of non-zero rows in the row-echelon form of the matrix.

If the rank of the matrix of size \(3 \times 3\) is \(3\), it means any point in \(\mathbb{R}^3\) can be reached. The solution is guaranteed! Let's formalize this result in the next section.

2.4 - Solvability of System Of Linear Equations

We know that a system of linear equations can be written compactly as \(A\mathbf{x} = \mathbf{b}\). If \(\mathbf{b} = \mathbf{0}\), the system is called homogeneous. Else, if \(\mathbf{b} \neq \mathbf{0}\), the system is non-homogeneous. The term homogeneous is inspired from physics where it means of the same kind. Think of it like, homogeneous systems always describe planes (or lines) that pass through the origin, whereas non-homogeneous systems describe planes that may be shifted away from the origin. In physics, sometimes you are only interested in the nature of solutions and not exactly if planes or lines pass through origin or not.

Given below is the example of homogeneous system (on left) and non-homogeneous system (on right).

A homogeneous system always has at least one solution: the zero solution \(\mathbf{x} = \mathbf{0}\). So for homogeneous systems, the only possibilities are: exactly one solution (the trivial solution), or infinitely many solutions. There is never a no solution case.

Case (1): Homogeneous system with only one solution

Consider the system,

The augmented matrix is \([A|\mathbf{b}] = \left[\begin{array}{ccc|c} 1 & 1 & 1 & 0 \\ 2 & 1 & 1 & 0 \\ 1 & 2 & 1 & 0 \end{array}\right]\).

Here, we observe that every variable is a pivot. In other words, we don't have any free variables. The only solution is the zero vector, which means the planes intersect only at the origin.

Case (2): Homogeneous system with infinitely many solutions

This case would have come if all the elements of the last row had become zero. If we modify the above system by changing \(z\) to \(2z\) in the third equation, we will get zeros in the reduced form.

Since the last row is all zeros, we only have 2 independent vectors. This means one vector is redundant, or in other words, one variable is free.

Let \(z = t\) (the free variable).

From the second row, we see \(-y -z = 0 \implies y = -z \implies y = -t\).

Finally, \(x + y + z = 0 \implies x -t + t = 0 \implies x = 0\).

The solution is \(\mathbf{x} = t \begin{bmatrix} 0 \\ -1 \\ 1 \end{bmatrix} \forall \text{ } t \in \mathbb{R}\). The solution set is a line through the origin.

Let's have a look at non-homogenous cases. Here, all three solutions are possible, including the one with no solution.

Case (3): Non-homogeneous system with a unique solution

Let's consider the system.

The augmented matrix is \([A|\mathbf{b}] = \left[\begin{array}{ccc|c} 1 & 1 & 1 & 6 \\ 2 & 1 & 1 & 9 \\ 1 & 2 & 1 & 8 \end{array}\right]\). After same set of row operations, we will get \(\left[\begin{array}{ccc|c} 1 & 1 & 1 & 6 \\ 0 & -1 & -1 & -3 \\ 0 & 0 & -1 & -1 \end{array}\right]\).

Upon back substitution, we get \(z = 1, y = 2, x = 3\).

Case (4): Non-homogeneous system with no solution

If we modify the third equation in a way that we get full zeros in the last row of \(A\), the same system will have no solution.

During Gaussian Elimination, we will reach \(\left[\begin{array}{ccc|c} 1 & 1 & 1 & 6 \\ 0 & -1 & -1 & -3 \\ 0 & 0 & 0 & -1 \end{array}\right]\).

If we change the target vector by a little, so that its third component also gets zero by the end of row operations, we reach the final case.

Case (5): Non-homogeneous system with infinitely many solutions

Consider the system,

By Gaussian Elimination, we will reach \(\left[\begin{array}{ccc|c} 1 & 1 & 1 & 6 \\ 0 & -1 & -1 & -3 \\ 0 & 0 & 0 & 0 \end{array}\right]\). The way to solve this is similar to the homogeneous case, by using free variable.

Let \(z = t\) (the free variable).

From the second row, we see \(-y -z = -3 \implies y = 3-z \implies y = 3-t\).

Finally, \(x + y + z = 6 \implies x + (3-t) + t = 6 \implies x = 3\).

The solution is \(\mathbf{x} = \begin{bmatrix} 3 \\ 3-t \\ t \end{bmatrix} = \begin{bmatrix} 3 \\ 3 \\ 0 \end{bmatrix} + t\begin{bmatrix} 0 \\ -1 \\ 1 \end{bmatrix} \forall \text{ } t \in \mathbb{R}\). The solution set is a line but not through the origin.

2.4.1 - Rouché-Capelli Theorem