Compute Gradient Descent of a Multivariate Linear Regression Model in R

x1, x2 …xn are the different features

Y is the value to be predicted.

For simplifying the notations, let’s consider we have another feature/variable X0 and this X0 = 1 for all the values in the dataset. Thus the above equation can be written as below since X0 = 1:

Y = h(𝛉) = 𝛉0X0  + 𝛉1X1 + 𝛉2X2 + 𝛉3X3 + 𝛉4X4… 𝛉nXn

Thus, all the features can be represented by an n+1 dimensional Feature Vector X like the below one:

And all the parameters can be represented by an n+1 dimensional Parameter Vector 𝚹:

Now, we can use the concept of Transpose of a Vector, and thus the Transpose of the above Vector 𝚹 is as below:

θT =[θ0 + θ1 + θ2 +θ3 +…+θn]

Finally, a Multivariate Linear Regression Model can be expressed like:

Y = h(𝛉) =θTX

where X is the Feature Vector and θT is the Transpose of the parameter vector.

Cost Function:

As we know, in a linear regression model, we need to find out the line that fits best with our current data set. To get the best fit for this line, we need to choose the best values for 𝛉1 and 𝛉2, 𝛉3, 𝛉4 … 𝛉n and so on. Thus, we can measure the accuracy of our prediction by using a cost function J(𝛉1,𝛉2,𝛉3…𝛉n).

J(θ0,θ1…θn) = 1/2m∑ (hθ(x(i)) – y(i))2

(θ0,θ1…θn) is an n+1 dimensional Parameter Vector

m = Total Number of examples in the DataSet.

hθ(x(i)) = predicted value for the ith data and can be represented as ŷ

So, the above Cost Function can be represented simply as below:

J(𝚹) = 1/2m∑ (ŷ(i) – y(i))2
where 𝚹 = a n+1 dimension vector for the parameter values

Gradient Descent:

Similar to the Gradient Descent for a Univariate Linear Regression Model, the Gradient Descent for a Multivariate Linear Regression Model can be represented by the below equation:

repeat until convergence
{
θj = θj  – α * 1/m∑ (hθ(x(i)) – y(i)). xj(i) 
where j = 0,1,2…n
}

Now, let’s discuss this with an example. Even in this case, I will use the dataset example of the Machine Learning course of Andrew Ng. We will implement a linear regression model with multiple variables to predict the prices of houses.

As earlier, we will divide the whole process into the below steps:

1. Load Data
2. Feature Scaling
3. Create a Cost Function
4. Create a Gradient  Descent Function and Plot the Gradient Descent Results
5. Check the results

Let’s start with the exercise!

1. Load Data: 

For this exercise, the Test Data is provided in a Text file with 3 columns and it contains a training set of housing prices in Portland, Oregon.

Let’s take a look at the data.

2. Feature Scaling