Implementing Linear Regression from Scratch in Python

Linear Regression Python Fundamentals: Build Models from Scratch

Hello everyone, in this tutorial you will learn how to Implement a Linear Regression Model from Scratch in Python without using libraries like scikit-learn.

What are we building in this project?

We are building a linear regression model from scratch using Python for this project. In this project we develop our model to analyze the relationship between the independent variables and Dependent variables, Implementing key concepts like cost function and gradient descent for optimization.

How does it work?

Linear Regression relies on the mathematical principles of fitting a line to data points and trying to minimize the error between the actual value and the predicted value. The core idea of  Linear Regression is to find the straight line that best fits a dataset. The equation of the line is

y=m.x+b

The process: Step by step

1. Importing necessary libraries
2. Load and preprocess data
3.  Initialize parameters
4. Define the hypothesis function
5. Implement the cost function
6. compute gradients
7. Update parameters
8. Train the model

1. Libraries we need to import:

import numpy as np 
import pandas as pd 
import matplotlib.pyplot as plt

Explanation of libraries used:

Numpy: 

The purpose of NumPy is to provide support for large, multi-dimensional arrays and matrices, along with mathematical functions to operate on those arrays.

Pandas:

This library provides you with robust tools for data manipulation and analysis, making your work much easier.

Matplotlib:

Matplotlib library is for creating interactive, animated visualizations such as charts, bar charts, scatter plots, histograms, etc.

2. Load and preprocess data

df = pd.read_csv('your datset link')
df.head()

pd.read_csv():

• pd.read_csv reads a CSV file and loads its content into a pandas DataFrame.
• The DataFrame is a tabular data structure similar to a spreadsheet, with labeled rows and columns.

df.head():

• df.head() method returns the first five rows of the DataFrame
• this method helps to preview the structure of the dataset

df.info()
df = df.dropna(subset=['y'])
X = df['x'].values.reshape(-1,1)
y = df['y'].values.reshape(-1,1)

df.info():

provides a summary of our Dataframes’ structure including:

• Numbers of rows and columns.
• Column names and their data types.
• Number of non-null entries in each column.

df.dropna():

removes rows where the missing values are present.

df.[‘column_Name’].values.reshape(-1,1):

Prepares the feature and target variable for use in machine learning.

3. Initialize parameters:

input_size = x.shape[1]
weight = np.random.rand(input_size)
bias = 0

input_size = x.shape[1]:

Determines the number of futures in the input data.

Weight = np.random.rand(input_size):

Initializing the weights of the Linear regression Model randomly.

Bias = 0:

Initializing bias as 0.

Reason: 

The Linear Regression Equation is:

          y=X.weight + bias

OR

y=X.W+B

Where X     = input matrix

 weight(W) = vector of weights

  bias(B)  =  intercept term

4. Define the hypothesis function:

def hypothesis(X,weight,bias):
    return weight * X + bias

In the above block of code, we create a function hypothesis where we pass X, weight, and bias as parameters. We can rename the “hypothesis” as we wish.

The hypothesis functions return weight*X*bias.

5. Implement the cost function:

def cost_fun(X,y,weight,bias):
   m=len(y)
   predictions = hypothesis(X,weight,bias)
   return(1/(2*m))*np.sum((predictions-y)**2)

Here, we created another function named cost_func() and passed X,y, weight, and bias as parameters. In this cost_func(), we calculate the Mean Squared Error cost function for a linear regression model.

The cost function measures how well the model’s predictions match the true values.

6. compute gradients:

def com_grad(X, y, weight, bias):
    m = len(y)
    predictions = hypothesis(X, weight, bias)
    cost_weight = (1/m)*np.sum((predictions-y) *x)
    cost_bias   = (1/m)*np.sum(predictions-y)
    return cost_weight, cost_bias

Here we create, the com_grad() function and pass X,y, weight, and bias as parameters. Which returns the computed gradients for both weight and bias, we Initialize them as cost_weight, and cost_bias.

Where m is equal to the length of the target variable(y), the hypothesis of X, weight, and bias is stored in predictions, cost_weight calculates the gradient of the cost function for the weight parameter, cost_bias calculates the gradient of the cost function for the bias parameter.

7. Update parameters:

def upd_para(weight,bias,cost_weight,cost_bias,learning_rate):
     weight -= learning_rate*cost_weight
     bias -= learning_rate*cost_bias
     return weight,bias

In this upd_para() function, we pass weight, bias,cost_weight, cost_bias, and learning_rate as parameters, which returns weight and bias.

weight -= learning_rate*cost_weight

By calculating the product of learning_rate and cost_weight, we can determine how much to just the weight parameter in the direction that reduces the cost function.

The minus(-=)  sign means that we are subtracting the gradient from the current weight value. By doing this we can adjust the weight to reduce the cost (error).

bias -= learning_rate*cost_bias

we update the bias parameter by subtracting the product of learning_rate and the cost_bias.This tells us how much bias should be to reduce the error.

8. Train the model:

learning rate = 0.0001 
epochs = 1000
training_loss = []
for epoch in range (epochs):
    cost_weight,cost_bias = com_grad(x,y,weight,bias)
    weight,bias = upd_para(weight,bias,coast_weight,cost_bias,learning_rate)
    cost = cost_func(X,y,weight,bias)
    training_loss.append(cost)
    if epoch % 100 == 0:
        print("Epoch", epoch, "cost", cost)

learning rate = 0.0001

learning_rate controls how much the weight and the bias parameters are adjusted during each update. We can change the learning_rate value based on the cost value we are expecting.

epochs = 1000

An epoch refers to the complete pass-through of the entire dataset during our model training. In the above code, we set our epoch value to 1ooo, we can increase and decrease the epoch number during model training. Increasing the epoch number will affect our training time.

training_loss = []

creating an empty list to store errors (cost) at each epoch.

for epoch in range(epochs):

creating a loop that controls how many times the model will pass through the training process.

cost_weight,cost_bias = com_grad(X,y,weight,bias)

This calls the gradient function(com_grad)  to compute the cost_weight and cost_bias of the cost function for weight and bias.

weight,bias=upd_para(weight,bias,coast_weight,cost_bias, learning_rate)

This calls the update parameter to update weight and bias. By using this, we can adjust model parameters to reduce the cost.

cost = cost_function(X,y,weight,bias)

cost is calculated using updated values of gradients(weight, bias).

training_loss.append(cost)

computed cost is stored in the training_loss list.

if epoch%100 ==0:

This condition ensures that the cost is printed at intervals of 100 epochs. We can change this number 100 into any number, we wish to print our cost. For example, we can change if epoch % 100 == 0: into
if epoch % 50 == 0:
print(“Epoch”, epoch, “cost”, cost)
print statement to print Epoch number and corresponding cost