Classification using Logistic Regression

The difference between Linear and Logistic Regression, Image from https://dev.to/adityaberi8/logistic-regression-eg1

Binary Classification

We will first take a look at Binary Classification which involves classifying inputs into only two classes. For example, classifying if a person has a tumour or not, an email is spam or not spam, a customer will buy the product or not, etc. Let’s look at an example where we will try to determine if the customer will purchase a product or not using the gender, age, and salary of the person. Computers work well with numbers so we will use 0 and 1 for the two classes with 0 being the customer did not purchase the product and 1 being the customer did purchase the product. In linear regression, we had a hypothesis function with theta as a parameter which we could use to predict values.

This was the hypothesis function for Linear Regression where theta was a vector of parameters
This is the hypothesis function where the bottom function is called the sigmoid function or the logistic function

Cost/Loss Function

The cost function or the loss function will measure the error of our model. The cost function is a very useful metric to know how our model is performing. For Linear Regression, we used the mean squared error cost function but for Logistic regression we will use a different cost function because mean squared error is not so great for a classification problem. Before we see the cost function, let’s quickly recap some machine learning symbols.

This is the i-th training example, this can be only one value or this can be a vector of values if you have multiple features
This is the answer or the output of i-th training example
This is the cost function for Logistic Regression, Image from https://stats.stackexchange.com/questions/278771/how-is-the-cost-function-from-logistic-regression-derivated
This is the gradient descent algorithm where alpha is the learning rate and we update all the parameters theta simultaneously in every iteration of Gradient Descent

Python Implementation

Now that we know how this algorithm works, we can now use implement in python to solve a classification task. Logistic Regression is a much more complicated algorithm than Linear regression so we will not implement it from scratch. Instead, we will use the Logistic Regression model from the sklearn library. We will predict if a customer will buy a product based on their gender, salary, and age. We will be using the dataset found here : https://www.kaggle.com/dragonheir/logistic-regression

import pandas as pd
from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
data_frame = pd.read_csv('/Social_Network_Ads.csv')
data_frame.Gender = pd.Categorical(data_frame.Gender).codes
X = data_frame[['Gender', 'Age', 'EstimatedSalary']]
y = data_frame['Purchased']
X = X.to_numpy()
y = y.to_numpy()
scaler = MinMaxScaler()
scaler.fit(X)
X = scaler.transform(X)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model = LogisticRegression()
model.fit(X_train, y_train)
def pred(x_value):
x_value = scaler.transform([x_value])
print(model.predict(x_value))
print(model.score(X_train, y_train))

Multiclass Classification (One vs All Algorithm)

This is going to be a short section explaining multiclass classification because multiclass classification is just essentially binary classification with some changes. In this type of classification you have multiple hypothesis functions. For example, if you have 4 classes, let’s say blue, yellow, green, and red, then you will split it into 4 binary classification problems. You will make one hypothesis function for if our x value is blue or not blue, then you will make a hypothesis function for if your x value is yellow or not, and similarly for all the classes. When you want to predict classes for a new x value, you use all the hypothesis functions and whichever one has the highest value, you return that class. The python implementation for multiclass classification is identical as the binary classification so I won’t be showing that here.

Conclusion

Logistic Regression is one of the most widely used classification algorithms today. It can also be implemented fast using python. I hope you liked this article and learnt something new.

Thanks for Reading!

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store