# Use of Linear and Logistic Regression Coefficients with Lasso (L1) and Ridge (L2) Regularization for Feature Selection in Machine Learning

## Linear Regression

Let’s first understand what exactly linear regression is, it is a straight forward approach to predict the response y on the basis of different prediction variables such x and ε. . There is a linear relation between x and y.

𝑦𝑖 = 𝛽0 + 𝛽1.𝑋𝑖 + 𝜀𝑖

y = dependent variable
β0 = population of intercept
βi = population of co-efficient
x = independent variable
εi= Random error

#### Basic Assumptions

• Linear relationship with the target y
• Feature space X should have gaussian distribution
• Features are not correlated with other
• Features are in same scale i.e. have same variance

## Lasso (L1) and Ridge (L2) Regularization

Regularization is a technique to discourage the complexity of the model. It does this by penalizing the loss function. This helps to solve the overfitting problem.

• L1 regularization (also called Lasso)
It shrinks the co-efficients which are less important to zero. That means with Lasso regularization we can remove some features.
• L2 regularization (also called Ridge)
It does’t reduce the co-efficients to zero but it reduces the regression co-efficients with this reduction we can identofy which feature has more important.
• L1/L2 regularization (also called Elastic net)

A regression model that uses L1 regularization technique is called Lasso Regression and model which uses L2 is called Ridge Regression.

# What is Lasso Regularisation

3 sources of error

• Noise:
We can’t do anything with the noise. Let’s focus on following errors.
• Bias error:
It is useful to quantify how much on an average are the predicted values different from the actual value.
• Variance:
On the other side quantifies how are the prediction made on the same observation different from each other.

Now we will try to understand bias - variance trade off from the following figure.
By increasing model complexity, total error will decrease till some point and then it will start to increase. W need to select optimum model complexity to get less error.

If you are getting high bias then you have a fair chance to increase model complexity. And otherside it you are getting high variance, you need to decrease model complexity that’s how any machine learning algorithm works.

The L1 regularization adds a penalty equal to the sum of the absolute value of the coefficients.

We can observe from the following figure. The L1 regularization will shrink some parameters to zero. Hence some variables will not play any role in the model to get final output, L1 regression can be seen as a way to select features in a model.

Let’s observe the evolution of test error by changing the value of λ from the following figure.

#### How to choose λ

Let’s move ahead and choose the best λ.

We have a sufficient amount of data. In that we can split our data into 3 sets those are

• Training set
• Validation set
• Test set
• In the training set, we fit our model and set regression co-efficients with the regularization.
• Then we test our model’s performance to select λ
• on validation set, if any thing wrong with the model like less accuracy we validate on the validation set then we change the parameter the we go back to the training set and so the optimization.
• Finally, it will do generalize testing on the test set.

### What is Ridge Regularisation

Let’s first understand what exactly Ridge regularization:

The L2 regularization adds a penalty equal to the sum of the squared value of the coefficients.

λ is the tuning parameter or optimization parameter.
w is the regression co-efficient.

In this regularization,
if λ is high then we will get high bias and low variance.
if λ is low then we will get low bias and high variance

So what we do we will find out the optimized value of λ by tuning the parameters. And we can say λ is the strength of the regularization.

The L2 regularization will force the parameters to be relatively small, the bigger the penalization, the smaller (and the more robust) the coefficients are.

## Difference between L1 and L2 regularization

Let’s discuss the difference between L1 and L2 regularization:

import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import SelectKBest, SelectPercentile
from sklearn.metrics import accuracy_score

from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.feature_selection import SelectFromModel

titanic = sns.load_dataset('titanic')
titanic.head()
titanic.isnull().sum()

survived         0
pclass           0
sex              0
age            177
sibsp            0
parch            0
fare             0
embarked         2
class            0
who              0
deck           688
embark_town      2
alive            0
alone            0
dtype: int64

Let’s remove age and deck from the feature list

titanic.drop(labels = ['age', 'deck'], axis = 1, inplace = True)
titanic = titanic.dropna()
titanic.isnull().sum()
survived       0
pclass         0
sex            0
sibsp          0
parch          0
fare           0
embarked       0
class          0
who            0
embark_town    0
alive          0
alone          0
dtype: int64

Let’s get the features of the data

data = titanic[['pclass', 'sex', 'sibsp', 'parch', 'embarked', 'who', 'alone']].copy()
data.head()
data.isnull().sum()

pclass      0
sex         0
sibsp       0
parch       0
embarked    0
who         0
alone       0
dtype: int64
sex = {'male': 0, 'female': 1}
data['sex'] = data['sex'].map(sex)
data.head()
ports = {'S': 0, 'C': 1, 'Q': 2}
data['embarked'] = data['embarked'].map(ports)
who = {'man': 0, 'woman': 1, 'child': 2}
data['who'] = data['who'].map(who)
alone = {True: 1, False: 0}
data['alone'] = data['alone'].map(alone)
data.head()

Let’s read the data into x

X = data.copy()
y = titanic['survived']
X.shape, y.shape
((889, 7), (889,))
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state = 43)


### Estimation of coefficients of Linear Regression

First we will go with estimation of coefficients of linear regression

sel = SelectFromModel(LinearRegression())

Let’s go ahead and fit the model

sel.fit(X_train, y_train)
SelectFromModel(estimator=LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None,
normalize=False),
max_features=None, norm_order=1, prefit=False, threshold=None)

With these we have trained our model

Let’s see the features of the model

sel.get_support()
array([ True,  True, False, False, False,  True, False])

Now we will try to get the co-efficients

sel.estimator_.coef_
array([-0.13750402,  0.26606466, -0.07470416, -0.0668525 ,  0.04793674,
0.23857799, -0.12929595])

Let’s get the mean of the co-efficients

mean = np.mean(np.abs(sel.estimator_.coef_))
mean
0.13727657291370804

And calculate the absolute value of the co-efficients

np.abs(sel.estimator_.coef_)
array([0.13750402, 0.26606466, 0.07470416, 0.0668525 , 0.04793674,
0.23857799, 0.12929595])
features = X_train.columns[sel.get_support()]
features
Index(['pclass', 'sex', 'who'], dtype='object')

Let’s get the transformed version of x_train and x_test.

X_train_reg = sel.transform(X_train)
X_test_reg = sel.transform(X_test)
X_test_reg.shape
(267, 3)

Let’s implement the randomForest function and we wil do the training of the model.

def run_randomForest(X_train, X_test, y_train, y_test):
clf = RandomForestClassifier(n_estimators=100, random_state=0, n_jobs=-1)
clf = clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)
print('Accuracy: ', accuracy_score(y_test, y_pred))
%%time
run_randomForest(X_train_reg, X_test_reg, y_train, y_test)

Accuracy:  0.8239700374531835
Wall time: 250 ms


Now we will get the accuracy and wall time on the original data set

%%time
run_randomForest(X_train, X_test, y_train, y_test)

Accuracy:  0.8239700374531835
Wall time: 252 ms

X_train.shape

(622, 7)

### Logistic Regression Coefficient with L1 Regularization

Let’s move ahead with the L1 regularization to select the features.

sel = SelectFromModel(LogisticRegression(penalty = 'l1', C = 0.05, solver = 'liblinear'))
sel.fit(X_train, y_train)
sel.get_support()
array([ True,  True,  True, False, False,  True, False])

Let’s get the regression co-efficients.

sel.estimator_.coef_
array([[-0.54045394, 0.78039608, -0.14081954, 0., 0.        ,0.94106713, 0.]])

Let’s get the transformed version of x_train and x_test by using the function transform()

X_train_l1 = sel.transform(X_train)
X_test_l1 = sel.transform(X_test)


Now we will get the accuracy

%%time
run_randomForest(X_train_l1, X_test_l1, y_train, y_test)

Accuracy:  0.8277153558052435
Wall time: 251 ms


## L2 Regularization

Let’s move ahead with the L2 regularization to select the features.

sel = SelectFromModel(LogisticRegression(penalty = 'l2', C = 0.05, solver = 'liblinear'))
sel.fit(X_train, y_train)
sel.get_support()

array([ True,  True, False, False, False,  True, False])
sel.estimator_.coef_

array([[-0.55749685,  0.85692344, -0.30436065, -0.11841967,  0.2435823 ,
1.00124155, -0.29875898]])
X_train_l1 = sel.transform(X_train)
X_test_l1 = sel.transform(X_test)


Let’s check the accuracy of the data

%%time
run_randomForest(X_train_l1, X_test_l1, y_train, y_test)

Accuracy:  0.8239700374531835
Wall time: 250 ms


Subscribe
Notify of 1 Comment
Inline Feedbacks