Example of Ridge and Lasso Regression

Data and background: https://charleshsliao.wordpress.com/2017/02/28/dplyr-rename-and-a-lame-regression/

Math comparison:

0228201703

Now let us build Ridge and Lasso regression to hunt down the smallest RMSE.

Ridge:

</pre>
library(car)

##
## Attaching package: 'car'

## The following object is masked from 'package:dplyr':
##
##     recode

vif(lmst)

##   lcavol  lweight      age     lbph      svi      lcp  gleason    pgg45
## 2.318496 1.472295 1.356604 1.383429 2.045313 3.117451 2.644480 3.313288

#Calculates variance-inflation and generalized variance-inflation factors

library(ridge)
ridgest<-linearRidge(lpsa~.,data=sctraining)
# With automatic lambda
summary(ridgest)

##
## Call:
## linearRidge(formula = lpsa ~ ., data = sctraining)
##
##
## Coefficients:
##             Estimate Scaled estimate Std. Error (scaled) t value (scaled)
## (Intercept)  2.45235              NA                  NA               NA
## lcavol       0.57254         4.65133             0.81585            5.701
## lweight      0.28198         2.29080             0.71988            3.182
## age         -0.09770        -0.79374             0.70738            1.122
## lbph         0.19669         1.59795             0.70707            2.260
## svi          0.27536         2.23705             0.79319            2.820
## lcp         -0.12023        -0.97673             0.86556            1.128
## gleason      0.02125         0.17261             0.81220            0.213
## pgg45        0.18652         1.51526             0.85700            1.768
##             Pr(>|t|)
## (Intercept)       NA
## lcavol      1.19e-08 ***
## lweight      0.00146 **
## age          0.26182
## lbph         0.02382 *
## svi          0.00480 **
## lcp          0.25914
## gleason      0.83170
## pgg45        0.07705 .
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Ridge parameter: 0.1128573, chosen automatically, computed using 5 PCs
##
## Degrees of freedom: model 6.541 , variance 5.451 , residual 7.632

#Ridge parameter: 0.1128573, chosen automatically, computed using 5 PCs
ridgepredictr<-predict(ridgest,newdata=testing)
actualr<-testing[,9]
ridgeRMSE=sqrt(mean((ridgepredictr - actualr)^2))
ridgeRMSE

## [1] 4.508077 ##the smallest RMSE here with the most fit lambda

ridge0.1<-linearRidge(lpsa~.,data=sctraining,lambda=0.1)
# With lambda of 0.1
ridge0.1predictr<-predict(ridge0.1,newdata=testing)
actualr<-testing[,9]
ridge0.1RMSE=sqrt(mean((ridge0.1predictr - actualr)^2))
ridge0.1RMSE

## [1] 4.655388

ridge0.01<-linearRidge(lpsa~.,data=sctraining,lambda=0.01)
# With lambda of 0.01
ridge0.01predictr<-predict(ridge0.01,newdata=testing)
actualr<-testing[,9]
ridge0.01RMSE=sqrt(mean((ridge0.01predictr - actualr)^2))
ridge0.01RMSE

## [1] 6.408921
<pre>

Lasso:


library(glmnet)

## Loading required package: Matrix

## Loading required package: foreach

## Loaded glmnet 2.0-5

lasso0.1<-glmnet(as.matrix(sctraining[,1:8]),as.matrix(sctraining[,9]),alpha=1,lambda = 0.1)
# With lambda of 0.1
lasso0.1predictr<-predict(lasso0.1,newx=as.matrix(testing[,1:8]),s=0.1)
actualr<-testing[,9]
lasso0.1RMSE=sqrt(mean((lasso0.1predictr - actualr)^2))
lasso0.1RMSE

## [1] 3.412481

coef(lasso0.1) # show the selected Vars

## 9 x 1 sparse Matrix of class "dgCMatrix"
##                     s0
## (Intercept) 2.45234509
## lcavol      0.57485458
## lweight     0.23032607
## age         .
## lbph        0.10584390
## svi         0.17231322
## lcp         .
## gleason     .
## pgg45       0.06584676

lasso0.01<-glmnet(as.matrix(sctraining[,1:8]),as.matrix(sctraining[,9]),alpha=1,lambda = 0.01)
# With lambda of 0.01
lasso0.01predictr<-predict(lasso0.01,newx=as.matrix(testing[,1:8]),s=0.1)
actualr<-testing[,9]
lasso0.01RMSE=sqrt(mean((lasso0.01predictr - actualr)^2))
lasso0.01RMSE

## [1] 5.510435

coef(lasso0.01) # show the selected Vars

## 9 x 1 sparse Matrix of class "dgCMatrix"
##                     s0
## (Intercept)  2.4523451
## lcavol       0.6853238
## lweight      0.2868119
## age         -0.1209862
## lbph         0.2008499
## svi          0.2886945
## lcp         -0.2243574
## gleason      .
## pgg45        0.2278427

#now let us find the idealist lambda for Lasso with CV of K=10
lassoseek<-cv.glmnet(as.matrix(sctraining[,1:8]),as.matrix(sctraining[,9]))
# Cross Validate and get the fit
plot(lassoseek)

<img class="alignnone size-full wp-image-371" src="https://charleshsliao.files.wordpress.com/2017/02/0228201704.png" alt="0228201704" width="1112" height="822" />

lassoseek$lambda.1se # best lambda

## [1] 0.1646861

lassobest<-lassoseek$glmnet.fit # the model
lbcoef<-coef(lassobest,s=lassoseek$lambda.1se)
lbcoef

## 9 x 1 sparse Matrix of class "dgCMatrix"
##                     1
## (Intercept) 2.4523451
## lcavol      0.5672725
## lweight     0.2055303
## age         .
## lbph        0.0444131
## svi         0.1266483
## lcp         .
## gleason     .
## pgg45       0.0262882

Ridge regression gives up partly accuracy to have a better fit with flawed data set, which is more practical than ordinary regression.

Lasso is typically useful for large dataset with high dimensions. It can select the most related variables and finish the job.

Lambda is the shrinkage parameter.

 

 

Advertisements

1 thought on “Example of Ridge and Lasso Regression”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s