Flexibility versus Interpretability

I started going through An Introduction to Statistical Learning, and the first thing that struck me is that linear regression gets two whole chapters devoted to it (Chapters 3 and 6). I was a bit surprised because my naive view was that linear regression was something primitive and completely separate from the concepts of machine learning, whereas in this book, linear regression is presented as a baseline or starting place for the concepts of machine learning. This left me with a question. Why is econometrics as usually presented as something separate methodologically from machine learning?

In chapter 2, I think I got the answer to my question. The authors note that there is always a tension between flexibility and interpret ability of a model. Linear regression is quite inflexible, but it is readily interpret-able. Non-linear models can be quite complex in how an independent variable is functionally related to the dependent variable. In economics, we care about interpretability. We go through great pains to find instruments, design experiments, get random samples, in order to argue causality of one variable on another. Inference and interpretability matter a great deal to us, so inflexible models are often attractive for this purpose. Additionally, less flexible models often perform fairly well compared to more flexible models in terms of predictive accuracy, at least in out of sample testing, due to the problem of over-fitting in flexible models. So you get a lot of mileage out of an inflexible model like linear regression, and more flexible models might not perform much better on medium data anyway.

Our analysis is often naturally constrained in terms of the number of independent variables one might include. First, economic theory often suggests what things might be important independent variables, so there is less need to throw everything into a regression and see if it predicts. Or, you might be constrained by the scarcity of relevant data. If you conduct a survey, you can only ask so many questions and then you only have that many potential independent variables. To use a concept from my first machine learning post, we often have ‘medium data’ not big data.

I think there are good reasons why econometrics has gravitated to linear regression models as our bread and butter.

Econometrics:

• We care very much about inference, less emphasis on prediction \(\Rightarrow\) Linear regression often works well
• We often have medium data \(\Rightarrow\) Linear regression often works well.

So does that mean we should scoff at this statistical learning stuff? No I don’t think so. Reading into chapter 5 (Re-sampling Methods) and chapter 6 (Linear Model Selection and Regularization), I think there are really nice opportunities to get a lot of benefit even in a fairly inflexible model and even with medium data. Chapter 5 covers re-sampling methods including Cross-Validation and Bootstrap methods. Bootstrap is ubiquitous in econometrics, but cross-validation could be successfully utilized more, I think. Among those of us working with large ‘medium data’, like the Census or Ag Census, the methods are chapter 6 are fairly commonly employed, I think.

Statistical Learning:

• More focus on prediction \(\Rightarrow\) Larger baseline set of models to choose from. Just so long as it predicts, who cares about causality!
• Means much more effort exerted at the model selection stage.

Chapters 2 and 5 discuss the bias-variance trade-off in model selection and the k-Fold cross validation method of model selection, respectively. I briefly summarize these concepts next.

Bias-Variance Tradeoff

A key to mastering model assessment and model selection is to understand the bias-variance trade-off. If you have a model, \(y = f(x) + \epsilon\), so that \(y\) is a function of \(f\) plus some noise. Then, if we want to find out how \(x\) affects \(y\), or use known values of \(x\) to predict \(y\), we need to find a function \(\hat{f}(x)\) that is as close as possible to \(f\). The most common way to assess if our \(\hat{f}\) is good or not is with mean squared error (MSE), or root mean squared error where we take the square root of the MSE. MSE is defined like this.

\[MSE = \frac{1}{n}\sum_{i=1}^n (y_i – \hat{f}(x_i))^2 \]

It just takes all the model’s predictions, \(\hat{f}(x_i)\), calculates the square of how far they are from the actual value, \(y_i\), and averages them. So a low MSE is good – your predictions were not far away from the actual values.

So what contributes to a model’s errors? Head over to Wikipedia for a proof, but for any particular observation the expectation of the terms contributing to the MSE can be broken down as follows,

\[E(y_i – \hat{f}x_i)^2 = Var(\hat{f}(x_i)) + Bias(\hat{f}(x_i))^2 + Var(\epsilon_i),\]

where, \(Bias(\hat{f}(x_i)) = E(\hat{f}(x_i) – f(x_i))\). Clearly to get a low MSE you need a model that will give you low variance and low bias. But, typically these things move in opposite directions. You get a high variance in \(\hat{f}\) with a more flexible model because when a model is flexible (think for example of a high degree polynomial versus a linear model where the polynomial has many more parameters to estimate) different training data sets will make the fitted function ‘wiggle’ around much more than the linear model. Conversely, models that are less flexible have higher bias, because it can not accommodate the true shape of the data generating process as easily as a more flexible model.

In ISL, figure 2.11 illustrates this very well.

This figure is taken from “An Introduction to Statistical Learning, with applications in R” (Springer, 2013) with permission from the authors: G. James, D. Witten, T. Hastie and R. Tibshirani

Here you see a scatter-plot of a data set with variables X and Y as circles in the left panel. To model the data they fit a linear model (in gold), and increasingly higher degree polynomials in black, blue, and green. Of course, the higher the degree of the polynomial the more ‘wiggly’ the \(\hat{f}\). Then in the right panel, they plot the MSE of these different candidate \(\hat{f}\)’s. The training set MSE is in grey and the test set (out of sample in the parlance of price forecasting) MSE is in red.

This illustrates an important point. Yes, variance and bias of a model move in opposite directions, but not necessarily at the same rate (as flexibility increases I mean). Here, when you move from a linear, to a 3rd degree, to a 5th degree polynomial, you get dramatic reductions in the MSE of both the training set and the test set.

This is because the reduction in bias as you move away from the linear model overwhelms the increase in variance as you move toward a higher degree polynomial. Though, after about a 5th degree polynomial, the gains disappear and you do not get much more out of adding higher polynomial terms to the \(\hat{f}\). So, if this were a model selection exercise in a real research problem, we would probably select a 5th degree polynomial.

The approach of using MSE for model selection seems to be pretty ubiquitous in machine learning contexts. In economics, it is commonly used to select among competing forecasting models, but I don’t think it is used that widely for model selection in models where inference is the goal.

Cross Validation

Cross validation is a technique for model selection that I do not believe is widely used by economists, but I think it has a lot of promise. It does not seem to me to require a particularly large data set, so it could be an effective tool even on our ‘medium’ data sets.

The idea behind it is pretty simple. At the model selection stage, you will have a number of candidate models. Conceptually, the example above is pretty simple, your candidate models are polynomials of higher and higher degree. But, you could have a number of candidate models that look very different from one another. The MSE approach to model selection is to just see which one predicts the best on a ‘hold out’ data set. That is basically what we did in the example shown in figure 2.11.

Cross Validation just takes this one step further. In k-Fold cross validation you do the MSE exercise with k different randomly selected hold out samples for every model you are considering. An illustration for a 5-Fold cross validation sample split is shown in figure 5.5 from the ISL book.

This figure is taken from “An Introduction to Statistical Learning, with applications in R” (Springer, 2013) with permission from the authors: G. James, D. Witten, T. Hastie and R. Tibshirani

The advantage of this is that it allows the variance and bias of the fitted models to be revealed a little more completely. By using different subsets of your data as the test set you get a sense of how much individual observations are moving around your MSE estimates. When conducting a k-Fold cross validation, you typically just average the MSE’s given by each of the k model estimates.

An Ag Example

To illustrate the potential of k-Fold cross validation in agricultural economics I will use it in a simple example on a pretty small data set. Every month the USDA puts out the World Agricultural Supply and Demand Estimates (WASDE). This report is followed very closely, especially during the growing season and harvest of commodities important to U.S. agriculture. It publishes what are called ‘balance sheets’ because they state in a table expectations for supply and demand from various sources for each commodity.

For store-able commodities the WASDE also have an estimate of ending stocks, the amount of the grain that is not consumed in the same marketing year it was grown in, and thus carried over to be consumed during the next marketing year. Ending stocks are an important indicator or scarcity or surplus, and thus correlate with prices quite nicely. Rather than simply use ending stocks, a ratio called \(Stocks\) \(to\) \(Use\) has come to be important in price analysis because it allows ending stocks to be expressed in a way that is not influenced by the (potentially growing) size of the market.

\[Stocks \text{ } to \text{ } Use_t = \frac{Ending \text{ } Stocks_t}{Total \text{ } Use_t} \]

Stocks to Use is simply the ratio of ending stocks to the total amount of the commodity ‘used’.

# If you haven't installed any of the packages listed below, do so with the "install.packages('tibble')" command. 
library(tibble)
library(dplyr)
library(tidyr)
library(ggplot2)
library(knitr)
library(kableExtra)
library(gridExtra)
library(boot)
stocks  <- read.csv('images/stocks.csv')
stocks  <- as_tibble(stocks)
stocks  <- mutate(stocks, USStockUse = USEndingStocks/USTotalUse, WorldStockUse = ROWEndingStocks/WorldTotalUse)

stocks %>% kable(caption = "U.S. and World Ending Stocks, Total Use, Prices Recieved by Farmers, and Stocks to USE, 1975-2017", "html") %>% 
  kable_styling(bootstrap_options = c("striped", "hover"))

us      <- ggplot(stocks, aes(x = USStockUse, y = PriceRecievedFarmers)) + geom_point() + theme_bw() + xlim(0, .7)

ROW      <- ggplot(stocks, aes(x = WorldStockUse, y = PriceRecievedFarmers)) + geom_point() + theme_bw()  + xlim(0, .7)

grid.arrange(us, ROW, nrow = 1, top = "U.S. and World Stocks to Use vs Prices Recieved by U.S. Farmers, 1975-2017")

set.seed(17)
cv.us=rep(0,5)
cv.World=rep(0,5)
cv.Lus=rep(0,5)
cv.LWorld=rep(0,5)
cv.World2=rep(0,5)
for (i in 1:5){
    glm.fit=glm(PriceRecievedFarmers~poly(1/USStockUse, i) ,data=stocks)
    cv.us[i]=cv.glm(stocks,glm.fit,K=5)$delta[1]
}

for (i in 1:5){
  glm.fit.World=glm(PriceRecievedFarmers~poly(1/USStockUse, i) + poly(1/WorldStockUse, i) ,data=stocks)
  cv.World[i]=cv.glm(stocks,glm.fit.World,K=5)$delta[1]
}

for (i in 1:5){
    glm.fit=glm(PriceRecievedFarmers~poly(log(USStockUse), i) ,data=stocks)
    cv.Lus[i]=cv.glm(stocks,glm.fit,K=5)$delta[1]
}

for (i in 1:5){
  glm.fit.World=glm(PriceRecievedFarmers~poly(log(USStockUse), i) + poly(log(WorldStockUse), i),data=stocks)
  cv.LWorld[i]=cv.glm(stocks,glm.fit.World,K=5)$delta[1]
}

for (i in 1:5){
  glm.fit.World=glm(PriceRecievedFarmers~poly(1/USStockUse, i) + poly(log(WorldStockUse), i),data=stocks)
  cv.World2[i]=cv.glm(stocks,glm.fit.World,K=5)$delta[1]
}

res_tibble        <- cbind(cv.us, cv.World, cv.Lus, cv.LWorld, cv.World2) %>% as_tibble() %>% add_column(PolynomialDegree = seq(1:5)) %>% round(2)

colnames(res_tibble) <- c('Model 1', 'Model 2', 'Model 3', 'Model 4', 'Model 5', 'PolynomialDegree')

p1 <- ggplot(res_tibble, aes(x = PolynomialDegree, y = cv.us)) + geom_line() + ylim(0, 75) +  labs(x = 'Model 1 Poly Degree', y = 'Cross Validation MSE') + theme_bw()  
p2 <- ggplot(res_tibble, aes(x = PolynomialDegree, y = cv.World)) + geom_line() + ylim(0, 75) + labs(x = 'Model 2 Poly Degree', y = '') + theme_bw()
p3 <- ggplot(res_tibble, aes(x = PolynomialDegree, y = cv.Lus)) + geom_line() + ylim(0, 75) + labs(x = 'Model 3 Poly Degree', y = '') + theme_bw()
p4 <- ggplot(res_tibble, aes(x = PolynomialDegree, y = cv.LWorld)) + geom_line() + ylim(0, 75) + labs(x = 'Model 4 Poly Degree', y = '') + theme_bw()
p5 <- ggplot(res_tibble, aes(x = PolynomialDegree, y = cv.World2)) + geom_line() + ylim(0, 75) + labs(x = 'Model 5 Poly Degree', y = '') + theme_bw()
grid.arrange(p1, p2, p3, p4, p5, nrow = 1, top = "Cross Validation MSE for Competing Models")

That’s It!

That’s it for this time! I hope you enjoyed this explanation and toy example of cross validation. I certainly learned a lot writing it! I think you would only consider using cross validation for model selection if your data set was at least modestly bigger that the one in this post. However, this example goes to show that the technique can be useful even in medium data!

In the next post I will talk about Tree-Based Methods, Random Forests, Boosting, and Bagging from chapter 8 of ISL. I really currently know zero about these methods, but I have heard a lot of buzz about them. It is my sense that these are ‘real’ machine learning topics, whereas the cross validation and model selection stuff is more like necessary background.

Stay Tuned!