Lecture Overview

Lecture 10 — Statistical Inference & Model Validation

• Distributions of OLS estimators under normality
• \(t\)-tests for individual coefficients
• Confidence intervals for \(\beta_j\)
• The \(F\)-test: overall model significance
• The ANOVA table for regression
• Point prediction and prediction intervals
• Validating the classical assumptions:
  • Linearity, normality, homoscedasticity, independence, multicollinearity

From Estimates to Inference

We have \(\hat{\beta}_j\) — point estimates. But how uncertain are they?

Under the classical assumptions A1–A6:

\[\hat{\beta}_j \sim N\!\left(\beta_j,\; \sigma^2_{\hat{\beta}_j}\right)\]

The Individual Significance Test

Question: Is the variable \(X_j\) statistically significant — i.e. does it contribute to explaining \(Y\)?

Reading the R Output

summary(mlr_model)

Call:
lm(formula = profit2 ~ revenue2 + employees)

Residuals:
     Min       1Q   Median       3Q      Max 
-21.8947  -9.0708  -0.1219   6.0029  25.3186 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) 21.46125    9.51476   2.256   0.0324 *  
revenue2     0.12902    0.02048   6.299 9.66e-07 ***
employees    0.61262    0.11789   5.196 1.80e-05 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 12.43 on 27 degrees of freedom
Multiple R-squared:  0.6543,    Adjusted R-squared:  0.6287 
F-statistic: 25.55 on 2 and 27 DF,  p-value: 5.923e-07

Interpreting the \(t\)-test Output

From the Coefficients table in summary():

\(F\)-Test in R

The \(F\)-statistic and its \(p\)-value appear at the bottom of summary():

summary(mlr_model)

Call:
lm(formula = profit2 ~ revenue2 + employees)

Residuals:
     Min       1Q   Median       3Q      Max 
-21.8947  -9.0708  -0.1219   6.0029  25.3186 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) 21.46125    9.51476   2.256   0.0324 *  
revenue2     0.12902    0.02048   6.299 9.66e-07 ***
employees    0.61262    0.11789   5.196 1.80e-05 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 12.43 on 27 degrees of freedom
Multiple R-squared:  0.6543,    Adjusted R-squared:  0.6287 
F-statistic: 25.55 on 2 and 27 DF,  p-value: 5.923e-07

Relationship Between \(F\) and \(R^2\)

In MLR, the \(F\)-statistic can be written as:

\[F = \frac{R^2/k}{(1-R^2)/(n-k-1)}\]

Prediction Intervals

A prediction interval accounts for two sources of uncertainty:

1. Uncertainty about the true mean \(E[Y^*]\) (same as a confidence interval for the mean)
2. Uncertainty about the individual outcome \(Y^*\) around that mean

predict(mlr_model, newdata = new_obs,
        interval = "prediction", level = 0.95)

       fit      lwr      upr
1 84.67332 58.73202 110.6146

Why Validation Matters

Our inference (t-tests, F-test, confidence intervals) is only valid if the classical assumptions hold.

If they fail:

• A2 violated (non-zero mean error, omitted variables): OLS is biased
• A3 violated (heteroscedasticity): OLS standard errors are wrong → invalid \(t\) and \(F\) tests
• A4 violated (autocorrelation): same problem as heteroscedasticity
• A6 violated (non-normality): \(t\) and \(F\) approximations may fail in small samples
• Multicollinearity (not a classical assumption violation): inflated standard errors, unstable estimates

Putting It All Together: Model Validation Workflow

1. Fit the model with lm() and check summary() for coefficient signs, magnitudes and significance.
2. Plot residuals vs. fitted — check for linearity and homoscedasticity.
3. Inspect the Q–Q plot — check for normality of residuals.
4. Run Durbin-Watson (if data may have time ordering) — check for autocorrelation.
5. Compute VIF (if \(k \geq 2\)) — check for multicollinearity.
6. If violations are found: consider transformations (log \(Y\), log \(X\)), robust standard errors, or model re-specification (add/remove variables).

End-to-End: From Data to Validated Model

# 1. Fit the model
model <- lm(Y ~ X1 + X2, data = mydata)

# 2. Inspect results
summary(model)
confint(model, level = 0.95)

# 3. Residual diagnostics
par(mfrow = c(2, 2))
plot(model)

# 4. Autocorrelation (if needed)
library(lmtest)
dwtest(model)

# 5. Multicollinearity (if k >= 2)
library(car)
vif(model)

# 6. Predict for new data
predict(model,
        newdata = data.frame(X1 = x1_new, X2 = x2_new),
        interval = "prediction",
        level = 0.95)

Reading a Regression Table

A typical regression output table (as seen in R or SPSS):

Key Formulas at a Glance