Machine Learning

1. Simple Linear Regression (SLR)

Models the relationship between exactly one independent variable and one dependent variable using a straight line.

• y: Dependent variable (target to predict)
• x: Independent variable (feature)
• β₀: Y-intercept (value of y when x = 0)
• β₁: Slope (how much y changes for a 1-unit change in x)
• ε: Error term (residuals)

2. Multiple Linear Regression (MLR)

Extends simple linear regression to model the relationship between two or more independent variables and a single dependent variable.

Matrix Representation & OLS Estimation

For m data points with n features, we define Y = Xβ + ε. The Ordinary Least Squares (OLS) method minimizes the Sum of Squared Residuals to find the optimal coefficients:

5 Key Assumptions (LINED)

• Linearity: Relationship between x and y is linear.
• Independence: Observations are independent of each other.
• Normality: Residuals (errors) are normally distributed.
• Equal Variance (Homoscedasticity): Residuals have constant variance.
• No Dependency (No Multicollinearity): Independent variables are not highly correlated.

3. Polynomial Regression

Linear regression fails when the true relationship is non-linear (curved). Polynomial regression extends MLR by adding polynomial (higher-power) features to fit these non-linear patterns.

Key Insight: Although the curve is non-linear with respect to x, the model is still considered linear in its parameters (β). Therefore, we can substitute z₁=x, z₂=x² and solve it exactly like Multiple Linear Regression!

The Overfitting Danger

Choosing the correct polynomial degree is a classic Bias-Variance Tradeoff:

• Low degree (e.g., n=1): High Bias (Underfitting) - model is too simple.
• Optimal degree: Captures the true underlying pattern.
• High degree (e.g., n=15): High Variance (Overfitting) - model fits the noise perfectly but fails on new data.

Solutions to Overfitting: Use Regularization (Ridge/LASSO) or Cross-Validation.