Machine Learning
Supervised Learning: Regression / PCA: Deep Dive
PCA: Deep Dive
Principal Component Analysis (PCA) is a critical technique for reducing dimensions while preserving data variance. It identifies the directions (principal components) of maximum variance in the data.
Step-by-Step Algorithm
- Standardization: Scale the data so each feature has mean=0 and variance=1. This ensures that features with larger ranges don't dominate the variance.
- Covariance Matrix: Calculate how each feature varies with every other feature.
- Eigen-Decomposition: Compute Eigenvalues (magnitude of variance) and Eigenvectors (directions of variance).
- Sorting: Rank the eigenvectors by their corresponding eigenvalues in descending order. The first eigenvector is the "Principal Component 1" (PC1).
- Projection: Multiply the original data by the top k eigenvectors to transform it into the new k-dimensional space.
The Scree Plot & Cumulative Variance
How many components should we keep? We use a Scree Plot, which plots eigenvalues (y-axis) against component numbers (x-axis).
We look for the "elbow" in the plot where the variance drop levels off. Typically, we aim to retain enough components to explain 90-95% of the total variance.
Below is a simplified worked example of how PCA reduces 2D data to 1D.
Mathematical Example: 2D to 1D
Original Data (Height, Weight): [170, 65], [175, 70], [160, 55], [180, 80], [165, 60]
- Standardize (Mean Center): Height mean = 170, Weight mean = 66. Centered data: [0,−1], [5,4], [−10,−11], [10,14], [−5,−6].
- Covariance Matrix: Cov(H,H)=62.5, Cov(W,W)=107.5, Cov(H,W)=81.25.
- Eigenvalues: λ1=161.8 (PC1), λ2=8.2 (PC2).
- Variance Captured: PC1 captures 161.8 / (161.8 + 8.2) = 95.2% variance!
Conclusion: We can reduce from 2D to 1D and still retain 95.2% of the information.
Ready to test your PCA: Deep Dive knowledge?
PCA: Deep Dive
Deepen your understanding of Principal Component Analysis, including Eigen-decomposition, covariance matrices, and variance retention.