Principal Component Analysis (PCA)

This note introduces the Principal Component Analysis (PCA) technique using scikit-learn, explains the step-by-step logic behind how it works, and then demonstrates a from-scratch implementation to show that the core idea is simple and intuitive to build.

What is PCA?

Principal Component Analysis is like finding the best view of your data.

It rotates and reorients the data to uncover the most important directions — the ones that capture the most variation — and lets you reduce dimensions while keeping as much information as possible.

PCA is widely used to:

• Compress data by reducing the number of features
• Visualize high-dimensional data in 2D or 3D
• Remove noise and redundancy before applying other machine learning algorithms

This notebook will:

• Use scikit-learn to demonstrate how PCA transforms and reduces data
• Explain the math and intuition behind the process
• Show how to implement PCA step by step from scratch using NumPy

Let’s dive in to understand how PCA works and why it’s so useful.

📦 Preparation

• Load the Wine dataset using sklearn.datasets.load_wine
• Standardize the features for fair comparison
• Visualize the raw data using a scatterplot matrix

Outputs:

👀 Data Observation

• The dataset contains 13 features, which makes it hard to interpret visually
• Many features are correlated or redundant, as seen in the scatterplot matrix and correlation analysis
• This motivates the use of dimensionality reduction techniques like PCA

⚙️ PCA with scikit-learn

• Applied PCA using sklearn.decomposition.PCA on the standardized data
• The first two principal components (PC1 and PC2) capture most of the variation in the data
• PCA successfully compresses the original 13 features into fewer uncorrelated components, while preserving structure
• The PCA scatterplot matrix shows that PC1 and PC2 reveal clear class separation, similar to the raw space but in a simpler, decorrelated form

Outputs:

🧠 Behind the Scenes: How PCA Works (Step-by-Step Math)

This section explains how PCA works under the hood, using clear, beginner-friendly math and simple reasoning.

1. 🎯 What Is the Goal of PCA?

PCA finds new axes (directions) in your dataset such that:

• The first axis (PC1) captures the most variation in the data
• The second axis (PC2) captures the next most variation, and so on
• These axes are uncorrelated and orthogonal (at 90° angles)
• We can reduce the number of dimensions while keeping most of the original information

2. 🔢 Centering the Data

We begin by centering the dataset — subtracting the mean of each feature (column):

$$X_{\text{centered}} = X - \bar{X}$$

PCA assumes the data is centered around the origin. If we don’t subtract the mean, the direction of maximum variance may be biased.

3. 🧮 Covariance Matrix

To measure how features relate to each other, we compute the covariance matrix:

$$\text{Cov}(X) = \frac{1}{n - 1} X_{\text{centered}}^T X_{\text{centered}}$$

Let’s break this down:

• $X$ is your centered data (shape: $n \times d$, where $n$ = number of samples, $d$ = number of features)
• $X^T$ is the transpose of $X$: it flips rows into columns
• $X^T X$ becomes a square matrix of size $d \times d$, showing how each feature relates to others
  • Diagonal = variance of each feature
  • Off-diagonal = how two features change together

💡 The covariance matrix is symmetric and contains all the pairwise relationships between features.

4. 🔍 Finding the Principal Axes: Eigenvectors & Eigenvalues

Now comes the key step: we compute the eigenvectors and eigenvalues of the covariance matrix.

What are they?

• An eigenvector is a direction that remains unchanged (except for scale) when a transformation is applied.
• An eigenvalue tells how much the data is stretched along that direction.

We solve the following equation:

$$A \cdot v = \lambda \cdot v$$

Where:

• $A$ is the covariance matrix
• $v$ is the eigenvector
• $\lambda$ is the eigenvalue

This is called an eigen decomposition.

How to compute eigenvalues and eigenvectors

Step 1: Rearranged equation:

$$(A - \lambda I) \cdot v = 0$$

Step 2: To find valid $\lambda$ values, solve this characteristic equation:

$$\det(A - \lambda I) = 0$$

This gives a polynomial in $\lambda$, and solving it gives all the eigenvalues.

Step 3: For each eigenvalue $\lambda$, plug it back into:

$$(A - \lambda I) \cdot v = 0$$

Solving this gives the corresponding eigenvector $v$.

✅ In practice, libraries like NumPy use efficient numerical algorithms to do this for large matrices.

5. 📊 Sorting the Components

Once we have all eigenvectors and eigenvalues:

• We sort the eigenvectors based on their eigenvalues, from largest to smallest.
• Each eigenvalue tells us how much variance is captured by its corresponding eigenvector.
• We can keep all components, or choose the top $k$ components based on how much variance we want to retain.

You might hear the term explained variance ratio — it’s just the proportion of variance captured by each component:

$$\text{Explained Variance Ratio}_k = \frac{\lambda_k}{\sum \lambda_i}$$

This helps us choose how many components are worth keeping.

6. 🔄 Projecting the Data

Now that we’ve chosen the top $k$ eigenvectors, we project the data onto these directions:

$$X_{\text{projected}} = X_{\text{centered}} \cdot W$$

Where:

• $W$ is a matrix whose columns are the top $k$ eigenvectors
  • If we keep 2 components, $W$ has shape $d \times 2$
• $X_{\text{projected}}$ is your data in the new $k$-dimensional space

💡 This is where dimensionality reduction happens — the data is now compressed while keeping most of its structure.

Next, let’s implement PCA from scratch in NumPy and compare it with sklearn.PCA.

Outputs:

It Works!!

Our scratch implementation of PCA successfully extracts the principal components from the standardized Wine dataset.

• The top 5 principal components explain over 80% of the total variance, confirming that most of the dataset's structure is captured in just a few directions.
• The PC1 vs PC2 scatter plot (shown above) reveals clear clustering of the three classes — closely matching the top-left corner of the PCA scatterplot matrix generated using sklearn.

This confirms that:

• The covariance computation, eigen decomposition, and projection steps behave exactly as expected.
• Our scratch PCA accurately replicates the key behavior of sklearn.PCA.

We've successfully built a PCA implementation from the ground up!