Unsupervised Learning Reference

Overview

Unsupervised learning discovers patterns in unlabeled data through clustering, dimensionality reduction, and density estimation.

Clustering

K-Means

KMeans (sklearn.cluster.KMeans)

Partition-based clustering into K clusters
Key parameters:
- n_clusters: Number of clusters to form
- init: Initialization method ('k-means++', 'random')
- n_init: Number of initializations (default=10)
- max_iter: Maximum iterations
Use when: Know number of clusters, spherical cluster shapes
Fast and scalable
Example:

python

from sklearn.cluster import KMeans

model = KMeans(n_clusters=3, init='k-means++', n_init=10, random_state=42)
labels = model.fit_predict(X)
centers = model.cluster_centers_

# Inertia (sum of squared distances to nearest center)
print(f"Inertia: {model.inertia_}")

MiniBatchKMeans

Faster K-Means using mini-batches
Use when: Large datasets, need faster training
Slightly less accurate than K-Means
Example:

python

from sklearn.cluster import MiniBatchKMeans

model = MiniBatchKMeans(n_clusters=3, batch_size=100, random_state=42)
labels = model.fit_predict(X)

Density-Based Clustering

DBSCAN (sklearn.cluster.DBSCAN)

Density-Based Spatial Clustering
Key parameters:
- eps: Maximum distance between two samples to be neighbors
- min_samples: Minimum samples in neighborhood to form core point
- metric: Distance metric
Use when: Arbitrary cluster shapes, presence of noise/outliers
Automatically determines number of clusters
Labels noise points as -1
Example:

python

from sklearn.cluster import DBSCAN

model = DBSCAN(eps=0.5, min_samples=5, metric='euclidean')
labels = model.fit_predict(X)

# Number of clusters (excluding noise)
n_clusters = len(set(labels)) - (1 if -1 in labels else 0)
n_noise = list(labels).count(-1)
print(f"Clusters: {n_clusters}, Noise points: {n_noise}")

HDBSCAN (sklearn.cluster.HDBSCAN)

Hierarchical DBSCAN with adaptive epsilon
More robust than DBSCAN
Key parameter: min_cluster_size
Use when: Varying density clusters
Example:

python

from sklearn.cluster import HDBSCAN

model = HDBSCAN(min_cluster_size=10, min_samples=5)
labels = model.fit_predict(X)

OPTICS (sklearn.cluster.OPTICS)

Ordering points to identify clustering structure
Similar to DBSCAN but doesn't require eps parameter
Key parameters: min_samples, max_eps
Use when: Varying density, exploratory analysis
Example:

python

from sklearn.cluster import OPTICS

model = OPTICS(min_samples=5, max_eps=0.5)
labels = model.fit_predict(X)

Hierarchical Clustering

AgglomerativeClustering

Bottom-up hierarchical clustering
Key parameters:
- n_clusters: Number of clusters (or use distance_threshold)
- linkage: 'ward', 'complete', 'average', 'single'
- metric: Distance metric
Use when: Need dendrogram, hierarchical structure important
Example:

python

from sklearn.cluster import AgglomerativeClustering

model = AgglomerativeClustering(n_clusters=3, linkage='ward')
labels = model.fit_predict(X)

# Create dendrogram using scipy
from scipy.cluster.hierarchy import dendrogram, linkage
Z = linkage(X, method='ward')
dendrogram(Z)

Other Clustering Methods

MeanShift

Finds clusters by shifting points toward mode of density
Automatically determines number of clusters
Key parameter: bandwidth
Use when: Don't know number of clusters, arbitrary shapes
Example:

python

from sklearn.cluster import MeanShift, estimate_bandwidth

# Estimate bandwidth
bandwidth = estimate_bandwidth(X, quantile=0.2, n_samples=500)
model = MeanShift(bandwidth=bandwidth)
labels = model.fit_predict(X)

SpectralClustering

Uses graph-based approach with eigenvalues
Key parameters: n_clusters, affinity ('rbf', 'nearest_neighbors')
Use when: Non-convex clusters, graph structure
Example:

python

from sklearn.cluster import SpectralClustering

model = SpectralClustering(n_clusters=3, affinity='rbf', random_state=42)
labels = model.fit_predict(X)

AffinityPropagation

Finds exemplars by message passing
Automatically determines number of clusters
Key parameters: damping, preference
Use when: Don't know number of clusters
Example:

python

from sklearn.cluster import AffinityPropagation

model = AffinityPropagation(damping=0.9, random_state=42)
labels = model.fit_predict(X)
n_clusters = len(model.cluster_centers_indices_)

BIRCH

Balanced Iterative Reducing and Clustering using Hierarchies
Memory efficient for large datasets
Key parameters: n_clusters, threshold, branching_factor
Use when: Very large datasets
Example:

python

from sklearn.cluster import Birch

model = Birch(n_clusters=3, threshold=0.5)
labels = model.fit_predict(X)

Clustering Evaluation

Metrics when ground truth is known:

python

from sklearn.metrics import adjusted_rand_score, normalized_mutual_info_score
from sklearn.metrics import adjusted_mutual_info_score, fowlkes_mallows_score

# Compare predicted labels with true labels
ari = adjusted_rand_score(y_true, y_pred)
nmi = normalized_mutual_info_score(y_true, y_pred)
ami = adjusted_mutual_info_score(y_true, y_pred)
fmi = fowlkes_mallows_score(y_true, y_pred)

Metrics without ground truth:

python

from sklearn.metrics import silhouette_score, calinski_harabasz_score
from sklearn.metrics import davies_bouldin_score

# Silhouette: [-1, 1], higher is better
silhouette = silhouette_score(X, labels)

# Calinski-Harabasz: higher is better
ch_score = calinski_harabasz_score(X, labels)

# Davies-Bouldin: lower is better
db_score = davies_bouldin_score(X, labels)

Elbow method for K-Means:

python

from sklearn.cluster import KMeans
import matplotlib.pyplot as plt

inertias = []
K_range = range(2, 11)
for k in K_range:
    model = KMeans(n_clusters=k, random_state=42)
    model.fit(X)
    inertias.append(model.inertia_)

plt.plot(K_range, inertias, 'bo-')
plt.xlabel('Number of clusters')
plt.ylabel('Inertia')
plt.title('Elbow Method')

Dimensionality Reduction

Principal Component Analysis (PCA)

PCA (sklearn.decomposition.PCA)

Linear dimensionality reduction using SVD
Key parameters:
- n_components: Number of components (int or float for explained variance)
- whiten: Whiten components to unit variance
Use when: Linear relationships, want to explain variance
Example:

python

from sklearn.decomposition import PCA

# Keep components explaining 95% variance
pca = PCA(n_components=0.95)
X_reduced = pca.fit_transform(X)

print(f"Original dimensions: {X.shape[1]}")
print(f"Reduced dimensions: {X_reduced.shape[1]}")
print(f"Explained variance ratio: {pca.explained_variance_ratio_}")
print(f"Total variance explained: {pca.explained_variance_ratio_.sum()}")

# Or specify exact number of components
pca = PCA(n_components=2)
X_2d = pca.fit_transform(X)

IncrementalPCA

PCA for large datasets that don't fit in memory
Processes data in batches
Key parameter: n_components, batch_size
Example:

python

from sklearn.decomposition import IncrementalPCA

pca = IncrementalPCA(n_components=50, batch_size=100)
X_reduced = pca.fit_transform(X)

KernelPCA

Non-linear dimensionality reduction using kernels
Key parameters: n_components, kernel ('linear', 'poly', 'rbf', 'sigmoid')
Use when: Non-linear relationships
Example:

python

from sklearn.decomposition import KernelPCA

pca = KernelPCA(n_components=2, kernel='rbf', gamma=0.1)
X_reduced = pca.fit_transform(X)

Manifold Learning

t-SNE (sklearn.manifold.TSNE)

t-distributed Stochastic Neighbor Embedding
Excellent for 2D/3D visualization
Key parameters:
- n_components: Usually 2 or 3
- perplexity: Balance between local and global structure (5-50)
- learning_rate: Usually 10-1000
- n_iter: Number of iterations (min 250)
Use when: Visualizing high-dimensional data
Note: Slow on large datasets, no transform() method
Example:

python

from sklearn.manifold import TSNE

tsne = TSNE(n_components=2, perplexity=30, learning_rate=200, n_iter=1000, random_state=42)
X_embedded = tsne.fit_transform(X)

# Visualize
import matplotlib.pyplot as plt
plt.scatter(X_embedded[:, 0], X_embedded[:, 1], c=labels, cmap='viridis')
plt.title('t-SNE visualization')

UMAP (not in scikit-learn, but compatible)

Uniform Manifold Approximation and Projection
Faster than t-SNE, preserves global structure better
Install: uv pip install umap-learn
Example:

python

from umap import UMAP

reducer = UMAP(n_components=2, n_neighbors=15, min_dist=0.1, random_state=42)
X_embedded = reducer.fit_transform(X)

Isomap

Isometric Mapping
Preserves geodesic distances
Key parameters: n_components, n_neighbors
Use when: Non-linear manifolds
Example:

python

from sklearn.manifold import Isomap

isomap = Isomap(n_components=2, n_neighbors=5)
X_embedded = isomap.fit_transform(X)

Locally Linear Embedding (LLE)

Preserves local neighborhood structure
Key parameters: n_components, n_neighbors
Example:

python

from sklearn.manifold import LocallyLinearEmbedding

lle = LocallyLinearEmbedding(n_components=2, n_neighbors=10)
X_embedded = lle.fit_transform(X)

MDS (Multidimensional Scaling)

Preserves pairwise distances
Key parameter: n_components, metric (True/False)
Example:

python

from sklearn.manifold import MDS

mds = MDS(n_components=2, metric=True, random_state=42)
X_embedded = mds.fit_transform(X)

Matrix Factorization

NMF (Non-negative Matrix Factorization)

Factorizes into non-negative matrices
Key parameters: n_components, init ('nndsvd', 'random')
Use when: Data is non-negative (images, text)
Interpretable components
Example:

python

from sklearn.decomposition import NMF

nmf = NMF(n_components=10, init='nndsvd', random_state=42)
W = nmf.fit_transform(X)  # Document-topic matrix
H = nmf.components_  # Topic-word matrix

TruncatedSVD

SVD for sparse matrices
Similar to PCA but works with sparse data
Use when: Text data, sparse matrices
Example:

python

from sklearn.decomposition import TruncatedSVD

svd = TruncatedSVD(n_components=100, random_state=42)
X_reduced = svd.fit_transform(X_sparse)
print(f"Explained variance: {svd.explained_variance_ratio_.sum()}")

FastICA

Independent Component Analysis
Separates multivariate signal into independent components
Key parameter: n_components
Use when: Signal separation (e.g., audio, EEG)
Example:

python

from sklearn.decomposition import FastICA

ica = FastICA(n_components=10, random_state=42)
S = ica.fit_transform(X)  # Independent sources
A = ica.mixing_  # Mixing matrix

LatentDirichletAllocation (LDA)

Topic modeling for text data
Key parameters: n_components (number of topics), learning_method ('batch', 'online')
Use when: Topic modeling, document clustering
Example:

python

from sklearn.decomposition import LatentDirichletAllocation

lda = LatentDirichletAllocation(n_components=10, random_state=42)
doc_topics = lda.fit_transform(X_counts)  # Document-topic distribution

# Get top words for each topic
feature_names = vectorizer.get_feature_names_out()
for topic_idx, topic in enumerate(lda.components_):
    top_words = [feature_names[i] for i in topic.argsort()[-10:]]
    print(f"Topic {topic_idx}: {', '.join(top_words)}")

Outlier and Novelty Detection

Outlier Detection

IsolationForest

Isolates anomalies using random trees
Key parameters:
- contamination: Expected proportion of outliers
- n_estimators: Number of trees
Use when: High-dimensional data, efficiency important
Example:

python

from sklearn.ensemble import IsolationForest

model = IsolationForest(contamination=0.1, random_state=42)
predictions = model.fit_predict(X)  # -1 for outliers, 1 for inliers

LocalOutlierFactor

Measures local density deviation
Key parameters: n_neighbors, contamination
Use when: Varying density regions
Example:

python

from sklearn.neighbors import LocalOutlierFactor

lof = LocalOutlierFactor(n_neighbors=20, contamination=0.1)
predictions = lof.fit_predict(X)  # -1 for outliers, 1 for inliers
outlier_scores = lof.negative_outlier_factor_

One-Class SVM

Learns decision boundary around normal data
Key parameters: nu (upper bound on outliers), kernel, gamma
Use when: Small training set of normal data
Example:

python

from sklearn.svm import OneClassSVM

model = OneClassSVM(nu=0.1, kernel='rbf', gamma='auto')
model.fit(X_train)
predictions = model.predict(X_test)  # -1 for outliers, 1 for inliers

EllipticEnvelope

Assumes Gaussian distribution
Key parameter: contamination
Use when: Data is Gaussian-distributed
Example:

python

from sklearn.covariance import EllipticEnvelope

model = EllipticEnvelope(contamination=0.1, random_state=42)
predictions = model.fit_predict(X)

Gaussian Mixture Models

GaussianMixture

Probabilistic clustering with mixture of Gaussians
Key parameters:
- n_components: Number of mixture components
- covariance_type: 'full', 'tied', 'diag', 'spherical'
Use when: Soft clustering, need probability estimates
Example:

python

from sklearn.mixture import GaussianMixture

gmm = GaussianMixture(n_components=3, covariance_type='full', random_state=42)
gmm.fit(X)

# Predict cluster labels
labels = gmm.predict(X)

# Get probability of each cluster
probabilities = gmm.predict_proba(X)

# Information criteria for model selection
print(f"BIC: {gmm.bic(X)}")  # Lower is better
print(f"AIC: {gmm.aic(X)}")  # Lower is better

Choosing the Right Method

Clustering:

Know K, spherical clusters: K-Means
Arbitrary shapes, noise: DBSCAN, HDBSCAN
Hierarchical structure: AgglomerativeClustering
Very large data: MiniBatchKMeans, BIRCH
Probabilistic: GaussianMixture

Dimensionality Reduction:

Linear, variance explanation: PCA
Non-linear, visualization: t-SNE, UMAP
Non-negative data: NMF
Sparse data: TruncatedSVD
Topic modeling: LatentDirichletAllocation

Outlier Detection:

High-dimensional: IsolationForest
Varying density: LocalOutlierFactor
Gaussian data: EllipticEnvelope