Deep Feature Clustering

Clustering Deep Features with ART Algorithms

What if you wanted to do something more sophisticated with ART?

Clustering on the Iris dataset (or even doing a supervised train/test split on it) is all well and good, but what if we tried to tackle higher-dimensional datasets?

Prelude

This notebook has more background text to introduce the reader to some relevant research questions and motivations. If you want to get straight to the code, jump to the Method section!

The Problem

ART algorithms are fundamentally prototype-based methods, and the number of those prototypes grows over the course of training. This means that the time complexity of ART algorithms actually varies over time. Translation: if you start getting too many prototypes, then the match rule search (and subsequently the number of activation/match function evaluations) takes an increasingly long time, slowing down your algorithm! In the ART literature, this is known as the problem of category proliferation.

Category Proliferation

One of the central challenges with using ART algorithms in real applications is reigning in this problem of category proliferation, which shows up in largely two ways:

1. If you have a big dataset (i.e., large sample count, streaming data, etc.), you are increasingly likely to accrue more and more prototypes in your ART algorithm over time. This is mainly a statistical statement about the diversity of samples in the sample state space increasing over time, which requires more prototypes to represent.
2. If you have a high-dimensional dataset, like anything to do with computer vision (e.g., photo and video), then the feature space is also increasingly likely to be sparse, which is another statistical statement that individual samples are not likely to be very near one another in the input feature space. Because high-dimensional input samples are more likely to be far apart from one another, it is likely that you will need increasingly more prototypes to represent them in that same feature space.

In both of these scenarios, we have a bit of an accuracy-speed tradeoff: we can set the vigilance parameter low and have get fewer prototypes, increasing our speed but decreasing our accuracy, or we can set the vigilance parameter high and get more prototypes, increasing our accuracy at the cost of speed.

But what if there were another way?

Deep Learning and the Manifold Hypothesis

One of the central tenets of the study of machine learning is the manifold hypothesis, which posits that most high-dimensional data can be represented along a lower-dimensional latent space manifold. This is one of the things that we explicitly look for in things like the latent variable space of variational auto encoders (VAEs), and it is suggested as one of the reasons that deep learning works at all!

Contrast that with Cover’s theorem, which states that casting a dataset into a higher dimensionality increases the likelihood that the transformed features are linearly separable.

Down projections in deep neural networks (i.e., going from layers with more neurons into layers with fewer neurons) forces the previous layer’s features into a smaller “manifold” space, while up projections increase the likelihood of their separability for downstream layers. By doing a series of nonlinear transformations, layer-by-layer, and with repeated up- and down-projections, deep neural networks are able given the machinery to learn useful transformations on the original dataset to achieve the performance that they do on high-dimensional datasets.

Where ART Steps In

However!

Show me a deep neural network that elegantly solves the stability-plasticity dilemma, and I will eat my own shoe.

Continual learning, also known as lifelong learning, involves introducing new classes of data over time to a model (e.g., train on cats and dogs, and then later introduce chickens), or even “shifting” the distributions of previous data classes (e.g., maybe new breeds of dogs appear in the dataset, but they are still technically dogs).

The same large, fixed architectures that benefit deep learning algorithms also cause them to be fragile to data distribution shifts in continual learning scenarios, and all continual learning frameworks in deep learning are workarounds of this problem. The stability-plasticity problem is analogous to the generalization-specialization problem; as a loose analogy, deep neural networks are examples of good generalizers, while ART algorithms are good specifiers. Deep learning seeks to find, at each layer, a non-linear transform that works as a good shared basis for downstream layers with respect to the entirety of the training dataset; that’s what we are training them to do via backpropagation! However, their representational capacity is necessarily fixed by virtue their fixed architectures, and they are fragile to tweaking their parameters no matter what fancy technique you use (e.g., replay methods, weight regularization etc.); push continual learning scenario limits, and they ultimately start failing at old tasks. ART algorithms, on the other hand, can provide a better guarantee of performance on previously-seen data, but they aren’t capable of doing the nonlinear feature learning that makes deep learning so effective.

What if we tried to combine the two?

That is the question at the center of a lot of ART research right now! One first-pass and relatively straightforward method of doing so is clustering on some middle features of a pre-trained deep neural network.

But wait, what is transfer learning?

Transfer Learning

Transfer learning is a relatively simple idea that works surprisingly well in practical deep learning applications. Consider that you have a small high-dimensional dataset that you would like to use deep learning on, the quintessential example of which is any kind of medical imagery. To train a deep neural network, we know that we need a lot of data to mitigate over-fitting; does that mean that we are excluded from modeling medical imagery with deep learning? No! One workaround comes from the intuition of how we interpret deep models: we often say that lower layers, closer to the input, perform transformations with low levels of abstraction (e.g., detecting corners and edges in images), while higher layers, closer to the output, perform more abstract transformations (e.g., distinguishing between different breeds of dogs). The middle layers? Well, they do some kind of “in between” magic that accumulates evidence relevant to whatever each subsequent layer is trying to do; again, this is more or less what we are doing when training with backpropagation, which is essentially penalizing and updating a layer based upon how it contributes to the downstream layers’ errors with respect to the data.

So, in a sense, once you train a big fancy image model on a big set of data (such as the ImageNet dataset, which contains over 14 million images), you get a model whose middle layers make a kind of embedding relevant for classifying images. What’s more, the higher layers tend to specialize more in the specific task that they are trained on, such as classifying ImageNet images, whereas lower layers learn feature transformations that are more generally applicable to all image processing.

Back to the medical imagery example, we get an idea: what if we take the backbone of a large pre-trained model, chop off the top of the network up to some layer we select, freeze those weights, and add a new “head” to the network that can fine tune to our specific dataset? That way, the data serves to train a much smaller set of weights that actually specialize on the task at hand, while the backbone does the “image processing black magic” learned by the larger model and larger dataset.

This is transfer learning.

There are a lot of different techniques with various bells and whistles for of how to do transfer learning better and more robustly, but it all essentially follows this paradigm.

Deep Feature Clustering

The last step is to ask, “where can ART fit in?” Well, consider instead of a set of feed-forward neural network layers on top of our transfer learning framework, we use an ART module instead? This will be our strategy in this notebook, which we will call “deep feature clustering”. We pick this clunky name because “deep clustering” is already a term in the deep learning literature that means something totally different.

In essence, our strategy will be to take a pre-trained deep network, trained on some computer vision task, chop off the head, freeze the weights of the body, and cluster the features with an ART module. This will require the following steps:

1. Downloading some data (MNIST handwritten digits, for example).
2. Downloading a pre-trained network definition and its weights.
3. Freezing the weights and putting it the network into evaluation (in case it uses dropout layers, for example).
4. Removing the top layers, up to some layer of our choosing.
5. Preprocessing the features of the network up to that layer via normalization by analyzing their statistics given our new input (the MNIST digits dataset).
6. Training and testing an ART module on top of those features, and seeing how we do!

Load Data

First, lets load the data like we do in other notebooks. We’ll use the MNIST handwritten digits dataset here for its ubiquity as a benchmark higher dimensional computer vision dataset.

# Import dependencies
import torch
import torchvision.transforms as transforms
from torchvision import datasets
from torch.utils.data import DataLoader
from pathlib import Path

# Init the random seed
torch.random.manual_seed(1234)
n_samples = 1000

# Define a function for loading the dataset
def get_mnist(
    train: bool = True,
    n_samples: int = 1000,
) -> DataLoader:
    # PyTorch API for combining multiple data transformations steps
    transform = transforms.Compose([
        transforms.ToTensor(),                      # Tensorize
        transforms.Lambda(lambda x: x.view(-1)),    # Flatten
    ])

    # Download and load the training data
    data = datasets.MNIST(
        root=Path('data'),
        train=train,
        download=True,
        transform=transform,
    )
    # Wrap the dataset in a DataLoader iterator so that the transform runs
    data_loader = DataLoader(data, batch_size=n_samples, shuffle=True)
    # Return the data loader
    return data_loader

# Get the train and test loaders
train_loader = get_mnist(True, n_samples)
test_loader = get_mnist(False, n_samples)
print(f"Train: {type(train_loader)}\n Test: {type(test_loader)}")

Train: <class 'torch.utils.data.dataloader.DataLoader'>
 Test: <class 'torch.utils.data.dataloader.DataLoader'>

Feature Extractor

Next, we get our feature extractor!

TODO

This notebook is a work in progress! If you see this, it means that there is more to come for this notebook.

Page built at: 2026-07-31 22:51:44 +0000