FuzzyART, Step-by-Step
FuzzyART in Parts
In the other FuzzyART notebook, we implemented a FuzzyART module as a class with all of the requisite methods for running it as a standalone module.
Here, we will look a little further down the rabbit hole to understand the moving parts of FuzzyART in finer detail.
In programming terms, this might look more like the “functional programming” paradigm/pattern in that we wish to atomically look at each moving part.
Dependencies
First, we load all of our dependencies for the notebook
# From scikit-learn, for casting the data to 2D for visualization.
# This is not how the data actually looks in 4D, but the best that we can do is to cast it to 2D such that relative distances are mostly maintained.
from sklearn.manifold import TSNE
# The most common way of importing matplotlib for plotting in Python
from matplotlib import pyplot as plt
# For manipulating axis tick locations
from matplotlib import ticker
Data
Next, we load our dataset! Here, we will use the UCI Iris dataset again as a relatively simple example. This time, we will modularize the preprocessing code a little more! First, we have the function to load the data:
# Pandas for loading and manipulating data as a DataFrame
import pandas as pd
# For loading the iris dataset as an example
from sklearn.datasets import load_iris
def load_data() -> pd.DataFrame:
# Load the iris dataset as a DataFrame
iris = load_iris(as_frame=True)
# Extract the DataFrame from the dictionary the loader provides
data = iris['frame']
# Return the data as a DataFrame
return data
# Load the data
data = load_data()
# Print the first several rows to get an idea of what it looks like
data.head()
| sepal length (cm) | sepal width (cm) | petal length (cm) | petal width (cm) | target | |
|---|---|---|---|---|---|
| 0 | 5.1 | 3.5 | 1.4 | 0.2 | 0 |
| 1 | 4.9 | 3.0 | 1.4 | 0.2 | 0 |
| 2 | 4.7 | 3.2 | 1.3 | 0.2 | 0 |
| 3 | 4.6 | 3.1 | 1.5 | 0.2 | 0 |
| 4 | 5.0 | 3.6 | 1.4 | 0.2 | 0 |
This dataset has 4 features, and it does include supervised labels.
Working directly with DataFrames is good for data analysis, but for machine learning, we often have to think more carefully about how we handle our datasets in efficient ways during training and inference, especially when words like “mini-batching”, “memory pinning”, and “transforms” come up!
Data Container
For a simple demonstration, we create a convenient container for the preprocessed samples:
# The PyTorch library containing neural network utilities and the Tensor datatype
import torch
# Dataclass for a structured way of passing around a dataset
from dataclasses import dataclass
# Create a simple container for our preprocessed data with some introspection methods
@dataclass
class PreprocessedDataset:
x: torch.Tensor
y: torch.Tensor
# The number of samples
def n_samples(self):
return y.shape[0]
# The "original" (not complement-coded) data feature dimension
def dim(self):
return int(x.shape[1] / 2)
# The complement-coded dimension
def dim_cc(self):
return x.shape[1]
# Check out added methods from the decorator.
help(PreprocessedDataset)
Help on class PreprocessedDataset in module __main__:
class PreprocessedDataset(builtins.object)
| PreprocessedDataset(x: torch.Tensor, y: torch.Tensor) -> None
|
| PreprocessedDataset(x: torch.Tensor, y: torch.Tensor)
|
| Methods defined here:
|
| __eq__(self, other)
| Return self==value.
|
| __init__(self, x: torch.Tensor, y: torch.Tensor) -> None
| Initialize self. See help(type(self)) for accurate signature.
|
| __repr__(self)
| Return repr(self).
|
| dim(self)
| # The "original" (not complement-coded) data feature dimension
|
| dim_cc(self)
| # The complement-coded dimension
|
| n_samples(self)
| # The number of samples
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables
|
| __weakref__
| list of weak references to the object
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| __annotations__ = {'x': <class 'torch.Tensor'>, 'y': <class 'torch.Ten...
|
| __dataclass_fields__ = {'x': Field(name='x',type=<class 'torch.Tensor'...
|
| __dataclass_params__ = _DataclassParams(init=True,repr=True,eq=True,or...
|
| __hash__ = None
|
| __match_args__ = ('x', 'y')
Let’s create a quick dataset with some random dummy data to see what creating a “formal” dataset class gets us:
# Create a dummy dataset with some dimension and number of samples
dim = 2
n_samples = 5
# Set a manual random seed for reproducibility
torch.manual_seed(1234)
dummy_data = PreprocessedDataset(
torch.rand((n_samples, dim)),
torch.randint(0, 3, (n_samples,))
)
dummy_data
PreprocessedDataset(x=tensor([[0.0290, 0.4019],
[0.2598, 0.3666],
[0.0583, 0.7006],
[0.0518, 0.4681],
[0.6738, 0.3315]]), y=tensor([0, 1, 1, 1, 0]))
One thing that we notice is that we got a fancier string representation (__repr__) of the object when it’s displayed on the terminal, and we didn’t have to write it ourselves.
This is a simple example, but it is one of the beneficial things that we can get when thinking a little bit about how we handle datasets (in this case, hooking into a Python standard library dataclass decorator).
Creating object to hold your data, such as annotating with the standard library decorator dataclasses.@dataclass, is a useful trick to inherit some boilerplate machinery.
In PyTorch, the most universal API to utilize and interface with datasets is in torch.utils.data, which includes the Dataset and DataLoader classes.
For now, we’ll stick with this simpler implementation for illustration.
Next up is how to preprocess the data!
Data Preprocessing
Now we define the function to preprocess the data. In this example, we do this on the full batch rather than incrementally for the following reasons:
First, we should say that FuzzyART requires the input data to be 1-D samples (a.k.a. one-dimensional real-valued vectors).
This is because of some preprocessing requirements for the algorithm; FuzzyART uses complement coding, which maps $x \rightarrow [x, 1-x]$ and is bounded in $[0, 1]$.
This is known as complement coding because we are taking the ones-complement of the vector and appending it to the sample.
To do this, we need the original $x$ to be also be bounded inside $[0, 1]$.
Most real data is not neatly normalized and bounded, so we need to do it ourselves at some point.
However, that requires knowing the bounds of the full data in advance!
Even though we are working with an incremental algorithm, we have the luxury of having the all of the Iris dataset up-front, and the dataset surely isn’t going to change any time soon. This is not always the case, especially if you are dealing with streaming datasets, where you are incrementally provided a sample one at a time! In those cases, you have two options:
- Know the statistics of the dataset in advance (e.g., the upper and lower bounds) and preprocess each sample incrementally off of that.
- Use some sort of intelligent normalization scheme that enforces the bounds of the data to $[0, 1]$, such as through incorporating a limiting function like the sigmoid function $\sigma = \dfrac{1}{1+e^{-x}}$ or some other hard limiting function.
# Numpy for handling numpy arrays
import numpy as np
# A sklearn utility for handling normalization of data automatically
from sklearn.preprocessing import MinMaxScaler
# Define a preprocess function that returns a PreprocessedDataset
def preprocess(
data: pd.DataFrame,
shuffle: bool = True,
random_seed: int = 12345,
) -> PreprocessedDataset:
# Shuffle the data if necessary
if shuffle:
np.random.seed(random_seed)
data = data.sample(frac=1).reset_index(drop=True)
# Whether shuffled or not, separate the labels
labels = torch.tensor(data.pop('target'), dtype=torch.int)
# Initialize the scalar and update the values in-place to be normalized between [0, 1]
scaler = MinMaxScaler()
# Linearly normalize the data and put it into a tensor
data_cc = torch.tensor(scaler.fit_transform(data))
# Complement code the data by appending the vector [1-x] along the feature dimension
data_cc = torch.cat((data_cc, 1 - data_cc), dim=1)
# What we get is a list of 8-dimensional samples
return PreprocessedDataset(data_cc, labels)
# Preprocess the data and check it out
data_cc = preprocess(data)
data_cc
PreprocessedDataset(x=tensor([[0.3611, 0.2083, 0.4915, ..., 0.7917, 0.5085, 0.5833],
[0.0278, 0.5000, 0.0508, ..., 0.5000, 0.9492, 0.9583],
[0.5556, 0.5417, 0.6271, ..., 0.4583, 0.3729, 0.3750],
...,
[0.5278, 0.3333, 0.6441, ..., 0.6667, 0.3559, 0.2917],
[0.8056, 0.4167, 0.8136, ..., 0.5833, 0.1864, 0.3750],
[0.1111, 0.5000, 0.1017, ..., 0.5000, 0.8983, 0.9583]],
dtype=torch.float64), y=tensor([1, 0, 1, 0, 0, 0, 1, 0, 1, 2, 0, 2, 1, 1, 2, 2, 0, 2, 1, 1, 1, 1, 0, 1,
0, 2, 0, 1, 0, 2, 0, 0, 2, 2, 1, 2, 2, 1, 0, 1, 0, 1, 0, 2, 1, 1, 0, 1,
1, 1, 2, 0, 1, 0, 2, 2, 0, 2, 0, 2, 2, 2, 2, 1, 2, 1, 0, 0, 1, 0, 2, 2,
0, 0, 2, 0, 2, 1, 0, 0, 1, 2, 0, 2, 1, 0, 1, 1, 1, 0, 0, 1, 2, 0, 1, 1,
2, 1, 1, 0, 0, 1, 2, 0, 1, 2, 2, 1, 2, 1, 1, 2, 2, 2, 2, 2, 0, 1, 0, 0,
0, 2, 1, 0, 1, 2, 1, 1, 0, 2, 0, 0, 0, 1, 1, 0, 2, 2, 1, 2, 1, 2, 2, 0,
2, 0, 2, 2, 2, 0], dtype=torch.int32))
Now that that we have our data defined and preprocessed (linearly normalized and complement coded), we can define some parts of our FuzzyART module!
FuzzyART
We need to define the following things to have a complete FuzzyART module:
- Weights: How the weights are defined in memory. This is a growing data structure in the sense that FuzzyART instantiates new nodes when necessary, so whatever it is that we do, we need a mechanism to append weights.
- Activation: The FuzzyART activation function; this takes the weights and input sample, and it spits out a number.
- Match: The FuzzyART match function. This also is a function of the weights and current sample, and it returns a number.
- Learning: One a winning node is selected, we need to define what learning is; the weight update function returns a new weight as a function of the old weight, the current sample, and the learning rate.
- Match Rule/Competition: This function iterates over all of the weights, calculating their activation and match values, selecting a winning node, and updating it. This is essentially the main loop that we will call during training and inference, performing the FuzzyART “match rule.”
What this translates to is an empty API to “fill out” in steps.
We often use the abstract class pattern in software development to do such a thing, which is implemented in Python using the standard library module abc, such as the following:
from abc import ABC, abstractmethod
class FuzzyARTBase(ABC):
# A method to initialize a new category, however it the weights are defined
@abstractmethod
def grow(self, x: torch.Tensor):
pass
# The activation function for a single sample and weight
@abstractmethod
def activation(self, x: torch.Tensor, j: int) -> torch.float:
pass
# The match function for a single sample and weight
@abstractmethod
def match(self, x: torch.Tensor, j: int) -> torch.float:
pass
# The learning function once a weight update has
@abstractmethod
def learn(self, x: torch.Tensor, j: int) -> torch.Tensor:
pass
# The FuzzyART match rule (competition) loop, which
@abstractmethod
def competition(self, x: torch.Tensor, j: int) -> int:
pass
# The training method
@abstractmethod
def train(self, x:torch.Tensor):
pass
# After training, the classification method
@abstractmethod
def classify(self, x:torch.Tensor) -> int:
pass
For a full implementation of almost exactly this, check out the notebook at this link that does this in a single FuzzyART class with PyTorch tensor weights.
In this notebook we’ll be inspecting each of these functions individually, so we’ll modularize it into a set of functions that take all of the info that they need and spit out the relevant information.
As a complete side note, you could conceivably combine both methods (creating an abstract base class and implementing the methods after the fact) with something like the following:
# The types module has all sorts of black magic within
from types import MethodType
# Define some basic class with data
class MyClass:
def __init__(self, x):
self.x = x
# Initialize the object
obj = MyClass(1)
# Verify that the class doesn't have the method we need
print(f"Checking if `obj` has 'added_method': {hasattr(MyClass, 'added_method')}")
# Define a dangling method after defining the class with a "self" argument
def added_method(self):
return f"My x is: {self.x}"
# Attach the function as a method of the instantiated object
obj.check = MethodType(added_method, obj)
# Verify that the instantiated object has the method, and see what it outputs
print(f"Checking if `obj` has an 'added_method': {hasattr(obj, 'check')}")
print(f"Output of 'added_method': {obj.check()}")
Checking if `obj` has 'added_method': False
Checking if `obj` has an 'added_method': True
Output of 'added_method': My x is: 1
That’s not what we’ll do here, but it’s neat what you can do with code when you think outside the box!
Weights
Next, we need to define our weights! But to do that, we need to think through what we need the weights to be, and then we should consider how to implement them in memory.
We need to define our weights abstractly as ultimately a list of categories.
These categories are really prototypes, in the vein of prototype-based clustering algorithms.
These prototypes are ultimately real-valued vectors of dimension $2|x|$, where $x$ is the input sample, and we are taking the “dimension” of $x$ through the size operator $|x|$.
Remember, in FuzzyART these weights/categories/prototypes are of size $2|x|$ because we linearly normalize and complement code the samples, doubling the length of that vector, before presenting to the FuzzyART module.
Furthermore, we will have an increasing number of categories throughout the module’s lifetime, so we need the data structure to be able to grow to size $2|x| \times M$, where $M$ is the arbitrary letter that I chose here to represent the number of categories after training $N$ samples (want a different letter here? You do you! You can use whatever notation you want! As long as you’re consistent with yourself, or otherwise that can get confusing. Why this long parenthetical? Because notation varies wildly between different fields and even between different papers in the same field, so the important thing is to get familiar with the core ideas behind what the notation represents. Anyway, back to the notebook.)
So, what data structure will accomplish these objective? Probably something that looks like a 2-D array that can grow in one dimension! Think about it: we can preallocate a very large 2-D array of some size $2|x| \times M$ where $M$ is very big and the unused weights are all zeros, but we still won’t know in theory that it is sufficient, since we could end up having more than $M$ categories after some long training sequence (well, we could technically cap the number of weights to $M$ and do some funky stuff to the training procedure after we hit that limit, but that’s a can of worms best saved for its own lesson).
So, we can either have a list of categories that grows each time we “append” a new weight, like the following
# Set some sample feature dimension and init a set of weigthts
dim, n_existing_weights = 3, 5
W = [np.random.rand(dim) for _ in range(n_existing_weights)]
print(f"Old number of weights: {len(W)}")
# Create a new "weight" after some kind of training process and use list.append
w_new = np.random.rand(dim)
W.append(w_new)
print(f"New number of weights: {len(W)}")
Old number of weights: 5
New number of weights: 6
Or, we could use a dynamic preallocation strategy for vectors like Facebook’s fbvector fancy data type, which preallocates some number of elements of the vector, and then preallocates even more when the previous list gets full.
What’s really common to see is a 2-D matrix strategy that dynamically grows along the $M$ dimension, such as:
# This should look familiar.... Define some random existing weight matrix
dim, n_existing_weights = 3, 5
W = np.random.rand(dim, n_existing_weights)
print(f"Old weight matrix dimension: {W.shape}")
# Create a new "vector" and append it to the weights dimension, which we chose as the second dimension here
# NOTE: we create a 2-D "vector" with dimension dim x 1 here so we can use np.concat correctly
w_new = np.random.rand(dim, 1)
W = np.concat((W, w_new), axis=1)
print(f"New weight dimension: {W.shape}")
Old weight matrix dimension: (3, 5)
New weight dimension: (3, 6)
This makes the weight matrix pretty explicit, but the downside to this method is that technically we are copying a bunch of memory around and recreating the W matrix under the hood each time we do a concatenation.
That can be super inefficient, so in practice we do a combination of these two techniques: represent the weights as a 2-D matrix and then use some library that handles dynamically resizing a matrix along one dimension efficiently through some growing preallocation strategy.
TODO
This notebook is a work in progress! If you see this, it means that there is more to come for this notebook.
Page built at: 2026-07-31 22:51:47 +0000