I wanted to play with a machine learning algorithm that doesn’t use gradient descent, so I threw together a self-organizing map. It’s an algorithm for fitting a graph of points to data, but the graph can be a different dimension from the data. The main application is to pick a 2D grid as the “graph,” fit that grid to some high-dimensional data, and then read the data off the grid as a dimensionality reduction technique.
I stumbled across self-organizing maps while traipsing through one of my all-too-frequent Wikipedia rabbit holes. It trains a model using small iterative updates, but it doesn’t have to calculate a gradient to figure out those small updates. It’s such a simple idea that makes intuitive sense to me. Wrap a cloth around something, and then flatten out the cloth to see it in 2D.
Here’s a little demo where I go from 3D to 2D. First, I make a 10-by-10 grid and stretch it out to fit the data.
Once the self-organizing map fits my data, I can flatten out the map. By counting how many data points are closest to each node in the grid, I can see the data on a 2D grid. I think it looks a bit like a quilt.
This plot still shows the distinct clusters, even though it only has two spatial dimensions to display this 3D data. It’s a little silly to do this for 3D data since I can just look at the data and see the clusters. But real data, such as text embeddings, can have thousands of dimensions. That’s where dimensionality reduction techniques come in handy.
The algorithm for self-organizing maps seems like it could be used for something more interesting than dimensionality reduction.
I don’t know what, exactly.
Maybe I could make graphs that have 1, 2, 3, ..., n 2D clusters in them and then see how well they fit my data to determine how many clusters are in my high-dimensional data.
People usually do that with K-Means.
Or maybe I could keep the edge length information when I flatten out the map instead of putting all the nodes into a perfect grid.
There must be something. I’ll keep thinking about it.
You can check out the code for the self-organizing map and these plots on my GitHub.