Simple Competitive Learning

In competitive networks, output units compete for the right to respond.

Goal: method of clustering - divide the data into a number of clusters such that the inputs in the same cluster are in some sense similar.

A basic competitive learning network has one layer of input nodes and one layer of output nodes. Binary valued outputs are often (but not always) used. There are as many output nodes as there are classes.

Often (but not always) there are lateral inhibitory connections between the output nodes.(in simulations, the function of the lateral connections can be replaced with a different algorithm)

The output units are also often called grandmother cells. The term grandmother cell comes from discussions as to whether your brain might contain cells that fire only when you encounter your maternal grandmother, or whether such higher level concepts are more distributed.

Vector Quantization (VQ)

Vector quantization is one example of competitive learning.

The goal here is to have the network "discover" structure in the data by finding how the data is clustered. The results can be used for data encoding and compression. One such method for doing this is called vector quantization.

In vector quantization, we assume there is a codebook which is defined by a set of M prototype vectors. (M is chosen by the user and the initial prototype vectors are chosen arbitrarily).

An input belongs to cluster i if i is the index of the closest prototype (closest in the sense of the normal euclidean distance). This has the effect of dividing up the input space into a Voronoi tesselation


Implementing Vector Quantization with a Network



VQ and Data Compression

Vector quantization can be used for (lossy) data compression. If we are sending information over a phone line, we

For a large amount of data, this can be a significant reduction. If M=64, then it takes only 6 bits to encode the index. If the data itself consists of floating point numbers (4 bytes) there is an 80% reduction ( 100*(1 - 6/32) ).

Learning Vector Quantization (LVQ)

This is a supervised version of vector quantization. Classes are predefined and we have a set of labelled data. The goal is to determine a set of prototypes the best represent each class.

 [Top] [Next: Kohonon] [Back to the first page]