EECS 349, Machine Learning

  Insturctor: Prof. Doug Downey
By Majed Valad Beigi
majed.beigi@northwestern.edu
Northwestern University
home Introduction Design Back Propagation NN LAMSTAR NN SVM Results Conclusion

Offline HandWritten Character Recognition
 

You can download the full project report and source code here:
FullReport_Source Code


LAMSTAR Neural Network

The LAMSTAR network has the following components:

(a) Input word and its subwords:

The input word is divided into a number of subwords. Each subword represents an attribute of the input word. For this part I have considered the 6*8 matrices obtained from the feature extraction algorithm.

In this case, each subword is 4 bit long and is obtained by running a 2*2 window on the feature matrix of a character (each cell of the feature matrix actually contains the information of a 15(pixels)*15(pixels) block of the original sub-image of a character). Thus I totally have 12 subwords for each character.

Figure 5: Obtaining each sub-word by running a 2*2 window on the feature matrix.

Figure 6: Layout of the LAMSTAR Neural Network.

(b) SOM modules for storing input subwords:

For every subword there is an associated Self Organizing Map (SOM) module with neurons that are designed to function as Kohonen `Winner Take All' neurons where the winning neuron has an output of 1 while all other neurons in that SOM module have a zero output.

In this algorithm, the SOM modules are built dynamically in the sense that instead of setting the number of neurons at some value arbitrarily, the network was built to have neurons depending on the class to which a given input to a particular subword might belong. For example if there are two subwords that have all their pixels as `1's, then these would fire the same neuron in their SOM layer and hence all they need is 1 neuron instead of 2 neurons. This way the network is designed with lesser number of neurons and the time taken to fire a particular neuron at the classification stage is reduced considerably.

(c) Output (decision) layer:

The present output layer is designed to have six layers, which have the following neuron firing patterns:

Class

A

B

C

D

E

F

G

H

I

J

K

L

M

N

O

P

Q

R

S

T

U

V

W

X

Y

Z

 

 

 

Output

0

0

0

0

0

0

0

0

0

0

0

1

0

0

0

0

1

0

0

0

0

0

1

1

0

0

0

1

0

0

0

0

0

1

0

1

0

0

0

1

1

0

0

0

0

1

1

1

0

0

1

0

0

0

0

0

1

0

0

1

0

0

1

0

1

0

0

0

1

0

1

1

0

0

1

1

0

0

0

0

1

1

0

1

0

0

1

1

1

0

0

0

1

1

1

1

0

1

0

0

0

0

0

1

0

0

0

1

0

1

0

0

1

0

0

1

0

0

1

1

0

1

0

1

0

0

0

1

0

1

0

1

0

1

0

1

1

0

0

1

0

1

1

1

0

1

1

0

0

0

0

1

1

0

0

1

Class

a

b

c

d

e

f

g

h

i

j

k

l

m

n

o

p

q

r

s

t

u

v

w

x

y

z

 

 

 

Output

1

0

0

1

1

0

1

0

0

1

1

1

1

0

1

0

0

0

1

0

1

0

0

1

1

0

1

0

1

0

1

0

1

0

1

1

1

0

1

1

0

0

1

0

1

1

0

1

1

0

1

1

1

0

1

0

1

1

1

1

1

1

0

0

0

0

1

1

0

0

0

1

1

1

0

0

1

0

1

1

0

0

1

1

1

1

0

1

0

0

1

1

0

1

0

1

1

1

0

1

1

0

1

1

0

1

1

1

1

1

1

0

0

0

1

1

1

0

0

1

1

1

1

0

1

0

1

1

1

0

1

1

1

1

1

1

0

0

1

1

1

1

0

1

1

1

1

1

1

0

1

1

1

1

1

1

Table 1: Values at the output layer for each character (class).

The link-weights from the input SOM modules to the output decision layer are adjusted during training on a reward/punishment principle. Furthermore, they continue being trained during normal operational runs. Specifically, if the output of the particular output neuron is what is desired, then the link weights to that neuron is rewarded by increasing it by a non-zero increment, while punishing it by a small non-zero number if the output is not what is desired.

o   Training algorithm:

The training of the LAMSTAR network with 208 character (4 per each of the 52 English alphabet).  The training was performed as follows:

1- Subword Formation:

The input patterns are to be divided into subwords before training/testing the LAMSTAR network.

2- Input Normalization:

Each subwords of every input pattern is normalized as follows:

Where j=1, 2, 3 and 4; 

xi is the un-normalized subword of an input pattern and xi' is the normalized subword of the same input pattern. During the process, those subwords, which are all zeros, are identified and their normalized values are manually set to zero.

3- Dynamic Neuron formation in the SOM modules:

The first neurons in all the SOM modules are constructed as Kohonen neurons as follows:

As the first pattern is input to the system, one neuron is built with 4 inputs and weights to start with are initialized the normalized input subword.

When the subwords of the subsequent patterns is input to the respective modules, the output at any of the previously built neuron is checked to see if it is close to 1 (with a tolerance of 0.05). If one of the neurons satisfies the condition, then this is declared as the winning neuron, i.e., a neuron whose weights closely resemble the input pattern. Else another neuron is built with new sets of weights that are normalized and adjusted as above to resemble the input subword.

During this process, if there is a subword with all zeros then this will not contribute to a change in the output and hence the output is made to zero and the process of finding a winning neuron is bypassed for such a case.

4- Desired neuron firing pattern:

The output neuron firing pattern for each character in the training set has been established as given in Table 1.

5- Link weights:

Link weights are defined as the weights that come from the winning neuron at every module to the 6 output neurons. If in the desired firing, a neuron is to be fired, then its corresponding link weights are rewarded by adding a small positive value of 0.05 till the output is as expected. On the other hand, if a neuron should not be fired then its link weights are reduced by 0.05 till the output is as expected.

This will result in the summed link weights at the output layer being a positive value indicating a fired neuron if the neuron has to be fired for the pattern and high negative value if it should not be fired.

6- The weights at the SOM neuron modules and the link weights are stored.

o   Testing:

The LAMSTAR network was tested with 520 patterns as follows, 10 for each 52 classes.

1- The patterns are processed to get 12 subwords as before. Normalization is done for the subwords as explained in the training.

2- The stored weights are loaded

3- The subwords are propagated through the network and the neuron with the maximum output at the Kohonen layer is found and their link weights are sent to the output neurons.

4- The output is a sum of all the link weights.