CS-449: Neural Networks

Fall 99

Instructor: Genevieve Orr

Willamette University

Lecture Notes prepared by Genevieve Orr, Nici Schraudolph, and Fred Cummins

[Content][Links]

Course content

Summary

Our goal is to introduce students to a powerful class of model, the Neural Network. In fact, this is a broad term which includes many diverse models and approaches. We will first motivate networks by analogy to the brain. The analogy is loose, but serves to introduce the idea of parallel and distributed computation.

We then introduce one kind of network in detail: the feedforward network trained by backpropagation of error. We discuss model architectures, training methods and data representation issues. We hope to cover everything you need to know to get backpropagation working for you. A range of applications and extensions to the basic model will be presented in the final section of the module.

Lecture 1: Introduction

• Questions
• Motivation and Applications
• Computation in the brain
• Artificial neuron models
• Linear regression
• Linear neural networks
• Multi-layer networks
• Error Backpropagation

Lecture 2: Classification

• Introduction
• Perceptron Learning
• Delta Learning
• Doing it Right

Lecture 3: Optimizing Linear Networks

• Weights and Learning Rates
• Summary

Lecture 4: The Backprop Toolbox

• 2-Layer Networks and Backprop
• Noise and Overtraining
• Momentum
• Delta-Bar-Delta
• Many layer Networks and Backprop
• Backprop: an example
• Overfitting and regularization
• Growing and pruning networks
• Preconditioning the network
• Momentum
• Delta-Bar-Delta

Lecture 5: Unsupervised Learning

• Introduction
• Linear Compression (PCA)
• NonLinear Compression
• Competitive Learning
• Kohonon Self-Organizing Nets

Lecture 6: Reinforcement Learning

• Introduction
• Components of RL
• Terminology and Bellman's Equation

Lecture 7: Advanced Topics

• Learning rate adaptation
• Classification
• Non-supervised learning
• Time-Delay Neural Networks
• Recurrent neural networks
• Real-Time Recurrent Learning
• Dynamics of RNNs
• Long Short-Term Memory

[Top]

Review for Midterm:

• Linear Nets
• Non-linear Nets

Links

Tutorials:

• The Nervous System - a very nice introduction, many pictures

• Neural Java - a neural network tutorial with Java applets
• Web Sim - A Java neural network simulator.
• a book chapter describing the Backpropagation Algorithm (Postscript)
• A short set of pages showing how a simple backprop net learns to recognize the digits 0-9, with C code
• Reinforcement Learning - A Tutorial

Simulators and code:

	Web Sim: Java neural network simulator.
	Brainwave: a Java based simulator
	tlearn: Windows, Macintosh and Unix implentation of backprop and variants. Written in C.
	PDP++: C++ software with every conceivable bell and whistle. Unix only. The manual also makes a good tutorial.

Data Sources:

• UCI machine learning database
• ai-faq/neural-nets data source list
• Handwritten Digits

Related stuff of interest:

• A page of neural network links
• Tesauro's backgammon network
• Lego Lab at University of Aarhus

[Top]

count: