With this binary function f, the problem reduces to finding weights such that
sign( W X) = t
That is, the weight must be chosen so that the projection of pattern X onto W has the same sign as the target t. But the boundary between positive and negative projections is just the plane W X = 0 , i.e. the same decision boundary we saw before.
For each training pattern (x, t):
- compute output activation y = f(w x)
- If y = t, don't change weights
- If y != t, update the weights:
w(new) = w(old) + 2 m t x
or
w(new) = w(old) + m (t - y ) x, for all t
Consider wht happens below when the training pattern p1 or p2 is chosen. Before updating the weight W, we note that both p1 and p2 are incorrectly classified (red dashed line is decision boundary). Suppose we choose p1 to update the weights as in picture below on the left. P1 has target value t=1, so that the weight is moved a small amount in the direction of p1. Suppose we choose p2 to update the weights. P2 has target value t=-1 so the weight is moved a small amount in the direction of -p2. In either case, the new boundary (blue dashed line) is better than before.
Come to class for proof!!
Outline: Find a lower bound L(k) for |w|^{2} as a function of iteration k. Then find an upper bound U(k) for |w|^{2}. Then show that the lower bound grows at a faster rate than the upper bound. Since the lower bound can't be larger than the upper bound, there must be a finite k such that the weight is no longer updated. However, this can only happen if all patterns are correctly classified.