Maple Examples of Activation Functions

Identity Function

The identity function is given by [Maple Math]

> f1 := proc(x) x; end;

[Maple Math]

> plot(f1(x),x=-10..10);

[Maple Plot]

Step Function

The step function is [Maple Math] if [Maple Math] and [Maple Math] if [Maple Math] . This is also called the heaviside function. Another common variation is for it to take on values -1 and +1 as shown below.

> f2 := proc(x) 2*Heaviside(x) -1 end;

[Maple Math]

> plot(f2(x),x=-10..10);

[Maple Plot]

Logistic Function (Sigmoid)

The logistic function has the form [Maple Math]

> f3 := proc(x,a) 1/(1 + exp(-a*x)) end;

[Maple Math]

> plot(f3(x,1),x=-10..10);

[Maple Plot]

> with(plots):

The parameter "a" in the logistic function determines how steep it is. The larger "a", the steeper it is.

> display([plot(f3(x,1),x=-10..10),

> plot(f3(x,2),x=-10..10, color = blue),

> plot(f3(x,.5),x=-10..10, color = green)]);

[Maple Plot]

Symmetric Sigmoid

The symmetric sigmoid is simply the sigmoid that is stretched so that the y range is 2 and then shifted down by 1 so that it ranges between -1 and 1. If g(x) is the standard sigmoid then the symmetric sigmoid is [Maple Math]

> f4 := proc(x) 2*f3(x,1)-1 end;

[Maple Math]

> plot(f4(x),x=-10..10);

[Maple Plot]

The symmetric sigmoid differs from the hyperbolic tangent by a constant factor. As you can see, the graph below is identical to the graph for the symmetric sigmoid.

> plot(tanh(.5*x),x=-10..10);

[Maple Plot]

Radial Basis Functions

A radial basis function is simply a gaussian, [Maple Math] . It is called local because, unlike the previous functions, it is essentially zero everywhere except in a small region.

> f5 := proc(x,a) exp(-a * x^2); end;

[Maple Math]

> plot(f5(x,1),x=-10..10);

[Maple Plot]


The derivative of the identity function is just 1. That is, if f(x) is the identity then [Maple Math]

The derivative of the step function is not defined which is exactly why it isn't used.

The nice feature of sigmoids is that their derivatives are easy to compute. If f(x) is the logistic function above then [Maple Math] .

> diff(f3(x,a),x);

[Maple Math]

> simplify(f3(x,a)*(1-f3(x,a)));

[Maple Math]

This is also true of hyperbolic tangent . If f(x) is tanh the [Maple Math]

> diff(tanh(x),x);

[Maple Math]