**Maple Examples of Activation Functions**

**Identity Function**

The identity function is given by

`> `
**f1 := proc(x) x; end;**

`> `
**plot(f1(x),x=-10..10);**

**Step Function**

The step function is if and if . This is also called the heaviside function. Another common variation is for it to take on values -1 and +1 as shown below.

`> `
**f2 := proc(x) 2*Heaviside(x) -1 end;**

`> `
**plot(f2(x),x=-10..10);**

**Logistic Function (Sigmoid)**

The logistic function has the form

`> `
**f3 := proc(x,a) 1/(1 + exp(-a*x)) end;**

`> `
**plot(f3(x,1),x=-10..10);**

`> `
**with(plots):**

The parameter "a" in the logistic function determines how steep it is. The larger "a", the steeper it is.

`> `
**display([plot(f3(x,1),x=-10..10),**

`> `
** plot(f3(x,2),x=-10..10, color = blue),**

`> `
** plot(f3(x,.5),x=-10..10, color = green)]);**

**Symmetric Sigmoid**

The symmetric sigmoid is simply the sigmoid that is stretched so that the y range is 2 and then shifted down by 1 so that it ranges between -1 and 1. If g(x) is the standard sigmoid then the symmetric sigmoid is

`> `
**f4 := proc(x) 2*f3(x,1)-1 end;**

`> `
**plot(f4(x),x=-10..10);**

The symmetric sigmoid differs from the hyperbolic tangent by a constant factor. As you can see, the graph below is identical to the graph for the symmetric sigmoid.

`> `
**plot(tanh(.5*x),x=-10..10);**

**Radial Basis Functions**

A radial basis function is simply a gaussian, . It is called local because, unlike the previous functions, it is essentially zero everywhere except in a small region.

`> `
**f5 := proc(x,a) exp(-a * x^2); end;**

`> `
**plot(f5(x,1),x=-10..10);**

**Derivatives**

The derivative of the
**identity function**
is just 1. That is, if f(x) is the identity then

The derivative of the
**step function**
is not defined which is exactly why it isn't used.

The nice feature of
**sigmoids**
is that their derivatives are easy to compute. If f(x) is the logistic function above then
.

`> `
**diff(f3(x,a),x);**

`> `
**simplify(f3(x,a)*(1-f3(x,a)));**

This is also true of
**hyperbolic tangent**
. If f(x) is tanh the

`> `
**diff(tanh(x),x);**

`> `