Mathematics/Sigmoid

The Sigmoidal Functions

One of the attractive qualities of the sigmoidal functions, otherwise known as the logistic functions, is their attractively simple (computationally and mathematically) derivatives.

The Derivative of the Sigmoidal Functions

Although in most artificial neural networks, the simple sigmoid is all that is required, here we will keep all constants and solve. At the end, A can be set to 0, and the other constants to 1 to simplify the derivative to that of the simple logistic function; but sometimes it is useful to have the non-simplified equation at hand.

First, starting with the generic logistic function, we have the equation rearranged for reasons that will become apparent later...

Equation

$\\ y\; =\; A\; +\; \frac{B}{1\; +\; e^{-cx}},\; c=1,\; A=-1,\; B=2\\ \therefore\; \frac{y\; -\; A}{B}\; =\; \frac{1}{1\; +\; e^{-cx}}\\ $

Next, also for reasons that will become apparent soon, we subtract the above equations from 1, this gives...

Equation

$\\ 1\; -\; \frac{y\; -\; A}{B}\; =\; 1\; -\; \frac{1}{1+e^{-cx}}\\ \therefore\; \frac{B\; -\; y\; +\; A}{B}\; =\; \frac{1\; +\; e^{-cx}\; -\; 1}{1\; +\; e^{-cx}}\; =\; \frac{e^{-cx}}{1\; +\; e^{-cx}}\\ $

Finally, we can start finding the derivative; doing so by use of the quotient rule...

Equation

$\\ \frac{dy}{dx}\; =\; \frac{0\cdot \left( 1\; +\; e^{-cx} \right)\; -\; B\cdot \left( -ce^{-cx} \right)}{\left( 1\; +\; e^{-cx} \right)^{2}}\; =\; \frac{B\cdot c\cdot e^{-cx}}{\left( 1\; +\; e^{-cx} \right)^{2}}\\ =\; B\cdot c\cdot \frac{1}{1\; +\; e^{-cx}}\cdot \frac{e^{-cx}}{1\; +\; e^{-cx}}\\ =\; B\cdot c\cdot \left( \frac{y\; -\; A}{B} \right)\cdot \left( \frac{B\; -\; y\; +\; A}{B} \right) $

Paying close attention to the two parts in the brackets above, you can see why we started with putting the equations in the formats we did. Next, we simply substitute these and simplify...

Equation

$\frac{dy}{dx}\; =\; \frac{c\cdot \left( y\; -\; A \right)\cdot \left( B\; -\; y\; +\; A \right)}{B}$

Setting A=0, B=1, and c=1, this simplifies to the simple logistic function as promised...

Equation

$y\; =\; y\cdot \left( 1\; -\; y \right)$

A useful property from the sigmoid that is sometimes used, it that by increasing the constant c, the sigmoid function can be made to approximate the step function. In the following illustration, the function has been plotted with c set to 1, 2, 3, 10, and 100...

In the following applet, the orange line is the logistic function with A=0,0, B=1.0, and the yellow line is the derivative. The mouse movement from left to right controls c.

Sorry, either your browser cannot handle Java applets, or Java is not enabled.

Ancestors ☣ Mathematics
Other ☣ Algorithm/Learning/GradientDescent ☣ Learning/Machine/Perceptron