Machine learning for Biomedical Engineers

Activation functions

Binary Step

\[\begin{split}f(x) = \left\{ \begin{array}{lll} 0 & for & x < x_{min} \\ mx+b & for & x_{min} \leq x \leq x_{max} \\ 1 & for & x > x_{max} \end{array} \right.\end{split}\]
\[\begin{split}f'(x) = \left\{ \begin{array}{lll} 0 & for & x \neq 0 \\ ? & for & x = 0 \end{array} \right.\end{split}\]

(Source code, png, hires.png, pdf)

../_images/binary_step.png

Piecewise Linear

\[\begin{split}f(x) = \left\{ \begin{array}{lll} 0 & for & x < x_{min} \\ mx+b & for & x_{min} \leq x \leq x_{max} \\ 1 & for & x > x_{max} \end{array} \right.\end{split}\]
\[\begin{split}f'(x) = \left\{ \begin{array}{lll} 0 & for & x < x_{min} \\ m & for & x_{min} \leq x \leq x_{max} \\ 0 & for & x > x_{max} \end{array} \right.\end{split}\]

(Source code, png, hires.png, pdf)

../_images/piecewise_linear.png

Bipolar

\[\begin{split}f(x) = \left\{ \begin{array}{lll} -1 & for & x \leq 0 \\ 1 & for & x > 0 \end{array} \right.\end{split}\]
\[\begin{split}f'(x) = \left\{ \begin{array}{lll} 0 & for & x \neq 0 \\ ? & for & x = 0 \end{array} \right.\end{split}\]

(Source code, png, hires.png, pdf)

../_images/bipolar.png

Sigmoid

\[f(x)={\frac {1}{1+e^{-x}}}\]
\[f'(x)=f(x)(1-f(x))\]

(Source code, png, hires.png, pdf)

../_images/sigmoid.png

Bipolar Sigmoid

\[f(x)={\frac {1-e^{-x}}{1+e^{-x}}}\]
\[f'(x)={\frac {2e^x}{(e^x+1)^2}}\]

(Source code, png, hires.png, pdf)

../_images/bipolar_sigmoid.png

Hyperbolic Tangent, TanH

\[f(x)={\frac {2}{1+e^{-2x}}}-1\]
\[f'(x)=1-f(x)^2\]

(Source code, png, hires.png, pdf)

../_images/tanh.png

Arctangent, ArcTan

\[f(x)=tan^{-1}(x)\]
\[f'(x)={\frac {1}{1+x^2}}\]

(Source code, png, hires.png, pdf)

../_images/arctan.png

Rectified Linear Units, ReLU

\[\begin{split}f(x) = \left\{ \begin{array}{lll} 0 & for & x \leq 0 \\ x & for & x > 0 \end{array} \right.\end{split}\]
\[\begin{split}f'(x) = \left\{ \begin{array}{lll} 0 & for & x \leq 0 \\ 1 & for & x > 0 \end{array} \right.\end{split}\]

(Source code, png, hires.png, pdf)

../_images/relu.png

Leaky Rectified Linear Units, Leaky ReLU

\[\begin{split}f(x) = \left\{ \begin{array}{lll} ax & for & x \leq 0 \\ x & for & x > 0 \end{array} \right.\end{split}\]
\[\begin{split}f'(x) = \left\{ \begin{array}{lll} a & for & x \leq 0 \\ 1 & for & x > 0 \end{array} \right.\end{split}\]

(Source code, png, hires.png, pdf)

../_images/leaky_relu.png

Exponential Linear Units, ELU

\[\begin{split}f(x) = \left\{ \begin{array}{lll} a(e^x-1) & for & x \leq 0 \\ x & for & x > 0 \end{array} \right.\end{split}\]
\[\begin{split}f'(x) = \left\{ \begin{array}{lll} f(x)+a & for & x \leq 0 \\ 1 & for & x > 0 \end{array} \right.\end{split}\]

(Source code, png, hires.png, pdf)

../_images/elu.png

SoftPlus

\[f(x)=ln(1+e^x)\]
\[f'(x)={\frac {1}{1+e^{-x}}}\]

(Source code, png, hires.png, pdf)

../_images/softplus.png