Machine learning for Biomedical Engineers¶
Table of Contents
Activation functions¶
Binary Step¶
\[\begin{split}f(x) = \left\{
\begin{array}{lll}
0 & for & x < x_{min} \\
mx+b & for & x_{min} \leq x \leq x_{max} \\
1 & for & x > x_{max}
\end{array}
\right.\end{split}\]
\[\begin{split}f'(x) = \left\{
\begin{array}{lll}
0 & for & x \neq 0 \\
? & for & x = 0
\end{array}
\right.\end{split}\]
(Source code, png, hires.png, pdf)

Piecewise Linear¶
\[\begin{split}f(x) = \left\{
\begin{array}{lll}
0 & for & x < x_{min} \\
mx+b & for & x_{min} \leq x \leq x_{max} \\
1 & for & x > x_{max}
\end{array}
\right.\end{split}\]
\[\begin{split}f'(x) = \left\{
\begin{array}{lll}
0 & for & x < x_{min} \\
m & for & x_{min} \leq x \leq x_{max} \\
0 & for & x > x_{max}
\end{array}
\right.\end{split}\]
(Source code, png, hires.png, pdf)

Bipolar¶
\[\begin{split}f(x) = \left\{
\begin{array}{lll}
-1 & for & x \leq 0 \\
1 & for & x > 0
\end{array}
\right.\end{split}\]
\[\begin{split}f'(x) = \left\{
\begin{array}{lll}
0 & for & x \neq 0 \\
? & for & x = 0
\end{array}
\right.\end{split}\]
(Source code, png, hires.png, pdf)

Bipolar Sigmoid¶
\[f(x)={\frac {1-e^{-x}}{1+e^{-x}}}\]
\[f'(x)={\frac {2e^x}{(e^x+1)^2}}\]
(Source code, png, hires.png, pdf)

Hyperbolic Tangent, TanH¶
\[f(x)={\frac {2}{1+e^{-2x}}}-1\]
\[f'(x)=1-f(x)^2\]
(Source code, png, hires.png, pdf)

Arctangent, ArcTan¶
\[f(x)=tan^{-1}(x)\]
\[f'(x)={\frac {1}{1+x^2}}\]
(Source code, png, hires.png, pdf)

Rectified Linear Units, ReLU¶
\[\begin{split}f(x) = \left\{
\begin{array}{lll}
0 & for & x \leq 0 \\
x & for & x > 0
\end{array}
\right.\end{split}\]
\[\begin{split}f'(x) = \left\{
\begin{array}{lll}
0 & for & x \leq 0 \\
1 & for & x > 0
\end{array}
\right.\end{split}\]
(Source code, png, hires.png, pdf)

Leaky Rectified Linear Units, Leaky ReLU¶
\[\begin{split}f(x) = \left\{
\begin{array}{lll}
ax & for & x \leq 0 \\
x & for & x > 0
\end{array}
\right.\end{split}\]
\[\begin{split}f'(x) = \left\{
\begin{array}{lll}
a & for & x \leq 0 \\
1 & for & x > 0
\end{array}
\right.\end{split}\]
(Source code, png, hires.png, pdf)

Exponential Linear Units, ELU¶
\[\begin{split}f(x) = \left\{
\begin{array}{lll}
a(e^x-1) & for & x \leq 0 \\
x & for & x > 0
\end{array}
\right.\end{split}\]
\[\begin{split}f'(x) = \left\{
\begin{array}{lll}
f(x)+a & for & x \leq 0 \\
1 & for & x > 0
\end{array}
\right.\end{split}\]
(Source code, png, hires.png, pdf)
