softmax

$S_i=\frac{e^i}{\sum_je^j}$

elu

$f(x)=\begin{cases} x & x \geq 0 \\ \alpha(e^x-1) & x < 0 \end{cases}$

$f(x)=\lambda\begin{cases} x & x \geq 0 \\ \alpha(e^x-1) & x < 0 \end{cases}$

$f(x)=ln(1+e^x)$

$f(x)=\frac{x}{1+|x|}$

$f(x)=max(0,x)$

$tanhx=\frac{sinhx}{coshx}=\frac{e^x-e^{-x}}{e^x+e^{-x}}$

$f(x)=\frac{1}{1+e^{-x}}$

$f(x)=\begin{cases} 0 & x < -2.5 \\ 0.2x+0.5 & -2.5 \leq x \leq 2.5\\ 1 & x > 2.5 \end{cases}$

Input tensor, unchanged.