Plot of the ReLU rectifier (blue) and GELU (green) functions near x = 0
Plot of the ReLU rectifier (blue) and GELU (green) functions near x = 0

In the context of artificial neural networks, the rectifier or ReLU (Rectified Linear Unit) activation function[1][2] is an activation function defined as the positive part of its argument:

where x is the input to a neuron. This is also known as a ramp function and is analogous to half-wave rectification in electrical engineering.

This activation function started showing up in the context of visual feature extraction in hierarchical neural networks starting in the late 1960s.[3][4] It was later argued that it has strong biological motivations and mathematical justifications.[5][6] In 2011 it was found to enable better training of deeper networks,[7] compared to the widely used activation functions prior to 2011, e.g., the logistic sigmoid (which is inspired by probability theory; see logistic regression) and its more practical[8] counterpart, the hyperbolic tangent. The rectifier is, as of 2017, the most popular activation function for deep neural networks.[9]

Rectified linear units find applications in computer vision[7] and speech recognition[10][11] using deep neural nets and computational neuroscience.[12][13][14]


Rectifying activation functions were used to separate specific excitation and unspecific inhibition in the neural abstraction pyramid, which was trained in a supervised way to learn several computer vision tasks.[15] In 2011,[7] the use of the rectifier as a non-linearity has been shown to enable training deep supervised neural networks without requiring unsupervised pre-training. Rectified linear units, compared to sigmoid function or similar activation functions, allow faster and effective training of deep neural architectures on large and complex datasets.

Potential problems


Piecewise-Linear Variants

Leaky ReLU

Leaky ReLUs allow a small, positive gradient when the unit is not active.[11]

Parametric ReLU

Parametric ReLUs (PReLUs) take this idea further by making the coefficient of leakage into a parameter that is learned along with the other neural-network parameters.[16]

Note that for a ≤ 1, this is equivalent to

and thus has a relation to "maxout" networks.[16]

Other Non-linear variants

Gaussian Error Linear Unit (GELU)

GELU is a smooth approximation to the rectifier. It has a non-monotonic “bump” when x < 0, and it serves as the default activation for models such as BERT.[17]


where Φ(x) is the cumulative distribution function of the standard normal distribution.

This activation function is illustrated in the figure at the start of this article.


Main article: swish function

The SiLU (Sigmoid Linear Unit) or swish function[18] is another smooth approximation first coined in the GELU paper. [17]

where is the sigmoid function.


A smooth approximation to the rectifier is the analytic function

which is called the softplus[19][7] or SmoothReLU function.[20] For large negative it is roughly so just above 0, while for large positive it is roughly so just above .

A sharpness parameter may be included:

The derivative of softplus is the logistic function. Starting from the parametric version,

The logistic sigmoid function is a smooth approximation of the derivative of the rectifier, the Heaviside step function.

The multivariable generalization of single-variable softplus is the LogSumExp with the first argument set to zero:

The LogSumExp function is

and its gradient is the softmax; the softmax with the first argument set to zero is the multivariable generalization of the logistic function. Both LogSumExp and softmax are used in machine learning.


Exponential linear units try to make the mean activations closer to zero, which speeds up learning. It has been shown that ELUs can obtain higher classification accuracy than ReLUs.[21]

where is a hyper-parameter to be tuned, and is a constraint.

The ELU can be viewed as a smoothed version of a shifted ReLU (SReLU), which has the form given the same interpretation of .


The mish function could also be used as a smooth approximation of the rectifier.[18] It is defined as

where is the hyperbolic tangent and is the softplus function.

Mish is non-monotonic and self-gated.[22] It was inspired by Swish, itself a variant of ReLU.[22]

See also


  1. ^ Brownlee, Jason (8 January 2019). "A Gentle Introduction to the Rectified Linear Unit (ReLU)". Machine Learning Mastery. Retrieved 8 April 2021.
  2. ^ Liu, Danqing (30 November 2017). "A Practical Guide to ReLU". Medium. Retrieved 8 April 2021.
  3. ^ Fukushima, K. (1969). "Visual feature extraction by a multilayered network of analog threshold elements". IEEE Transactions on Systems Science and Cybernetics. 5 (4): 322–333. doi:10.1109/TSSC.1969.300225.
  4. ^ Fukushima, K.; Miyake, S. (1982). "Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition". In Competition and Cooperation in Neural Nets. Lecture Notes in Biomathematics. Springer. 45: 267–285. doi:10.1007/978-3-642-46466-9_18. ISBN 978-3-540-11574-8.
  5. ^ Hahnloser, R.; Sarpeshkar, R.; Mahowald, M. A.; Douglas, R. J.; Seung, H. S. (2000). "Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit". Nature. 405 (6789): 947–951. Bibcode:2000Natur.405..947H. doi:10.1038/35016072. PMID 10879535. S2CID 4399014.
  6. ^ Hahnloser, R.; Seung, H. S. (2001). Permitted and Forbidden Sets in Symmetric Threshold-Linear Networks. NIPS 2001.
  7. ^ a b c d e Xavier Glorot, Antoine Bordes and Yoshua Bengio (2011). Deep sparse rectifier neural networks (PDF). AISTATS. Rectifier and softplus activation functions. The second one is a smooth version of the first.((cite conference)): CS1 maint: uses authors parameter (link)
  8. ^ Yann LeCun, Leon Bottou, Genevieve B. Orr and Klaus-Robert Müller (1998). "Efficient BackProp" (PDF). In G. Orr; K. Müller (eds.). Neural Networks: Tricks of the Trade. Springer.((cite encyclopedia)): CS1 maint: uses authors parameter (link)
  9. ^ Ramachandran, Prajit; Barret, Zoph; Quoc, V. Le (October 16, 2017). "Searching for Activation Functions". arXiv:1710.05941 [cs.NE].
  10. ^ László Tóth (2013). Phone Recognition with Deep Sparse Rectifier Neural Networks (PDF). ICASSP.((cite conference)): CS1 maint: uses authors parameter (link)
  11. ^ a b Andrew L. Maas, Awni Y. Hannun, Andrew Y. Ng (2014). Rectifier Nonlinearities Improve Neural Network Acoustic Models.
  12. ^ Hansel, D.; van Vreeswijk, C. (2002). "How noise contributes to contrast invariance of orientation tuning in cat visual cortex". J. Neurosci. 22 (12): 5118–5128. doi:10.1523/JNEUROSCI.22-12-05118.2002. PMC 6757721. PMID 12077207.
  13. ^ Kadmon, Jonathan; Sompolinsky, Haim (2015-11-19). "Transition to Chaos in Random Neuronal Networks". Physical Review X. 5 (4): 041030. arXiv:1508.06486. Bibcode:2015PhRvX...5d1030K. doi:10.1103/PhysRevX.5.041030. S2CID 7813832.
  14. ^ Engelken, Rainer; Wolf, Fred; Abbott, L. F. (2020-06-03). "Lyapunov spectra of chaotic recurrent neural networks". arXiv:2006.02427 [nlin.CD].
  15. ^ Behnke, Sven (2003). Hierarchical Neural Networks for Image Interpretation. Lecture Notes in Computer Science. Vol. 2766. Springer. doi:10.1007/b11963. ISBN 978-3-540-40722-5. S2CID 1304548.
  16. ^ a b He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing; Sun, Jian (2015). "Delving Deep into Rectifiers: Surpassing Human-Level Performance on Image Net Classification". arXiv:1502.01852 [cs.CV].
  17. ^ a b Hendrycks, Dan; Gimpel, Kevin (2016). "Gaussian Error Linear Units (GELUs)". arXiv:1606.08415 [cs.LG].
  18. ^ a b Diganta Misra (23 Aug 2019), Mish: A Self Regularized Non-Monotonic Activation Function (PDF), arXiv:1908.08681v1, retrieved 26 March 2022
  19. ^ Dugas, Charles; Bengio, Yoshua; Bélisle, François; Nadeau, Claude; Garcia, René (2000-01-01). "Incorporating second-order functional knowledge for better option pricing" (PDF). Proceedings of the 13th International Conference on Neural Information Processing Systems (NIPS'00). MIT Press: 451–457. Since the sigmoid h has a positive first derivative, its primitive, which we call softplus, is convex.
  20. ^ "Smooth Rectifier Linear Unit (SmoothReLU) Forward Layer". Developer Guide for Intel Data Analytics Acceleration Library. 2017. Retrieved 2018-12-04.
  21. ^ Clevert, Djork-Arné; Unterthiner, Thomas; Hochreiter, Sepp (2015). "Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)". arXiv:1511.07289 [cs.LG].
  22. ^ a b Shaw, Sweta (2020-05-10). "Activation Functions Compared with Experiments". W&B. Retrieved 2022-07-11.