The Hadamard product operates on identically shaped matrices and produces a third matrix of the same dimensions.
The Hadamard product operates on identically shaped matrices and produces a third matrix of the same dimensions.

In mathematics, the Hadamard product (also known as the element-wise product, entrywise product[1]: ch. 5  or Schur product[2]) is a binary operation that takes in two matrices of the same dimensions and returns a matrix of the multiplied corresponding elements. This operation can be thought as a "naive matrix multiplication" and is different from the matrix product. It is attributed to, and named after, either French-Jewish mathematician Jacques Hadamard or German-Jewish mathematician Issai Schur.

The Hadamard product is associative and distributive. Unlike the matrix product, it is also commutative.[3]

Definition

For two matrices A and B of the same dimension m × n, the Hadamard product (or [4][5][6]) is a matrix of the same dimension as the operands, with elements given by[3]

For matrices of different dimensions (m × n and p × q, where mp or nq), the Hadamard product is undefined.

For example, the Hadamard product for two arbitrary 2 × 3 matrices is:

Properties

The mixed-product property

where  is Kronecker product, assuming has the same dimensions of and with .

where  denotes face-splitting product.[13]

where  is column-wise Khatri–Rao product.

Schur product theorem

Main article: Schur product theorem

The Hadamard product of two positive-semidefinite matrices is positive-semidefinite.[3][8] This is known as the Schur product theorem,[7] after Russian mathematician Issai Schur. For two positive-semidefinite matrices A and B, it is also known that the determinant of their Hadamard product is greater than or equal to the product of their respective determinants:[8]

In programming languages

Hadamard multiplication is built into certain programming languages under various names. In MATLAB, GNU Octave, GAUSS and HP Prime, it is known as array multiplication, or in Julia broadcast multiplication, with the symbol .*.[14] In Fortran, R,[15] APL, J and Wolfram Language (Mathematica), it is done through simple multiplication operator * or ×, whereas the matrix product is done through the function matmul, %*%, +.×, +/ .* and the . operators, respectively. In Python with the NumPy numerical library, multiplication of array objects as a*b produces the Hadamard product, and multiplication as a@b produces the matrix product. With the SymPy symbolic library, multiplication of array objects as both a*b and a@b will produce the matrix product, the Hadamard product can be obtained with a.multiply_elementwise(b).[16] In C++, the Eigen library provides a cwiseProduct member function for the Matrix class (a.cwiseProduct(b)), while the Armadillo library uses the operator % to make compact expressions (a % b; a * b is a matrix product). R package matrixcalc introduces function hadamard.prod() for Hadamard Product of numeric matrices or vectors.

Applications

The Hadamard product appears in lossy compression algorithms such as JPEG. The decoding step involves an entry-for-entry product, in other words the Hadamard product.[citation needed]

In image processing, the Hadamard operator can be used for enhancing, suppressing or masking image regions. One matrix represents the original image, the other acts as weight or masking matrix.

It is used in the machine learning literature, for example, to describe the architecture of recurrent neural networks as GRUs or LSTMs.[17]

It is also used to study the statistical properties of random vectors and matrices. [18][19]

Analogous operations

Other Hadamard operations are also seen in the mathematical literature,[20] namely the Hadamard root and Hadamard power (which are in effect the same thing because of fractional indices), defined for a matrix such that:

For

and for

The Hadamard inverse reads:[20]

A Hadamard division is defined as:[21][22]

The penetrating face product

The penetrating face product of matrices
The penetrating face product of matrices

According to the definition of V. Slyusar the penetrating face product of the p×g matrix and n-dimensional matrix (n > 1) with p×g blocks () is a matrix of size of the form:[23]

Example

If

then

Main properties

[23]

where denotes the face-splitting product of matrices,

where is a vector.

Applications

The penetrating face product is used in the tensor-matrix theory of digital antenna arrays.[23] This operation can also be used in artificial neural network models, specifically convolutional layers.[24]

See also

References

  1. ^ a b Horn, Roger A.; Johnson, Charles R. (2012). Matrix analysis. Cambridge University Press.
  2. ^ Davis, Chandler (1962). "The norm of the Schur product operation". Numerische Mathematik. 4 (1): 343–44. doi:10.1007/bf01386329. S2CID 121027182.
  3. ^ a b c Million, Elizabeth (April 12, 2007). "The Hadamard Product" (PDF). buzzard.ups.edu. Retrieved September 6, 2020.
  4. ^ "Hadamard product - Machine Learning Glossary". machinelearning.wtf.
  5. ^ "linear algebra - What does a dot in a circle mean?". Mathematics Stack Exchange.
  6. ^ "Element-wise (or pointwise) operations notation?". Mathematics Stack Exchange.
  7. ^ a b Million, Elizabeth. "The Hadamard Product" (PDF). Retrieved 2 January 2012.
  8. ^ a b c Styan, George P. H. (1973), "Hadamard Products and Multivariate Statistical Analysis", Linear Algebra and Its Applications, 6: 217–240, doi:10.1016/0024-3795(73)90023-2, hdl:10338.dmlcz/102190
  9. ^ Liu, Shuangzhe; Trenkler, Götz (2008). "Hadamard, Khatri-Rao, Kronecker and other matrix products". International Journal of Information and Systems Sciences. 4 (1): 160–177.
  10. ^ Liu, Shuangzhe; Leiva, Víctor; Zhuang, Dan; Ma, Tiefeng; Figueroa-Zúñiga, Jorge I. (2022). "Matrix differential calculus with applications in the multivariate linear model and its diagnostics". Journal of Multivariate Analysis. 188: 104849. doi:10.1016/j.jmva.2021.104849. S2CID 239598156.
  11. ^ Hiai, Fumio; Lin, Minghua (February 2017). "On an eigenvalue inequality involving the Hadamard product". Linear Algebra and Its Applications. 515: 313–320. doi:10.1016/j.laa.2016.11.017.
  12. ^ "Project" (PDF). buzzard.ups.edu. 2007. Retrieved 2019-12-18.
  13. ^ Slyusar, V. I. (1998). "End products in matrices in radar applications" (PDF). Radioelectronics and Communications Systems. 41 (3): 50–53.
  14. ^ "Arithmetic Operators + - * / \ ^ ' -". MATLAB documentation. MathWorks. Archived from the original on 24 April 2012. Retrieved 2 January 2012.
  15. ^ "Matrix multiplication". An Introduction to R. The R Project for Statistical Computing. 16 May 2013. Retrieved 24 August 2013.
  16. ^ "Common Matrices — SymPy 1.9 documentation".
  17. ^ Sak, Haşim; Senior, Andrew; Beaufays, Françoise (2014-02-05). "Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition". arXiv:1402.1128 [cs.NE].
  18. ^ Neudecker, Heinz; Liu, Shuangzhe; Polasek, Wolfgang (1995). "The Hadamard product and some of its applications in statistics". Statistics. 26 (4): 365–373. doi:10.1080/02331889508802503.
  19. ^ Neudecker, Heinz; Liu, Shuangzhe (2001). "Some statistical properties of Hadamard products of random matrices". Statistical Papers. 42 (4): 475–487. doi:10.1007/s003620100074. S2CID 121385730.
  20. ^ a b Reams, Robert (1999). "Hadamard inverses, square roots and products of almost semidefinite matrices". Linear Algebra and Its Applications. 288: 35–43. doi:10.1016/S0024-3795(98)10162-3.
  21. ^ Wetzstein, Gordon; Lanman, Douglas; Hirsch, Matthew; Raskar, Ramesh. "Supplementary Material: Tensor Displays: Compressive Light Field Synthesis using Multilayer Displays with Directional Backlighting" (PDF). MIT Media Lab.
  22. ^ Cyganek, Boguslaw (2013). Object Detection and Recognition in Digital Images: Theory and Practice. John Wiley & Sons. p. 109. ISBN 9781118618363.
  23. ^ a b c Slyusar, V. I. (March 13, 1998). "A Family of Face Products of Matrices and its properties" (PDF). Cybernetics and Systems Analysis C/C of Kibernetika I Sistemnyi Analiz. 1999. 35 (3): 379–384. doi:10.1007/BF02733426. S2CID 119661450.
  24. ^ Ha D., Dai A.M., Le Q.V. (2017). "HyperNetworks". The International Conference on Learning Representations (ICLR) 2017. – Toulon, 2017.: Page 6. arXiv:1609.09106.((cite journal)): CS1 maint: multiple names: authors list (link)