An AI accelerator is a class of specialized hardware accelerator[1] or computer system[2][3] designed to accelerate artificial intelligence and machine learning applications, including artificial neural networks and machine vision. Typical applications include algorithms for robotics, Internet of Things, and other data-intensive or sensor-driven tasks.[4] They are often manycore designs and generally focus on low-precision arithmetic, novel dataflow architectures or in-memory computing capability. As of 2018, a typical AI integrated circuit chip contains billions of MOSFET transistors.[5] A number of vendor-specific terms exist for devices in this category, and it is an emerging technology without a dominant design.

History

Computer systems have frequently complemented the CPU with special-purpose accelerators for specialized tasks, known as coprocessors. Notable application-specific hardware units include video cards for graphics, sound cards, graphics processing units and digital signal processors. As deep learning and artificial intelligence workloads rose in prominence in the 2010s, specialized hardware units were developed or adapted from existing products to accelerate these tasks. Benchmarks such as MLPerf may be used to evaluate the performance of AI accelerators.[6]

Early attempts

First attempts like Intel's ETANN 80170NX[7] incorporated analog circuits to compute neural functions. Later all-digital chips like the Nestor/Intel Ni1000 followed. As early as 1993, digital signal processors were used as neural network accelerators to accelerate optical character recognition software.[8] Already in 1988, Wei Zhang et al. discussed fast optical implementations of convolutional neural networks for alphabet recognition.[9][10] In the 1990s, there were also attempts to create parallel high-throughput systems for workstations aimed at various applications, including neural network simulations.[11][12][13] FPGA-based accelerators were also first explored in the 1990s for both inference[14] and training.[15] Smartphones began incorporating AI accelerators starting with the Qualcomm Snapdragon 820 in 2015.[16][17]

Heterogeneous computing

Main article: Heterogeneous computing

Heterogeneous computing incorporates many specialized processors in a single system, or a single chip, each optimized for a specific type of task. Architectures such as the Cell microprocessor[18] have features significantly overlapping with AI accelerators including: support for packed low precision arithmetic, dataflow architecture, and prioritizing throughput over latency. The Cell microprocessor has been applied to a number of tasks[19][20][21] including AI.[22][23][24]

In the 2000s, CPUs also gained increasingly wide SIMD units, driven by video and gaming workloads; as well as support for packed low-precision data types.[25] Due to increasing performance of CPUs, they are also used for running AI workloads. CPUs are superior for DNNs with small or medium-scale parallelism, for sparse DNNs and in low-batch-size scenarios.

Use of GPU

Graphics processing units or GPUs are specialized hardware for the manipulation of images and calculation of local image properties. The mathematical basis of neural networks and image manipulation are similar, embarrassingly parallel tasks involving matrices, leading GPUs to become increasingly used for machine learning tasks.[26][27][28] As of 2016, GPUs are popular for AI work, and they continue to evolve in a direction to facilitate deep learning, both for training[29] and inference in devices such as self-driving cars.[30] GPU developers such as Nvidia NVLink are developing additional connective capability for the kind of dataflow workloads AI benefits from.[31] As GPUs have been increasingly applied to AI acceleration, GPU manufacturers have incorporated neural network-specific hardware to further accelerate these tasks.[32][33] Tensor cores are intended to speed up the training of neural networks.[33]

Use of FPGAs

Deep learning frameworks are still evolving, making it hard to design custom hardware. Reconfigurable devices such as field-programmable gate arrays (FPGA) make it easier to evolve hardware, frameworks, and software alongside each other.[34][14][15][35]

Microsoft has used FPGA chips to accelerate inference.[36]

Emergence of dedicated AI accelerator ASICs

While GPUs and FPGAs perform far better than CPUs for AI-related tasks, a factor of up to 10 in efficiency[37][38] may be gained with a more specific design, via an application-specific integrated circuit (ASIC).[citation needed] These accelerators employ strategies such as optimized memory use[citation needed] and the use of lower precision arithmetic to accelerate calculation and increase throughput of computation.[39][40] Some low-precision floating-point formats used for AI acceleration are half-precision and the bfloat16 floating-point format.[41][42][43][44][45][46][47] Companies such as Google, Qualcomm, Amazon, Apple, Facebook, AMD and Samsung are all designing their own AI ASICs.[48][49][50][51][52][53] Cerebras Systems has built a dedicated AI accelerator based on the largest processor in the industry, the second-generation Wafer Scale Engine (WSE-2), to support deep learning workloads.[54][55]

Ongoing research

In-memory computing architectures

This section needs expansion. You can help by adding to it. (October 2018)

In June 2017, IBM researchers announced an architecture in contrast to the Von Neumann architecture based on in-memory computing and phase-change memory arrays applied to temporal correlation detection, intending to generalize the approach to heterogeneous computing and massively parallel systems.[56] In October 2018, IBM researchers announced an architecture based on in-memory processing and modeled on the human brain's synaptic network to accelerate deep neural networks.[57] The system is based on phase-change memory arrays.[58]

In-memory computing with analog resistive memories

In 2019, researchers from Politecnico di Milano found a way to solve systems of linear equations in a few tens of nanoseconds via a single operation. Their algorithm is based on in-memory computing with analog resistive memories which performs with high efficiencies of time and energy, via conducting matrix–vector multiplication in one step using Ohm's law and Kirchhoff's law. The researchers showed that a feedback circuit with cross-point resistive memories can solve algebraic problems such as systems of linear equations, matrix eigenvectors, and differential equations in just one step. Such an approach improves computational times drastically in comparison with digital algorithms.[59]

Atomically thin semiconductors

In 2020, Marega et al. published experiments with a large-area active channel material for developing logic-in-memory devices and circuits based on floating-gate field-effect transistors (FGFETs).[60] Such atomically thin semiconductors are considered promising for energy-efficient machine learning applications, where the same basic device structure is used for both logic operations and data storage. The authors used two-dimensional materials such as semiconducting molybdenum disulfide.[60]

Integrated photonic tensor core

In 2021, J. Feldmann et al. proposed an integrated photonic hardware accelerator for parallel convolutional processing.[61] The authors identify two key advantages of integrated photonics over its electronic counterparts: (1) massively parallel data transfer through wavelength division multiplexing in conjunction with frequency combs, and (2) extremely high data modulation speeds.[61] Their system can execute trillions of multiply-accumulate operations per second, indicating the potential of integrated photonics in data-heavy AI applications.[61]

Nomenclature

As of 2016, the field is still in flux and vendors are pushing their own marketing term for what amounts to an "AI accelerator", in the hope that their designs and APIs will become the dominant design. There is no consensus on the boundary between these devices, nor the exact form they will take; however several examples clearly aim to fill this new space, with a fair amount of overlap in capabilities.

In the past when consumer graphics accelerators emerged, the industry eventually adopted Nvidia's self-assigned term, "the GPU",[62] as the collective noun for "graphics accelerators", which had taken many forms before settling on an overall pipeline implementing a model presented by Direct3D.

All models of Intel Meteor Lake processors have a Versatile Processor Unit (VPU) built-in for accelerating inference for computer vision and deep learning.[63]

Potential applications

See also

References

  1. ^ "Intel unveils Movidius Compute Stick USB AI Accelerator". July 21, 2017. Archived from the original on August 11, 2017. Retrieved August 11, 2017.
  2. ^ "Inspurs unveils GX4 AI Accelerator". June 21, 2017.
  3. ^ Wiggers, Kyle (November 6, 2019) [2019], Neural Magic raises $15 million to boost AI inferencing speed on off-the-shelf processors, archived from the original on March 6, 2020, retrieved March 14, 2020
  4. ^ "Google Designing AI Processors". Google using its own AI accelerators.
  5. ^ "13 Sextillion & Counting: The Long & Winding Road to the Most Frequently Manufactured Human Artifact in History". Computer History Museum. April 2, 2018. Retrieved July 28, 2019.
  6. ^ "Nvidia claims 'record performance' for Hopper MLPerf debut".
  7. ^ John C. Dvorak: Intel's 80170 chip has the theoretical intelligence of a cockroach in PC Magazine Volume 9 Number 10 (May 1990), p. 77, [1], retrieved May 16, 2021
  8. ^ "convolutional neural network demo from 1993 featuring DSP32 accelerator". YouTube.
  9. ^ Zhang, Wei (1988). "Shift-invariant pattern recognition neural network and its optical architecture". Proceedings of Annual Conference of the Japan Society of Applied Physics.
  10. ^ Zhang, Wei (1990). "Parallel distributed processing model with local space-invariant interconnections and its optical architecture". Applied Optics. 29 (32): 4790–7. Bibcode:1990ApOpt..29.4790Z. doi:10.1364/AO.29.004790. PMID 20577468.
  11. ^ "design of a connectionist network supercomputer".
  12. ^ "The end of general purpose computers (not)". YouTube.This presentation covers a past attempt at neural net accelerators, notes the similarity to the modern SLI GPGPU processor setup, and argues that general purpose vector accelerators are the way forward (in relation to RISC-V hwacha project. Argues that NN's are just dense and sparse matrices, one of several recurring algorithms)
  13. ^ Ramacher, U.; Raab, W.; Hachmann, J.A.U.; Beichter, J.; Bruls, N.; Wesseling, M.; Sicheneder, E.; Glass, J.; Wurz, A.; Manner, R. (1995). Proceedings of 9th International Parallel Processing Symposium. pp. 774–781. CiteSeerX 10.1.1.27.6410. doi:10.1109/IPPS.1995.395862. ISBN 978-0-8186-7074-9. S2CID 16364797.
  14. ^ a b "Space Efficient Neural Net Implementation".
  15. ^ a b Gschwind, M.; Salapura, V.; Maischberger, O. (1996). "A Generic Building Block for Hopfield Neural Networks with On-Chip Learning". 1996 IEEE International Symposium on Circuits and Systems. Circuits and Systems Connecting the World. ISCAS 96. pp. 49–52. doi:10.1109/ISCAS.1996.598474. ISBN 0-7803-3073-0. S2CID 17630664.
  16. ^ "Qualcomm Helps Make Your Mobile Devices Smarter With New Snapdragon Machine Learning Software Development Kit". Qualcomm.
  17. ^ Rubin, Ben Fox. "Qualcomm's Zeroth platform could make your smartphone much smarter". CNET. Retrieved September 28, 2021.
  18. ^ Gschwind, Michael; Hofstee, H. Peter; Flachs, Brian; Hopkins, Martin; Watanabe, Yukio; Yamazaki, Takeshi (2006). "Synergistic Processing in Cell's Multicore Architecture". IEEE Micro. 26 (2): 10–24. doi:10.1109/MM.2006.41. S2CID 17834015.
  19. ^ De Fabritiis, G. (2007). "Performance of Cell processor for biomolecular simulations". Computer Physics Communications. 176 (11–12): 660–664. arXiv:physics/0611201. Bibcode:2007CoPhC.176..660D. doi:10.1016/j.cpc.2007.02.107. S2CID 13871063.
  20. ^ Video Processing and Retrieval on Cell architecture. CiteSeerX 10.1.1.138.5133.
  21. ^ Benthin, Carsten; Wald, Ingo; Scherbaum, Michael; Friedrich, Heiko (2006). 2006 IEEE Symposium on Interactive Ray Tracing. pp. 15–23. CiteSeerX 10.1.1.67.8982. doi:10.1109/RT.2006.280210. ISBN 978-1-4244-0693-7. S2CID 1198101.
  22. ^ "Development of an artificial neural network on a heterogeneous multicore architecture to predict a successful weight loss in obese individuals" (PDF). Archived from the original (PDF) on August 30, 2017. Retrieved November 14, 2017.
  23. ^ Kwon, Bomjun; Choi, Taiho; Chung, Heejin; Kim, Geonho (2008). 2008 5th IEEE Consumer Communications and Networking Conference. pp. 1030–1034. doi:10.1109/ccnc08.2007.235. ISBN 978-1-4244-1457-4. S2CID 14429828.
  24. ^ Duan, Rubing; Strey, Alfred (2008). Euro-Par 2008 – Parallel Processing. Lecture Notes in Computer Science. Vol. 5168. pp. 665–675. doi:10.1007/978-3-540-85451-7_71. ISBN 978-3-540-85450-0.
  25. ^ "Improving the performance of video with AVX". February 8, 2012.
  26. ^ "microsoft research/pixel shaders/MNIST".
  27. ^ "How GPU came to be used for general computation".
  28. ^ "ImageNet Classification with Deep Convolutional Neural Networks" (PDF).
  29. ^ "nvidia driving the development of deep learning". May 17, 2016.
  30. ^ "Nvidia introduces supercomputer for self driving cars". January 6, 2016.
  31. ^ "how nvlink will enable faster easier multi GPU computing". November 14, 2014.
  32. ^ "A Survey on Optimized Implementation of Deep Learning Models on the NVIDIA Jetson Platform", 2019
  33. ^ a b Harris, Mark (May 11, 2017). "CUDA 9 Features Revealed: Volta, Cooperative Groups and More". Retrieved August 12, 2017.
  34. ^ Sefat, Md Syadus; Aslan, Semih; Kellington, Jeffrey W; Qasem, Apan (August 2019). "Accelerating HotSpots in Deep Neural Networks on a CAPI-Based FPGA". 2019 IEEE 21st International Conference on High Performance Computing and Communications; IEEE 17th International Conference on Smart City; IEEE 5th International Conference on Data Science and Systems (HPCC/SmartCity/DSS). pp. 248–256. doi:10.1109/HPCC/SmartCity/DSS.2019.00048. ISBN 978-1-7281-2058-4. S2CID 203656070.
  35. ^ "FPGA Based Deep Learning Accelerators Take on ASICs". The Next Platform. August 23, 2016. Retrieved September 7, 2016.
  36. ^ "Project Brainwave". Microsoft Research. Retrieved June 16, 2020.
  37. ^ "Google boosts machine learning with its Tensor Processing Unit". May 19, 2016. Retrieved September 13, 2016.
  38. ^ "Chip could bring deep learning to mobile devices". www.sciencedaily.com. February 3, 2016. Retrieved September 13, 2016.
  39. ^ "Deep Learning with Limited Numerical Precision" (PDF).
  40. ^ Rastegari, Mohammad; Ordonez, Vicente; Redmon, Joseph; Farhadi, Ali (2016). "XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks". arXiv:1603.05279 [cs.CV].
  41. ^ Khari Johnson (May 23, 2018). "Intel unveils Nervana Neural Net L-1000 for accelerated AI training". VentureBeat. Retrieved May 23, 2018. ...Intel will be extending bfloat16 support across our AI product lines, including Intel Xeon processors and Intel FPGAs.
  42. ^ Michael Feldman (May 23, 2018). "Intel Lays Out New Roadmap for AI Portfolio". TOP500 Supercomputer Sites. Retrieved May 23, 2018. Intel plans to support this format across all their AI products, including the Xeon and FPGA lines
  43. ^ Lucian Armasu (May 23, 2018). "Intel To Launch Spring Crest, Its First Neural Network Processor, In 2019". Tom's Hardware. Retrieved May 23, 2018. Intel said that the NNP-L1000 would also support bfloat16, a numerical format that's being adopted by all the ML industry players for neural networks. The company will also support bfloat16 in its FPGAs, Xeons, and other ML products. The Nervana NNP-L1000 is scheduled for release in 2019.
  44. ^ "Available TensorFlow Ops | Cloud TPU | Google Cloud". Google Cloud. Retrieved May 23, 2018. This page lists the TensorFlow Python APIs and graph operators available on Cloud TPU.
  45. ^ Elmar Haußmann (April 26, 2018). "Comparing Google's TPUv2 against Nvidia's V100 on ResNet-50". RiseML Blog. Archived from the original on April 26, 2018. Retrieved May 23, 2018. For the Cloud TPU, Google recommended we use the bfloat16 implementation from the official TPU repository with TensorFlow 1.7.0. Both the TPU and GPU implementations make use of mixed-precision computation on the respective architecture and store most tensors with half-precision.
  46. ^ Tensorflow Authors (February 28, 2018). "ResNet-50 using BFloat16 on TPU". Google. Retrieved May 23, 2018.[permanent dead link]
  47. ^ Joshua V. Dillon; Ian Langmore; Dustin Tran; Eugene Brevdo; Srinivas Vasudevan; Dave Moore; Brian Patton; Alex Alemi; Matt Hoffman; Rif A. Saurous (November 28, 2017). TensorFlow Distributions (Report). arXiv:1711.10604. Bibcode:2017arXiv171110604D. Accessed May 23, 2018. All operations in TensorFlow Distributions are numerically stable across half, single, and double floating-point precisions (as TensorFlow dtypes: tf.bfloat16 (truncated floating point), tf.float16, tf.float32, tf.float64). Class constructors have a validate_args flag for numerical asserts
  48. ^ "Google Reveals a Powerful New AI Chip and Supercomputer". MIT Technology Review. Retrieved July 27, 2021.
  49. ^ "What to Expect From Apple's Neural Engine in the A11 Bionic SoC – ExtremeTech". www.extremetech.com. Retrieved July 27, 2021.
  50. ^ "Facebook has a new job posting calling for chip designers". April 19, 2018.[permanent dead link]
  51. ^ "Facebook joins Amazon and Google in AI chip race". Financial Times. February 18, 2019.
  52. ^ Amadeo, Ron (May 11, 2021). "Samsung and AMD will reportedly take on Apple's M1 SoC later this year". Ars Technica. Retrieved July 28, 2021.
  53. ^ Smith, Ryan. "The AI Race Expands: Qualcomm Reveals "Cloud AI 100" Family of Datacenter AI Inference Accelerators for 2020". www.anandtech.com. Retrieved September 28, 2021.
  54. ^ Woodie, Alex (November 1, 2021). "Cerebras Hits the Accelerator for Deep Learning Workloads". Datanami. Retrieved August 3, 2022.
  55. ^ "Cerebras launches new AI supercomputing processor with 2.6 trillion transistors". VentureBeat. April 20, 2021. Retrieved August 3, 2022.
  56. ^ Abu Sebastian; Tomas Tuma; Nikolaos Papandreou; Manuel Le Gallo; Lukas Kull; Thomas Parnell; Evangelos Eleftheriou (2017). "Temporal correlation detection using computational phase-change memory". Nature Communications. 8 (1): 1115. arXiv:1706.00511. Bibcode:2017NatCo...8.1115S. doi:10.1038/s41467-017-01481-9. PMC 5653661. PMID 29062022.
  57. ^ "A new brain-inspired architecture could improve how computers handle data and advance AI". American Institute of Physics. October 3, 2018. Retrieved October 5, 2018.
  58. ^ Carlos Ríos; Nathan Youngblood; Zengguang Cheng; Manuel Le Gallo; Wolfram H.P. Pernice; C. David Wright; Abu Sebastian; Harish Bhaskaran (2018). "In-memory computing on a photonic platform". Science Advances. 5 (2): eaau5759. arXiv:1801.06228. Bibcode:2019SciA....5.5759R. doi:10.1126/sciadv.aau5759. PMC 6377270. PMID 30793028. S2CID 7637801.
  59. ^ Zhong Sun; Giacomo Pedretti; Elia Ambrosi; Alessandro Bricalli; Wei Wang; Daniele Ielmini (2019). "Solving matrix equations in one step with cross-point resistive arrays". Proceedings of the National Academy of Sciences. 116 (10): 4123–4128. Bibcode:2019PNAS..116.4123S. doi:10.1073/pnas.1815682116. PMC 6410822. PMID 30782810.
  60. ^ a b Marega, Guilherme Migliato; Zhao, Yanfei; Avsar, Ahmet; Wang, Zhenyu; Tripati, Mukesh; Radenovic, Aleksandra; Kis, Anras (2020). "Logic-in-memory based on an atomically thin semiconductor". Nature. 587 (2): 72–77. Bibcode:2020Natur.587...72M. doi:10.1038/s41586-020-2861-0. PMC 7116757. PMID 33149289.
  61. ^ a b c Feldmann, J.; Youngblood, N.; Karpov, M.; et al. (2021). "Parallel convolutional processing using an integrated photonic tensor". Nature. 589 (2): 52–58. arXiv:2002.00281. doi:10.1038/s41586-020-03070-1. PMID 33408373. S2CID 211010976.
  62. ^ "NVIDIA launches the World's First Graphics Processing Unit, the GeForce 256". Archived from the original on February 27, 2016.
  63. ^ "Intel to Bring a 'VPU' Processor Unit to 14th Gen Meteor Lake Chips". PCMAG.
  64. ^ "Development of a machine vision system for weed control using precision chemical application" (PDF). University of Florida. CiteSeerX 10.1.1.7.342. Archived from the original (PDF) on June 23, 2010.
  65. ^ "Self-Driving Cars Technology & Solutions from NVIDIA Automotive". NVIDIA.
  66. ^ "movidius powers worlds most intelligent drone". March 16, 2016.
  67. ^ "Qualcomm Research brings server class machine learning to everyday devices–making them smarter [VIDEO]". October 2015.

External links

Hardware acceleration
Categories