You do not have permission to edit this page, for the following reasons:

This IP address has been blocked from editing Wikipedia.
This does not affect your ability to read Wikipedia pages.
Most people who see this message have done nothing wrong. Some kinds of blocks restrict editing from specific service providers or telecom companies in response to recent abuse or vandalism, and can sometimes affect other users who are unrelated to that abuse. Review the information below for assistance if you do not believe that you have done anything wrong.

The IP address or range 44.199.0.0/16 has been blocked (disabled) by ‪Blablubbs‬ for the following reason(s):

The IP address that you are currently using has been blocked because it is believed to be a web host provider or colocation provider. To prevent abuse, web hosts and colocation providers may be blocked from editing Wikipedia.
You will not be able to edit Wikipedia using a web host or colocation provider because it hides your IP address, much like a proxy or VPN.
We recommend that you attempt to use another connection to edit. For example, if you use a proxy or VPN to connect to the internet, turn it off when editing Wikipedia. If you edit using a mobile connection, try using a Wi-Fi connection, and vice versa. If you are using a corporate internet connection, switch to a different Wi-Fi network. If you have a Wikipedia account, please log in.
If you do not have any other way to edit Wikipedia, you will need to request an IP block exemption.

How to appeal if you are confident that your connection does not use a colocation provider's IP address:
If you are confident that you are not using a web host, you may appeal this block by adding the following text on your talk page: ((unblock|reason=Caught by a colocation web host block but this host or IP is not a web host. My IP address is _______. Place any further information here. ~~~~)). You must fill in the blank with your IP address for this block to be investigated. Your IP address can be determined here. Alternatively, if you wish to keep your IP address private you can use the unblock ticket request system. There are several reasons you might be editing using the IP address of a web host or colocation provider (such as if you are using VPN software or a business network); please use this method of appeal only if you think your IP address is in fact not a web host or colocation provider.

Administrators: The IP block exemption user right should only be applied to allow users to edit using web host in exceptional circumstances, and requests should usually be directed to the functionaries team via email. If you intend to give the IPBE user right, a CheckUser needs to take a look at the account. This can be requested most easily at SPI Quick Checkuser Requests. Unblocking an IP or IP range with this template is highly discouraged without at least contacting the blocking administrator.

This block will expire on 12:56, 12 May 2024.

Even when blocked, you will usually still be able to edit your user talk page, as well as email administrators and other editors.

For information on how to proceed, please read the FAQ for blocked users and the guideline on block appeals. The guide to appealing blocks may also be helpful.

Other useful links: Blocking policy · Help:I have been blocked
Your IP address is in a range that has been blocked on all Wikimedia Foundation wikis. The block was made by ‪AntiCompositeNumber‬. The reason given is Open proxy/Webhost: Visit the FAQ if you are affected .
- Start of block: 13:55, 18 August 2022
- Expiry of block: 13:55, 18 May 2025
Your current IP address is 44.199.189.47. The blocked range is 44.199.0.0/16. Please include all above details in any queries you make. If you believe you were blocked by mistake, you can find additional information and instructions in the No open proxies global policy. Otherwise, to discuss the block please post a request for review on Meta-Wiki. You could also send an email to the stewards VRT queue at stewards@wikimedia.org including all above details.

You can view and copy the source of this page:

((Short description|Resilience of systems to component failures or errors))
((Use American English|date = February 2019))
((More citations needed|date=January 2008))

'''Fault tolerance''' is the ability of a [[system]] to maintain proper operation in the event of failures or faults in one or more of its components. Any decrease in operating quality is proportional to the severity of the failure, unlike a naively designed system in which even a small failure can lead to [[catastrophic failure|total breakdown]]. Fault tolerance is particularly sought after in [[high-availability]], [[mission critical|mission-critical]], or even [[life-critical system]]s. The ability to maintain functionality when portions of a system break down is referred to as '''graceful degradation'''.<ref>[http://scholarworks.umass.edu/cgi/viewcontent.cgi?article=1186&context=cs_faculty_pubs Adaptive Fault Tolerance and Graceful Degradation], Oscar González et al., 1997, University of Massachusetts - Amherst</ref>

A '''fault-tolerant design''' enables a system to continue its intended operation, possibly at a reduced level, rather than failing completely when some part of the system [[failure|fails]].<ref>Johnson, B. W. (1984). "[https://www.computer.org/csdl/mags/mi/1984/06/04071150.pdf Fault-Tolerant Microprocessor-Based Systems]", IEEE Micro, vol. 4, no. 6, pp. 6–21</ref> The term is most commonly used to describe [[computer system]]s designed to continue operation in the event of a partial failure, with perhaps a reduction in [[throughput]] or an increase in [[response time (technology)|response time]]. That is, the system as a whole is not stopped due to problems either in the [[computer hardware|hardware]] or the [[software]]. Non-computing examples include a motor vehicle designed to remain drivable if one of the tires is punctured, or a structure that retains its integrity despite damage caused by [[fatigue (material)|fatigue]], [[corrosion]], manufacturing flaws, or impact.

For an ''individual'' system, fault tolerance can be achieved by anticipating exceptional conditions and building the system to cope with them, and aiming for [[self-stabilization]] so that the system converges towards an error-free state. However, if the consequences of a system failure are catastrophic, or the cost of making it sufficiently reliable is very high, a better solution may be to use some form of duplication. In any case, if the consequence of a system failure is so catastrophic, the system must be able to use reversion to fall back to a safe mode. This is similar to roll-back recovery but can be a human action if humans are present in the loop.

== History ==
The first known fault-tolerant computer was [[SAPO (computer)|SAPO]], built in 1951 in [[Czechoslovakia]] by [[Antonín Svoboda (computer scientist)|Antonín Svoboda]].<ref name="computer-structures">((cite book|url=https://archive.org/details/computerstructur01siew|title=Computer Structures: Principles and Examples|author1=Daniel P. Siewiorek|author2=C. Gordon Bell|author3=Allen Newell|publisher=[[McGraw-Hill]]|year=1982|isbn=0-07-057302-6|url-access=registration))</ref>((rp|155)) Its basic design was [[Drum memory|magnetic drums]] connected via relays, with a voting method of [[RAM parity|memory error]] detection ([[triple modular redundancy]]). Several other machines were developed along this line, mostly for military use. Eventually, they separated into three distinct categories:

# Machines that would last a long time without any maintenance, such as the ones used on [[NASA]] [[space probe]]s and [[satellite]]s;
# Computers that were very dependable but required constant monitoring, such as those used to monitor and control [[nuclear power plant]]s or [[Collider|supercollider]] experiments; and
# Computers with a high amount of runtime that would be under heavy use, such as many of the supercomputers used by [[insurance companies]] for their [[probability]] monitoring.

Most of the development in the so-called LLNM (Long Life, No Maintenance) computing was done by NASA during the 1960s,<ref>((cite web|url=https://www.computer.org/csdl/proceedings/ftcsh/1995/7150/00/00532604.pdf|title=The STAR (Self-Testing And Repairing) Computer: An Investigation Of the Theory and Practice Of Fault-tolerant Computer Design|author1=Algirdas Avižienis|author2=George C. Gilley|author3=Francis P. Mathur|author4=David A. Rennels|author5=John A. Rohr|author6=David K. Rubin))</ref> in preparation for [[Project Apollo]] and other research aspects. NASA's first machine went into a [[space observatory]], and their second attempt, the JSTAR computer, was used in [[Voyager program|Voyager]]. This computer had a backup of memory arrays to use memory recovery methods and thus it was called the JPL Self-Testing-And-Repairing computer. It could detect its own errors and fix them or bring up redundant modules as needed. The computer is still working, as of early 2022.<ref>((Cite web|title=Voyager Mission state (more often than not at least three months out of date) |url=https://voyager.jpl.nasa.gov/mission/weekly-reports/|access-date=2022-04-01|website=NASA|language=en))</ref>

Hyper-dependable computers were pioneered mostly by [[aircraft]] manufacturers,<ref name="computer-structures"/>((rp|210)) [[nuclear power]] companies, and the [[rail transport in the United States|railroad industry]] in the United States. These entities needed computers with massive amounts of uptime that would [[Graceful failure|fail gracefully]] enough during a fault to allow continued operation, while relying on constant human monitoring of computer output to detect faults. Again, IBM developed the first computer of this kind for NASA for guidance of [[Saturn V]] rockets, but later on [[BNSF]], [[Unisys]], and [[General Electric]] built their own.<ref name="computer-structures"/>((rp|223))

In the 1970s, much work happened in the field.<ref>((cite journal |author-first1=Brian |author-last=Randell |author-link1=Brian Randell |author-first2=P.A. |author-last2=Lee |author-first3=P. C. |author-last3=Treleaven |title=Reliability Issues in Computing System Design | journal = [[ACM Computing Surveys]] | pages=123–165 | volume =10 | issue = 2 |date=June 1978 | url = http://portal.acm.org/citation.cfm?id=356729&coll=&dl=ACM&CFID=15151515&CFTOKEN=6184618 |doi=10.1145/356725.356729 |s2cid=16909447 | issn= 0360-0300 ))</ref><ref>((cite journal | author= P. J. Denning | author-link= P. J. Denning |title=Fault tolerant operating systems | journal = ACM Computing Surveys | pages=359–389 | volume =8 | issue = 4 |date=December 1976 | url = http://portal.acm.org/citation.cfm?id=356680&dl=ACM&coll=&CFID=15151515&CFTOKEN=6184618 |doi=10.1145/356678.356680 |s2cid=207736773 | issn= 0360-0300 ))</ref><ref>((cite journal | author= Theodore A. Linden |title=Operating System Structures to Support Security and Reliable Software | journal = ACM Computing Surveys | pages=409–445 | volume =8 | issue = 4 |date=December 1976 | url = http://portal.acm.org/citation.cfm?id=356682&coll=&dl=ACM&CFID=15151515&CFTOKEN=6184618 |doi=10.1145/356678.356682 | issn= 0360-0300 | hdl=2027/mdp.39015086560037 |s2cid=16720589 | hdl-access=free ))</ref> For instance, [[F14 CADC]] had [[built-in self-test]] and redundancy.<ref>
Ray Holt.
[http://www.firstmicroprocessor.com/documents/lsistate-97.pdf "The F14A Central Air Data Computer, and the LSI Technology State-of-the-Art in 1968"].
</ref>

In general, the early efforts at fault-tolerant designs were focused mainly on internal diagnosis, where a fault would indicate something was failing and a worker could replace it. SAPO, for instance, had a method by which faulty memory drums would emit a noise before failure.<ref>Fault tolerant computing in computer design
Neilforoshan, M.R
Journal of Computing Sciences in Colleges archive
Volume 18,  Issue 4  (April 2003) Pages: 213 – 220, ((ISSN|1937-4771))</ref> Later efforts showed that to be fully effective, the system had to be self-repairing and diagnosing – isolating a fault and then implementing a redundant backup while alerting a need for repair. This is known as N-model redundancy, where faults cause automatic fail-safes and a warning to the operator, and it is still the most common form of level one fault-tolerant design in use today.

Voting was another initial method, as discussed above, with multiple redundant backups operating constantly and checking each other's results. For example, if four components reported an answer of 5 and one component reported an answer of 6, the other four would "vote" that the fifth component was faulty and have it taken out of service. This is called M out of N majority voting.

Historically, the trend has been to move away from N-model and toward M out of N, as the complexity of systems and the difficulty of ensuring the transitive state from fault-negative to fault-positive did not disrupt operations.

[[Tandem Computers]], in 1976<ref>((Cite web |title=History of TANDEM COMPUTERS, INC.  |url=http://www.fundinguniverse.com/company-histories/tandem-computers-inc-history/ |access-date=2023-03-01 |website=FundingUniverse ))</ref> and [[Stratus Technologies|Stratus]] were among the first companies specializing in the design of fault-tolerant computer systems for [[online transaction processing]].

== Examples ==
[[File:Twitter M2 mobile website.png|thumb|"M2 Mobile Web", the original mobile web front end of Twitter, later served as fallback legacy version to [[client (computing)|clients]] without JavaScript support and/or incompatible browsers until December 2020.]]
Hardware fault tolerance sometimes requires that broken parts be taken out and replaced with new parts while the system is still operational (in computing known as ''[[hot swapping]]''). Such a system implemented with a single backup is known as '''single point tolerant''' and represents the vast majority of fault-tolerant systems. In such systems the [[mean time between failure]]s should be long enough for the operators to have sufficient time to fix the broken devices ([[mean time to repair]]) before the backup also fails. It is helpful if the time between failures is as long as possible, but this is not specifically required in a fault-tolerant system.

Fault tolerance is notably successful in computer applications. [[Tandem Computers]] built their entire business on such machines, which used single-point tolerance to create their [[NonStop (server computers)|NonStop]] systems with [[uptime]]s measured in years.

[[Fail-safe]] architectures may encompass also the computer software, for example by process [[replication (computer science)|replication]].

Data formats may also be designed to degrade gracefully. [[HTML]] for example, is designed to be [[forward compatibility|forward compatible]], allowing [[Web browser]]s to ignore new and unsupported HTML entities without causing the document to be unusable. Additionally, some sites, including popular platforms such as Twitter (until December 2020), provide an optional lightweight front end that does not rely on [[JavaScript]] and has a [[minimalism (computing)|minimal]] layout, to ensure wide [[Computer accessibility|accessibility]] and [[outreach]], such as on [[game console]]s with limited web browsing capabilities.<ref>((Cite web|title=Why your website should work without JavaScript.|url=https://dev.to/shadowfaxrodeo/why-your-website-should-work-without-javascript-3kko |author1=Nathaniel |access-date=2021-05-16|website=DEV Community|date=17 March 2021 |language=en))</ref><ref>((Cite web|last=Fairfax|first=Zackerie|date=2020-11-28|title=Legacy Twitter Shutdown Means You Can't Tweet From The 3DS Anymore|url=https://screenrant.com/twitter-legacy-nintendo-3ds-shut-down-date-december-2020/|access-date=2021-07-01|website=[[Screen Rant]]|language=en-US))</ref>

== Terminology ==
[[Image:Graceful Degradation of Transparency.png|thumb|256px|right|An example of graceful degradation by design in an image with transparency. Each of the top two images is the result of viewing the composite image in a viewer that recognises transparency. The bottom two images are the result in a viewer with no support for transparency. Because the transparency mask (center bottom) is discarded, only the overlay (center top) remains. The image on the left has been designed to degrade gracefully, hence is still meaningful without its transparency information.]]
A highly fault-tolerant system might continue at the same level of performance even though one or more components have failed. For example, a building with a backup electrical generator will provide the same voltage to wall outlets even if the grid power fails.

A system that is designed to [[fail safe]], or fail-secure, or '''fail gracefully''', whether it functions at a reduced level or fails completely, does so in a way that protects people, property, or data from injury, damage, intrusion, or disclosure. In computers, a program might fail-safe by executing a [[graceful exit]] (as opposed to an uncontrolled crash) to prevent data corruption after an error occurs.<ref>((Cite journal |last1=Hudak |first1=J.J. |last2=Suh |first2=B.-H. |last3=Siewiorek |first3=D.P. |last4=Segall |first4=Z. |date=1993 |title=Evaluation and comparison of fault-tolerant software techniques |url=https://ieeexplore.ieee.org/document/229487 |journal=IEEE Transactions on Reliability |volume=42 |issue=2 |pages=190–204 |doi=10.1109/24.229487 |issn=1558-1721))</ref> A similar distinction is made between "failing well" and "[[failing badly]]".

A system designed to experience '''graceful degradation''', or to '''fail soft''' (used in computing, similar to "fail safe"<ref>Stallings, W (2009): Operating Systems. Internals and Design Principles'', sixth edition</ref>) operates at a reduced level of performance after some component fails. For example, if grid power fails, a building may operate lighting at reduced levels or elevators at reduced speeds. In computing, if insufficient network bandwidth is available to stream an online video, a lower-resolution version might be streamed in place of the high-resolution version. [[Progressive enhancement]] is another example, where web pages are available in a basic functional format for older, small-screen, or limited-capability web browsers, but in an enhanced version for browsers capable of handling additional technologies or that have a larger display.

In fault-tolerant computer systems, programs that are considered [[Robustness (computer science)|robust]] are designed to continue operation despite an error, exception, or invalid input, instead of crashing completely. [[Software brittleness]] is the opposite of robustness.  [[Resilience (network)|Resilient networks]] continue to transmit data despite the failure of some links or nodes. [[Resilience (engineering and construction)|Resilient buildings and infrastructure]] are likewise expected to prevent complete failure in situations like earthquakes, floods, or collisions.

A system with high [[failure transparency]] will alert users that a component failure has occurred, even if it continues to operate with full performance, so that failure can be repaired or imminent complete failure anticipated.<ref>((cite arXiv|last=Thampi|first=Sabu M.|date=2009-11-23|title=Introduction to Distributed Systems|class=cs.DC|eprint=0911.4395))</ref> Likewise, a [[Fail-fast system|fail-fast]] component is designed to report at the first point of failure, rather than generating reports when downstream components fail. This allows easier diagnosis of the underlying problem, and may prevent improper operation in a broken state.

A '''single fault condition''' is a situation where one means for [[protection mechanism|protection]] against a [[hazard]] is defective. If a single fault condition results unavoidably in another single fault condition, the two failures are considered one single fault condition.<ref>((cite web|url=http://grouper.ieee.org/groups/1461/glossary.htm |title=Control |website=Grouper.ieee.org |archive-url=https://web.archive.org/web/19991008210224/http://grouper.ieee.org/groups/1461/glossary.htm |access-date=2016-04-06|archive-date=1999-10-08 ))</ref> A source offers the following example: ((blockquote|A ''single-fault condition'' is a condition when a single means for protection against hazard in equipment is defective or a single external abnormal condition is present, e.g. short circuit between the live parts and the applied part.<ref>Baha Al-Shaikh, Simon G. Stacey, ''Essentials of Equipment in Anaesthesia, Critical Care, and Peri-Operative Medicine'' (2017), p. 247.</ref>))

==Criteria==
Providing fault-tolerant design for every component is normally not an option. Associated redundancy brings a number of penalties: increase in weight, size, power consumption, cost, as well as time to design, verify, and test. Therefore, a number of choices have to be examined to determine which components should be fault tolerant:<ref>Dubrova, E. (2013). "Fault-Tolerant Design", Springer, 2013, ((ISBN|978-1-4614-2112-2))</ref>

* '''How critical is the component?''' In a car, the radio is not critical, so this component has less need for fault tolerance.
* '''How likely is the component to fail?''' Some components, like the drive shaft in a car, are not likely to fail, so no fault tolerance is needed.
* '''How expensive is it to make the component fault tolerant?''' Requiring a redundant car engine, for example, would likely be too expensive both economically and in terms of weight and space, to be considered.

An example of a component that passes all the tests is a car's occupant restraint system. While the ''primary'' occupant restraint system is not normally thought of, it is [[gravity]]. If the vehicle rolls over or undergoes severe g-forces, then this primary method of occupant restraint may fail. Restraining the occupants during such an accident is absolutely critical to safety, so the first test is passed. Accidents causing occupant ejection were quite common before [[seat belt]]s, so the second test is passed. The cost of a redundant restraint method like seat belts is quite low, both economically and in terms of weight and space, so the third test is passed. Therefore, adding seat belts to all vehicles is an excellent idea. Other "supplemental restraint systems", such as [[airbag]]s, are more expensive and so pass that test by a smaller margin.

Another excellent and long-term example of this principle being put into practice is the braking system: whilst the actual brake mechanisms are critical, they are not particularly prone to sudden (rather than progressive) failure, and are in any case necessarily duplicated to allow even and balanced application of brake force to all wheels. It would also be prohibitively costly to further double-up the main components and they would add considerable weight. However, the similarly critical systems for actuating the brakes under driver control are inherently less robust, generally using a cable (can rust, stretch, jam, snap) or hydraulic fluid (can leak, boil and develop bubbles, absorb water and thus lose effectiveness). Thus in most modern cars the footbrake hydraulic brake circuit is diagonally divided to give two smaller points of failure, the loss of either only reducing brake power by 50% and not causing as much dangerous brakeforce imbalance as a straight front-back or left-right split, and should the hydraulic circuit fail completely (a relatively very rare occurrence), there is a failsafe in the form of the cable-actuated parking brake that operates the otherwise relatively weak rear brakes, but can still bring the vehicle to a safe halt in conjunction with transmission/engine braking so long as the demands on it are in line with normal traffic flow. The cumulatively unlikely combination of total foot brake failure with the need for harsh braking in an emergency will likely result in a collision, but still one at lower speed than would otherwise have been the case.

In comparison with the foot pedal activated service brake, the parking brake itself is a less critical item, and unless it is being used as a one-time backup for the footbrake, will not cause immediate danger if it is found to be nonfunctional at the moment of application. Therefore, no redundancy is built into it per se (and it typically uses a cheaper, lighter, but less hardwearing cable actuation system), and it can suffice, if this happens on a hill, to use the footbrake to momentarily hold the vehicle still, before driving off to find a flat piece of road on which to stop. Alternatively, on shallow gradients, the transmission can be shifted into Park, Reverse or First gear, and the transmission lock / engine compression used to hold it stationary, as there is no need for them to include the sophistication to first bring it to a halt.

On motorcycles, a similar level of fail-safety is provided by simpler methods; first, the front and rear brake systems are entirely separate, regardless of their method of activation (that can be cable, rod or hydraulic), allowing one to fail entirely while leaving the other unaffected. Second, the rear brake is relatively strong compared to its automotive cousin, being a powerful disc on some sports models, even though the usual intent is for the front system to provide the vast majority of braking force; as the overall vehicle weight is more central, the rear tire is generally larger and has better traction, so that the rider can lean back to put more weight on it, therefore allowing more brake force to be applied before the wheel locks. On cheaper, slower utility-class machines, even if the front wheel should use a hydraulic disc for extra brake force and easier packaging, the rear will usually be a primitive, somewhat inefficient, but exceptionally robust rod-actuated drum, thanks to the ease of connecting the footpedal to the wheel in this way and, more importantly, the near impossibility of catastrophic failure even if the rest of the machine, like a lot of low-priced bikes after their first few years of use, is on the point of collapse from neglected maintenance.

==Requirements==
The basic characteristics of fault tolerance require:
# No [[single point of failure]] – If a system experiences a failure, it must continue to operate without interruption during the repair process.
# [[Fault isolation]] to the failing component – When a failure occurs, the system must be able to isolate the failure to the offending component. This requires the addition of dedicated failure detection mechanisms that exist only for the purpose of fault isolation. Recovery from a fault condition requires classifying the fault or failing component. The [[National Institute of Standards and Technology]] (NIST) categorizes faults based on locality, cause, duration, and effect.((Where|date=June 2017))((clarify|date=June 2017))
# Fault containment to prevent propagation of the failure – Some failure mechanisms can cause a system to fail by propagating the failure to the rest of the system. An example of this kind of failure is the "rogue transmitter" that can swamp legitimate communication in a system and cause overall system failure. [[Firewall (computing)|Firewall]]s or other mechanisms that isolate a rogue transmitter or failing component to protect the system are required.
# Availability of [[Reversion (software development)|reversion modes]]((clarify|date=June 2014))

In addition, fault-tolerant systems are characterized in terms of both planned service outages and unplanned service outages. These are usually measured at the application level and not just at a hardware level. The figure of merit is called [[availability]] and is expressed as a percentage. For example, a [[5 nines|five nines]] system would statistically provide 99.999% availability.

Fault-tolerant systems are typically based on the concept of redundancy.

== Fault tolerance techniques ==
Research into the kinds of tolerances needed for critical systems involves a large amount of interdisciplinary work. The more complex the system, the more carefully all possible interactions have to be considered and prepared for. Considering the importance of high-value systems in transport, [[public utilities]] and the military, the field of topics that touch on research is very wide: it can include such obvious subjects as [[software modeling]] and reliability, or [[hardware design]], to arcane elements such as [[stochastic]] models, [[graph theory]], formal or exclusionary logic, [[Parallel computing|parallel processing]], remote [[data transmission]], and more.<ref>((cite book |title=Reliability evaluation of some fault-tolerant computer architectures |date=November 1980 |publisher=Springer-Verlag |isbn=978-3-540-10274-8))</ref>

===Replication===
Spare components address the first fundamental characteristic of fault tolerance in three ways:
* [[Replication (computer science)|Replication]]: Providing multiple identical instances of the same system or subsystem, directing tasks or requests to all of them in [[parallel computing|parallel]], and choosing the correct result on the basis of a [[quorum]];
* [[Redundancy (engineering)|Redundancy]]: Providing multiple identical instances of the same system and switching to one of the remaining instances in case of a failure ([[failover]]);
* Diversity: Providing multiple ''different'' implementations of the same specification, and using them like replicated systems to cope with errors in a specific implementation.

All implementations of [[RAID]], [[RAID|redundant array of independent disks]], except RAID 0, are examples of a fault-tolerant [[data storage device|storage device]] that uses [[data redundancy]].

A [[lockstep (computing)|lockstep]] fault-tolerant machine uses replicated elements operating in parallel. At any time, all the replications of each element should be in the same state. The same inputs are provided to each [[replication (computing)|replication]], and the same outputs are expected. The outputs of the replications are compared using a voting circuit. A machine with two replications of each element is termed [[dual modular redundancy|dual modular redundant]] (DMR). The voting circuit can then only detect a mismatch and recovery relies on other methods. A machine with three replications of each element is termed [[triple modular redundancy|triple modular redundant]] (TMR). The voting circuit can determine which replication is in error when a two-to-one vote is observed. In this case, the voting circuit can output the correct result, and discard the erroneous version. After this, the internal state of the erroneous replication is assumed to be different from that of the other two, and the voting circuit can switch to a DMR mode. This model can be applied to any larger number of replications.

[[Lockstep (computing)|Lockstep]] fault-tolerant machines are most easily made fully [[Synchronization (computer science)|synchronous]], with each gate of each replication making the same state transition on the same edge of the clock, and the clocks to the replications being exactly in phase. However, it is possible to build lockstep systems without this requirement.

Bringing the replications into synchrony requires making their internal stored states the same. They can be started from a fixed initial state, such as the reset state. Alternatively, the internal state of one replica can be copied to another replica.

One variant of DMR is '''pair-and-spare'''. Two replicated elements operate in lockstep as a pair, with a voting circuit that detects any mismatch between their operations and outputs a signal indicating that there is an error. Another pair operates exactly the same way. A final circuit selects the output of the pair that does not proclaim that it is in error. Pair-and-spare requires four replicas rather than the three of TMR, but has been used commercially.

===Failure-oblivious computing===
''Failure-oblivious computing'' is a technique that enables [[computer programs]] to continue executing despite [[data loss|errors]].<ref>((Cite book|last1=Herzberg|first1=Amir|last2=Shulman|first2=Haya|title=2012 Seventh International Conference on Availability, Reliability and Security |chapter=Oblivious and Fair Server-Aided Two-Party Computation |date=2012|chapter-url=http://dx.doi.org/10.1109/ares.2012.28|pages=75–84|publisher=IEEE|doi=10.1109/ares.2012.28|isbn=978-1-4673-2244-7|s2cid=6579295))</ref> The technique can be applied in different contexts. It can handle invalid memory reads by returning a manufactured value to the program,<ref>((Citation|last1=Rigger|first1=Manuel|title=Context-Aware Failure-Oblivious Computing as a Means of Preventing Buffer Overflows|date=2018|url=http://dx.doi.org/10.1007/978-3-030-02744-5_28|work=Network and System Security|pages=376–390|place=Cham|publisher=Springer International Publishing|isbn=978-3-030-02743-8|access-date=2020-10-07|last2=Pekarek|first2=Daniel|last3=Mössenböck|first3=Hanspeter|series=Lecture Notes in Computer Science |volume=11058 |doi=10.1007/978-3-030-02744-5_28|arxiv=1806.09026))</ref> which in turn, makes use of the manufactured value and ignores the former [[computer memory|memory]] value it tried to access, this is a great contrast to [[RAM parity|typical memory checkers]], which inform the program of the error or abort the program.

The approach has performance costs: because the technique rewrites code to insert dynamic checks for address validity, execution time will increase by 80% to 500%.<ref>((Citation
  | first = Angelos D.
  | last = Keromytis
  | editor-last = Gorodetski 
  | editor-first = Vladimir I. 
  | editor2-last = Kotenko
  | editor2-first = Igor 
  | editor3-last = Skormin
  | editor3-first = Victor A.
  | title = Characterizing Software Self-Healing Systems
  | contribution = Characterizing Software Self-Healing Systems
  | contribution-url = https://books.google.com/books?id=N2uIjckxHSoC&q=failure-oblivious&pg=PA27
  | series = Computer network security: Fourth International Conference on Mathematical Methods, Models, and Architectures for Computer Network Security
  | year = 2007
  | publisher = [[Springer Science+Business Media|Springer]]
  | isbn = 978-3-540-73985-2 ))
</ref>

===Recovery shepherding===
Recovery shepherding is a lightweight technique to enable software programs to recover from otherwise fatal errors such as null pointer dereference and divide by zero.<ref name=rcv>((cite book
|last1=Long
|first1=Fan
|last2=Sidiroglou-Douskos
|first2=Stelios
|last3=Rinard
|first3=Martin
|date=2014
|chapter=Automatic Runtime Error Repair and Containment via Recovery Shepherding
|title=Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation
|series=PLDI '14'
|location=New York, NY, US
|publisher=ACM
|pages=227–238
|isbn=978-1-4503-2784-8
|doi=10.1145/2594291.2594337 |doi-access=free
|s2cid=6252501 |s2cid-access=free
))</ref> Comparing to the failure oblivious computing technique, recovery shepherding works on the compiled program binary directly and does not need to recompile to program.

It uses the [[Just-in-time compilation|just-in-time]] [[instrumentation (computer programming)|binary instrumentation]] framework [[Pin (computer program)|Pin]]. It attaches to the application process when an error occurs, repairs the execution, 
tracks the repair effects as the execution continues, contains the repair effects within the application process, and detaches from the process after all repair effects are flushed from the process state. It does not interfere with the normal execution of the program and therefore incurs negligible overhead.<ref name=rcv/> For 17 of 18 systematically collected real world null-dereference and divide-by-zero errors, a prototype implementation enables the application to continue to execute to provide acceptable output and service to its users on the error-triggering inputs.<ref name=rcv/>

===Circuit breaker===
((Main|Circuit breaker design pattern))
The [[circuit breaker design pattern]] is a technique to avoid catastrophic failures in distributed systems.

== Redundancy ==
((main article|Redundancy (engineering)))
Redundancy is the provision of functional capabilities that would be unnecessary in a fault-free environment.<ref>Laprie, J. C. (1985). "[http://www.macedo.ufba.br/conceptsANDTermonology.pdf Dependable Computing and Fault Tolerance: Concepts and Terminology]", Proceedings of 15th International Symposium on Fault-Tolerant Computing (FTSC-15), pp. 2–11</ref>
This can consist of backup components that automatically "kick in" if one component fails. For example, large cargo trucks can lose a tire without any major consequences. They have many tires, and no one tire is critical (with the exception of the front tires, which are used to steer, but generally carry less load, each and in total, than the other four to 16, so are less likely to fail).
The idea of incorporating redundancy in order to improve the reliability of a system was pioneered by [[John von Neumann]] in the 1950s.<ref>von Neumann, J. (1956). "[http://www.cyclify.com/wiki/images/a/af/Von_Neumann_Probabilistic_Logics_and_the_Synthesis_of_Reliable_Organisms_from_Unreliable_Components.pdf Probabilistic Logics and Synthesis of Reliable Organisms from Unreliable Components]", in Automata Studies, eds. C. Shannon and J. McCarthy,  Princeton University Press, pp. 43–98</ref>

Two kinds of redundancy are possible:<ref>Avizienis, A. (1976). "[https://www.computer.org/csdl/trans/tc/1976/12/01674598.pdf Fault-Tolerant Systems]", IEEE Transactions on Computers, vol. 25, no. 12, pp. 1304–1312</ref> space redundancy and time redundancy. Space redundancy provides additional components, functions, or data items that are unnecessary for fault-free operation. Space redundancy is further classified into hardware, software and information redundancy, depending on the type of redundant resources added to the system.  In time redundancy the computation or data transmission is repeated and the result is compared to a stored copy of the previous result. The current terminology for this kind of testing is referred to as 'In Service Fault Tolerance Testing or ISFTT for short.

== Disadvantages ==
Fault-tolerant design's advantages are obvious, while many of its disadvantages are not:

* '''Interference with fault detection in the same component.''' To continue the above passenger vehicle example, with either of the fault-tolerant systems it may not be obvious to the driver when a tire has been punctured. This is usually handled with a separate "automated fault-detection system". In the case of the tire, an air pressure monitor detects the loss of pressure and notifies the driver. The alternative is a "manual fault-detection system", such as manually inspecting all tires at each stop.
* '''Interference with fault detection in another component.''' Another variation of this problem is when fault tolerance in one component prevents fault detection in a different component. For example, if component B performs some operation based on the output from component A, then fault tolerance in B can hide a problem with A. If component B is later changed (to a less fault-tolerant design) the system may fail suddenly, making it appear that the new component B is the problem. Only after the system has been carefully scrutinized will it become clear that the root problem is actually with component A.
* '''Reduction of priority of fault correction.''' Even if the operator is aware of the fault, having a fault-tolerant system is likely to reduce the importance of repairing the fault. If the faults are not corrected, this will eventually lead to system failure, when the fault-tolerant component fails completely or when all redundant components have also failed.
* '''Test difficulty.''' For certain critical fault-tolerant systems, such as a [[nuclear reactor]], there is no easy way to verify that the backup components are functional. The most infamous example of this is [[Chernobyl disaster|Chernobyl]], where operators tested the emergency backup cooling by disabling primary and secondary cooling. The backup failed, resulting in a core meltdown and massive release of radiation.
* '''Cost.''' Both fault-tolerant components and redundant components tend to increase cost. This can be a purely economic cost or can include other measures, such as weight. [[Human spaceflight|Crewed spaceships]], for example, have so many redundant and fault-tolerant components that their weight is increased dramatically over uncrewed systems, which do not require the same level of safety.
* '''Inferior components.''' A fault-tolerant design may allow for the use of inferior components, which would have otherwise made the system inoperable.  While this practice has the potential to mitigate the cost increase, use of multiple inferior components may lower the reliability of the system to a level equal to, or even worse than, a comparable non-fault-tolerant system.

== Related terms ==
There is a difference between fault tolerance and systems that rarely have problems. For instance, the [[Western Electric]] [[Crossbar switch|crossbar]] systems had failure rates of two hours per forty years, and therefore were highly ''fault resistant''. But when a fault did occur they still stopped operating completely, and therefore were not ''fault tolerant''.

==See also==
((Div col))
*[[Byzantine fault tolerance]]
*[[Control reconfiguration]]
*[[Damage tolerance]]
*[[Data redundancy]]
*[[Defence in depth]]
*[[Ecological resilience]]
*[[Elegant degradation]]
*[[Error detection and correction]]
*[[Error-tolerant design]] ([[human error]]-tolerant design)
*[[Fail-safe]]
*[[Failure semantics]]
*[[Fall back and forward]]
*[[Graceful exit]]
*[[Intrusion tolerance]]
*[[List of system quality attributes]]
*[[Progressive enhancement]]
*[[Resilience (network)]]
*[[Robustness (computer science)]]
*[[Rollback (data management)]]
*[[Self-management (computer science)]]
*[[Crash-only software]]
((Div col end))

==References==
((Reflist))
((Authority control))
((Systems engineering))

[[Category:Fault tolerance| ]]
[[Category:Reliability engineering]]
[[Category:Computer systems]]
[[Category:Control engineering]]
[[Category:Systems engineering]]
[[Category:Software quality]]
[[Category:RAID]]

Pages transcluded onto the current version of this page (help):

Fault tolerance (edit)
Template:Ambox (view source) (template editor protected)
Template:Authority control (view source) (template editor protected)
Template:Blockquote (view source) (template editor protected)
Template:Blockquote/styles.css (view source) (template editor protected)
Template:Catalog lookup link (view source) (template editor protected)
Template:Category handler (view source) (protected)
Template:Citation (view source) (protected)
Template:Cite arXiv (view source) (template editor protected)
Template:Cite book (view source) (protected)
Template:Cite journal (view source) (protected)
Template:Cite web (view source) (protected)
Template:Clarify (view source) (template editor protected)
Template:DMCA (view source) (template editor protected)
Template:Dated maintenance category (view source) (template editor protected)
Template:Dated maintenance category (articles) (view source) (template editor protected)
Template:Delink (view source) (protected)
Template:Div col (view source) (template editor protected)
Template:Div col/styles.css (view source) (template editor protected)
Template:Div col end (view source) (template editor protected)
Template:Error-small (view source) (template editor protected)
Template:FULLROOTPAGENAME (view source) (template editor protected)
Template:Find sources mainspace (view source) (template editor protected)
Template:Fix (view source) (protected)
Template:Fix-span (view source) (template editor protected)
Template:Fix/category (view source) (protected)
Template:Hlist/styles.css (view source) (protected)
Template:ISBN (view source) (template editor protected)
Template:ISSN (view source) (template editor protected)
Template:Main (view source) (template editor protected)
Template:Main article (view source) (template editor protected)
Template:Main other (view source) (protected)
Template:More citations needed (view source) (template editor protected)
Template:Navbox (view source) (template editor protected)
Template:Ns has subpages (view source) (protected)
Template:Pagetype (view source) (protected)
Template:R/superscript (view source) (template editor protected)
Template:R/where (view source) (template editor protected)
Template:Reflist (view source) (protected)
Template:Reflist/styles.css (view source) (protected)
Template:Rp (view source) (template editor protected)
Template:SDcat (view source) (protected)
Template:Short description (view source) (protected)
Template:Short description/lowercasecheck (view source) (protected)
Template:Small (view source) (protected)
Template:Systems engineering (edit)
Template:Template link (view source) (protected)
Template:Tl (view source) (protected)
Template:Trim (view source) (protected)
Template:Use American English (view source) (template editor protected)
Template:Where (view source) (template editor protected)
Template:Yesno (view source) (protected)
Template:Yesno-no (view source) (template editor protected)
Template:Yesno-yes (view source) (template editor protected)
Module:Arguments (view source) (protected)
Module:Authority control (view source) (template editor protected)
Module:Authority control/config (view source) (template editor protected)
Module:Catalog lookup link (view source) (template editor protected)
Module:Category handler (view source) (protected)
Module:Category handler/blacklist (view source) (protected)
Module:Category handler/config (view source) (protected)
Module:Category handler/data (view source) (protected)
Module:Category handler/shared (view source) (protected)
Module:Check for unknown parameters (view source) (protected)
Module:Check isxn (view source) (template editor protected)
Module:Citation/CS1 (view source) (protected)
Module:Citation/CS1/COinS (view source) (protected)
Module:Citation/CS1/Configuration (view source) (protected)
Module:Citation/CS1/Date validation (view source) (protected)
Module:Citation/CS1/Identifiers (view source) (protected)
Module:Citation/CS1/Utilities (view source) (protected)
Module:Citation/CS1/Whitelist (view source) (protected)
Module:Citation/CS1/styles.css (view source) (protected)
Module:DecodeEncode (view source) (template editor protected)
Module:Delink (view source) (protected)
Module:Disambiguation/templates (view source) (protected)
Module:Error (view source) (template editor protected)
Module:Find sources (view source) (template editor protected)
Module:Find sources/config (view source) (template editor protected)
Module:Find sources/links (view source) (template editor protected)
Module:Find sources/templates/Find sources mainspace (view source) (template editor protected)
Module:Format link (view source) (template editor protected)
Module:GetParameters (view source) (template editor protected)
Module:Hatnote (view source) (template editor protected)
Module:Hatnote/styles.css (view source) (template editor protected)
Module:Hatnote list (view source) (template editor protected)
Module:Labelled list hatnote (view source) (template editor protected)
Module:Message box (view source) (protected)
Module:Message box/ambox.css (view source) (protected)
Module:Message box/configuration (view source) (protected)
Module:Namespace detect/config (view source) (protected)
Module:Namespace detect/data (view source) (protected)
Module:Navbar (view source) (protected)
Module:Navbar/configuration (view source) (protected)
Module:Navbar/styles.css (view source) (protected)
Module:Navbox (view source) (template editor protected)
Module:Navbox/configuration (view source) (template editor protected)
Module:Navbox/styles.css (view source) (template editor protected)
Module:Ns has subpages (view source) (protected)
Module:Pagetype (view source) (protected)
Module:Pagetype/config (view source) (protected)
Module:Pagetype/disambiguation (view source) (protected)
Module:Pagetype/setindex (view source) (protected)
Module:Pagetype/softredirect (view source) (protected)
Module:Plain text (view source) (template editor protected)
Module:SDcat (view source) (protected)
Module:String (view source) (protected)
Module:String2 (view source) (template editor protected)
Module:Unsubst (view source) (protected)
Module:Wikitext Parsing (view source) (protected)
Module:Yesno (view source) (protected)

Return to Fault tolerance.

Retrieved from "https://en.wikipedia.org/wiki/Fault_tolerance"