Asynchronous System Trap (AST) refers to a mechanism used in several computer operating systems designed by the former Digital Equipment Corporation (DEC) of Maynard, Massachusetts.

Mechanism

Various events within these systems can be optionally signalled back to the user processes via the AST mechanism. These ASTs act like subroutine calls but they are delivered asynchronously, that is, without any regard to the context of the main thread. Because of this, care must be taken:

• to ensure that any code that is shared between the main thread and the AST must be designed to be reentrant, and
• any data that is shared must be safe against corruption if modified at any time by the AST. Otherwise, the data must be guarded by blocking ASTs during critical sections.

ASTs are most commonly encountered as a result of issuing QIO calls to the kernel. Completion of the I/O can be signalled by the issuance of an AST to the calling process/task. Certain runtime errors could also be signalled using the AST mechanism. Within OpenVMS, Special Kernel-Mode ASTs are used as the standard mechanism for getting relatively convenient access to a process context (including getting the process paged into physical memory as may be needed). These types of ASTs are executed at the highest possible per-process priority the next time the scheduler makes that process current, and are used among other things for retrieving process-level information (in response to a $GETJPI "getjob/process information" system call) and for performing process deletion.

The following operating systems implement ASTs:

• RSX-11 (including all of the variants)
• RSTS/E
• OpenVMS

ASTs are roughly analogous to Unix signals. The important differences are:

• There are no "signal codes" assigned to ASTs: instead of assigning a handler to a signal code and raising that code, the AST is specified directly by its address. This allows any number of ASTs to be pending at once (subject to process quotas).
• ASTs never abort any system call in progress. In fact, it is possible for a process to put itself into a "hibernate" state (with the $HIBER system call), or to wait for an event flag by calling e.g. $WAITFR, whereupon it does nothing but wait for ASTs to be delivered. When an AST is delivered (triggered by an IO completion, timer, or other event), the process is temporarily taken out of the wait to execute the AST. After the AST procedure completes, the call that put the process into hibernation or the event flag wait is made again; in essence, the reason for the wait is re-evaluated. The only way to get out of this loop (apart from process deletion) is to execute a $WAKE or $SETEF system call to satisfy the wait. This can be done by the process itself by invoking $WAKE or $SETEF within the AST, or (if a global event flag is used) $SETEF within another process.

VAX/VMS V4 and later implemented an interesting optimization to the problem of synchronizing between AST-level and non-AST-level code. A system service named $SETAST could be used to disable or enable the delivery of ASTs for the current and all less-privileged access modes (the OpenVMS term for ring-based security features). However, if the critical section needing protection from ASTs was only a few instructions long, then the overhead of making the $SETAST calls could far outweigh the time to execute those instructions.

So for user mode only (the least privileged ring, normally used by ordinary user programs), a pair of bit flags was provided at a predefined user-writable memory location (in per-process "P1" space). The meanings of these two flags could be construed as "don't deliver any ASTs" and "ASTs have been disabled". Instead of the usual pair of $SETAST calls, the user-mode code would set the first flag before executing the sequence of instructions during which ASTs need to be blocked, and clear it after the sequence. Then (note the ordering here, to avoid race conditions) it would check the second flag to see if it had become set during this time: if so, then ASTs really have become disabled, and $SETAST should be called to re-enable them. In the most common case, no ASTs would have become pending during this time, so there would be no need to call $SETAST at all.

The kernel AST delivery code, for its part, would check the first flag before trying to deliver a user-mode AST; if it was set, then it would directly set the ASTs-disabled bit in the process control block (the same bit that would be set by an explicit $SETAST call from user mode), and also set the second flag, before returning and leaving the AST undelivered.

The asynchronous procedure call mechanism in the Windows NT family of operating systems is a similar mechanism.