SYN cookie is a technique used to resist SYN flood attacks. The technique's primary inventor Daniel J. Bernstein defines SYN cookies as "particular choices of initial TCP sequence numbers by TCP servers." In particular, the use of SYN cookies allows a server to avoid dropping connections when the SYN queue fills up. Instead of storing additional connections, a SYN queue entry is encoded into the sequence number sent in the SYN+ACK response. If the server then receives a subsequent ACK response from the client with the incremented sequence number, the server is able to reconstruct the SYN queue entry using information encoded in the TCP sequence number and proceed as usual with the connection.
In order to initiate a TCP connection, the client sends a TCP SYN packet to the server. In response, the server sends a TCP SYN+ACK packet back to the client. One of the values in this packet is a sequence number, which is used by the TCP to reassemble the data stream. According to the TCP specification, that first sequence number sent by an endpoint can be any value as decided by that endpoint. As the sequence number is chosen by the sender, returned by the recipient, and has no otherwise-defined internal structure, it can be overloaded to carry additional data. The following describes one possible implementation, however as there is no public standard to follow, the order, length, and semantics of the fields may differ between SYN cookie implementations.
SYN cookies are initial sequence numbers that are carefully constructed according to the following rules:
The initial TCP sequence number, i.e. the SYN cookie, is computed as follows:
(Note: since m must be encoded using 3 bits, the server is restricted to sending up to 8 unique values for m when SYN cookies are in use.)
When a client sends back a TCP ACK packet to the server in response to the server's SYN+ACK packet, the client must (according to the TCP spec) use n+1 in the packet's Acknowledgement number, where n is the initial sequence number sent by the server. The server then subtracts 1 from the acknowledgement number to reveal the SYN cookie sent to the client.
The server then performs the following operations.
From this point forward, the connection proceeds as normal.
The use of SYN cookies does not break any protocol specifications, and therefore should be compatible with all TCP implementations. There are, however, two caveats that take effect when SYN cookies are in use. Firstly, the server is limited to only 8 unique MSS values, as that is all that can be encoded in 3 bits. Secondly, early implementations rejected all TCP options (such as large windows or timestamps), because the server discarded the SYN queue entry where that information would otherwise be stored.;[1] however v2.6.26 of the Linux kernel added partial support of TCP options by encoding them into the timestamp option.[2] Finally, SYN cookies place increased load on server resources. Encrypting responses is computationally expensive. The SYN cookie does not reduce traffic, which makes it ineffective against SYN flooding attacks that target bandwidth as the attack vector.
While these restrictions necessarily lead to a sub-optimal experience, their effect is rarely noticed by clients because they are only applied when under attack. In such a situation, the loss of the TCP options in order to save the connection is usually considered to be a reasonable compromise.
A problem arises when the connection-finalizing ACK packet sent by the client is lost, and the application layer protocol requires the server to speak first (SMTP and SSH are two examples). In this case, the client assumes that the connection was established successfully and waits for the server to send its protocol banner, or resend the SYN+ACK packet; however, the server is not aware of the session and will not resend the SYN+ACK because it discarded the backlog queue entry that would enable it to do so. Eventually, the client will abort the connection due to an application layer timeout, but this may take a relatively long time.[3]
TCP Cookie Transactions (TCPCT) standard was designed to overcome these shortcomings of SYN cookies and improve it on a couple of aspects. Unlike SYN cookies, TCPCT is a TCP extension and required support from both endpoints. It was moved to "Historic" status by RFC 7805 in 2016.
Simple firewalls that are configured to allow all outgoing connections but to restrict which ports an incoming connection can reach (for example, allow incoming connections to a Web server on port 80 but restrict all other ports), work by blocking only incoming SYN requests to unwanted ports. If SYN cookies are in operation, care should be taken to ensure an attacker is not able to bypass such a firewall by forging ACKs instead, trying random sequence numbers until one is accepted. SYN cookies should be switched on and off on a per-port basis, so that SYN cookies being enabled on a public port does not cause them to be recognised on a non-public port. The original Linux kernel implementation misunderstood this part of Bernstein's description and used a single global variable to switch on SYN cookies for all ports;[4] this was pointed out by a research student[5] and subsequently fixed in CVE-2001-0851.[6]
The technique was created by Daniel J. Bernstein and Eric Schenk in September 1996. The first implementation (for SunOS) was released by Jeff Weisberg a month later, and Eric Schenk released his Linux implementation in February 1997. FreeBSD implements syncookies since FreeBSD 4.5 (January 2002).[7]