What is RSB (Return address prediction buffer)

RSB (Return Stack Buffer) is a small hardware cache inside the processor that remembers the address to return to after a subroutine completes. When a function is called, the address is saved into the buffer, and upon return it is retrieved from there without waiting for a read from RAM, which eliminates pipeline stalls.

The return stack (RSB) is used in the branch prediction units of modern high-performance processor cores. The mechanism is engaged exclusively for paired CALL and RET instructions. When the decoder encounters a function call, the address of the next instruction is pushed into the internal buffer. At the moment of processing a return instruction, the predictor ignores general history tables and uses the value from the top of the RSB, which ensures speculative continuation of code execution.

A typical problem is buffer underflow or overflow during deep nesting or unbalanced branches when the number of returns does not match the number of calls. This causes mispredictions and pipeline flushes. The most critical vulnerability is a mismatch between the speculative and architectural states of the RSB: an attacker can poison the predictor through recursive calls, causing a Spectre-like data leak through speculative windows upon returning to an illegitimate address.

How RSB works

The operating principle of the RSB is based on simulating a shadow stack with a Last-In-First-Out discipline, implemented as a circular array with a top pointer. Unlike the Branch Target Buffer (BTB), which predicts indirect branches based on global history and is not bound to stack semantics, the RSB is not indexed by instruction address but works as an autonomous storage device with deterministic behavior. This makes it more accurate than the BTB for return instructions, but critically dependent on maintaining symmetry between calls and returns. In comparison with an indirect predictor that analyzes historical patterns, the RSB provides zero latency when stack depth matches perfectly. Modern implementations include state recovery mechanisms after mispredictions with checksums and the technique of multiplicative fillers during context switches to protect against return address spoofing attacks.

BTB (Prediction of the next branch instruction address)

RSB functionality

Purpose and place in the pipeline. The Return Stack Buffer is a hardware prediction stack physically located in the instruction prefetch unit. Its task is to speculatively provide target addresses for near RET return instructions. The RSB operates independently of the main branch predictor, ensuring low latency.
Internal write organization. When decoding a near CALL instruction, the pipeline calculates the return address. This value is speculatively pushed onto the top of the RSB shadow stack before the instruction commits, capturing the software stack state in a microarchitectural structure.
Shadow stack structure. Physically, the RSB is implemented as a circular buffer or a set of registers with a top pointer. The stack depth is strictly limited by the number of hardware entries. The top pointer increments on CALL and decrements on RET, mirroring the dynamics of the frame pointer and software stack.
Speculative prediction mechanism. When encountering a RET instruction in the predecode queue, the prefetch unit does not wait for the real address to be computed. Instead, the top of the RSB is read with a simultaneous pointer shift. The read value is immediately used as the speculative target for the next fetch block.
Correction upon pointer miss. If the pipeline detects that the call nesting depth has exceeded the RSB capacity, an overflow occurs. In this case, new CALLs cannot write data, and when the buffer is exhausted (underflow) prediction is disabled for RET. Control is passed to the indirect branch predictor or BTB.
Deferred update on a wrong path. In the case of speculative CALLs on a mispredicted path, the pipeline modifies the RSB pointer. Upon pipeline flush, the shadow stack state is fully restored from a checkpoint of the saved state corresponding to the last correct execution boundary. The physical pointer is rolled back.
Recovery after misprediction. If the speculative value from the RSB turns out to be incorrect (mismatch with the computed return address after RET execution), the pipeline is restarted. The RSB state is forcibly synchronized with the actual stack pointer position, and the instruction queue is cleared of dependent speculative instructions.
Collision resolution logic. The hardware unit tracks conflicts between the real stack in memory and the shadow cache. If a speculative RET reads an address while a parallel stack write operation modifies that memory area, the return buffer cannot track such a dependency without involving memory monitoring mechanisms and flushing speculative reads.
Interaction with pipeline depth. In processors with a deep instruction queue, the physical RSB pointer shifts speculatively many positions forward. This requires shadow copies of the pointer for each active block so that a RET deep in the instruction window can correctly match its position in the speculative stack.
Overflow during deep nesting. When the software call nesting exceeds the physical RSB size, new entries overwrite the oldest elements. The hardware logic tracks the fact of circular overwriting. Predictions for RET instructions corresponding to overwritten addresses are not issued, as the data is lost irretrievably.
Training on indirect branches. In some architectures, the RSB is associatively linked to the ITLB. If a RET is classified by the decoder not as a return but as an indirect branch, a write to the RSB may not occur. This requires precise static byte-code analysis to distinguish RET from register-addressed JMP during predecoding.
Synchronization during context switch. When privileged modes or threads are switched, the software stack is replaced. The microarchitectural RSB does not distinguish contexts and contains addresses of the previous task. Without forced hardware clearing, speculative return predictions will direct execution into the address space of another process, creating a vulnerability.
Speculative execution and security. The isolation of the RSB from access rights checking mechanisms creates a surface for speculative execution attacks (SpectreRSB). An attacker can poison the shadow return stack so that the victim speculatively executes instructions at a data disclosure address before the pipeline is flushed due to a memory protection exception.
Underflow protection mechanisms. To prevent underflow, the technique of filling the RSB with alternative predictions is used. Upon detecting an empty shadow return stack, the prediction unit substitutes a special guard address that triggers an exception, or borrows data from the indirect branch predictor, which has a larger history budget.
Hardware recovery in SMT mode. With simultaneous multithreading, each logical thread has its own dedicated RSB or dynamically shares a common entry array through thread ID tagging. With a shared resource, the eviction logic guarantees that one thread cannot monopolistically evict the addresses of another without resetting the binding.
SMT (Hardware emulation of two logical processors)
Static code analysis by the decoder. Modern decoders classify instructions byte by byte. Upon detecting a RET opcode, the decoder marks it in the uOP queue as an RSB consumer. Similarly, CALL instructions and their micro-operations (PUSH and JMP micro-opcodes) are marked as producers, triggering a signal to write the stack top.
Stack mismatch correction. If the code modifies the RSP stack pointer arithmetically, without a paired CALL, a desynchronization threat arises. The disconnect detection scheme tracks divergence between the shadow sum of RSP offsets and the RSB depth. Upon detecting a divergence, the scheme forcibly triggers a shadow stack flush and pointer resynchronization.
Address predictor over RSB. For return addresses evicted during overflow, a hybrid predictor may be used. It caches CALL address — return address pairs in a separate table. When the RSB cannot service a RET due to depth loss, the request is delegated to this structure, which works on the principle of associative search without a stack.
Energy efficiency of the shadow stack. Reading a prediction from a limited set of RSB registers is orders of magnitude more energy-efficient than accessing large BTB tables. Disabling this structure when compiling flat code without calls minimizes the dynamic power consumption of the prediction unit by gating the clock signal of the state capture group.
Looping mode for microarchitectures. When predicting short functions that fit in a loop, the RSB can block eviction of its top entries. The hardware state machine detects a repeating CALL–RET pattern without changing stack depth and disables the pointer increment/decrement, saving physical buffer entries.
Flush on system event edges. The hardware power management unit, when exiting low-power C-states where the RSB state is lost, initiates a recovery sequence. The control finite state machine hardware-generates a series of dummy writes to initialize the shadow stack with safe values before resuming user computations.

Comparisons

RSB vs RAS (Return Address Stack). The return stack buffer is functionally identical to the return address stack; however, the term RSB is more often used in the context of Intel microarchitecture to denote the hardware structure that predicts the target of a RET instruction. Unlike the abstract concept of RAS, RSB focuses on the implementation of speculative execution and recovery mechanisms after branch path misprediction.
RSB vs BTB (Branch Target Buffer). The branch target buffer caches arbitrary targets of branch instructions, whereas the RSB is specialized exclusively for CALL and RET pairs. BTB requires complex indexing schemes based on the source instruction address, while the RSB operates under the LIFO discipline, which guarantees hardware simplicity, minimal access latency, and virtually absolute prediction accuracy for balanced subroutine calls.
RSB vs Indirect Branch Predictor. The indirect branch predictor handles instructions with dynamically computed addresses using global history, whereas the RSB stores not the branch address but the return address placed there by the CALL instruction. The indirect predictor is vulnerable to history poisoning attacks (Spectre v2), while the isolated LIFO structure of the RSB requires more specific stack overflow or underflow techniques (Spectre RSB) for compromise.
RSB vs Shadow Stack. The shadow stack is a hardware control-flow integrity verification mechanism (CET) that creates a duplicate copy of return addresses in a protected memory area, whereas the RSB is a speculative microarchitectural cache structure inside the processor core, invisible to the programmer. Shadow Stack detects integrity attacks upon function exit, while the RSB provides only performance, completely trusting the speculative data stored within it.
RSB vs Software Return Prediction. In the absence of a hardware RSB, older processor cores used static prediction based on calling conventions, comparing the return address with the value on the memory stack. Such software-oriented prediction suffers from data cache read latencies and pipeline conflicts, whereas a dedicated physical RSB allows the core to immediately decode the next instruction after RET, completely eliminating dependence on the memory subsystem during speculative execution.

OS and driver support

The operating system interacts with the RSB indirectly through context management mechanisms: during thread switches the kernel dispatcher saves the kernel stack pointer in the TSS and manages the MSR_IA32_S_CET register for the shadow stack, whereas the RSB itself is filled exclusively by hardware CALL instructions. Therefore, drivers must not clear or modify the buffer manually, and correctly written interrupt and DPC code must use symmetric CALL/RET pairs within each procedure to avoid overflow and indirect corruption of the return predictor.

Security

The main attack vector, such as Spectre v2 (the return target address injection variant), exploits RSB underfill during deep call chains or its overflow with subsequent speculative execution at a stolen address. Therefore, manufacturers have implemented hardware filling techniques (RSB stuffing): upon entering kernel or virtual machine mode, the microcode executes a series of dummy calls filling the buffer with safe addresses. In processors supporting eIBRS and automatic RSB shadowing at protection ring boundaries, this procedure is triggered without software involvement, making buffer poisoning from a less privileged context impossible.

Diagnostics and state logging

Direct reading of the RSB contents is unavailable through architecturally visible registers, so logging is implemented via performance counters: events such as BR_MISP_RETIRED.RETURN_STACK for mismatches between predicted and actual return addresses or RSB_OVERFLOW_CYCLES capturing active cycles with a full buffer are read by the kernel via the perf_event_open interface. Debug extensions like Intel Processor Trace can reconstruct branch history and show moments in TNT/INDIRECT_BRANCH packets where the return prediction was invalidated or emulated due to RSB underflow, allowing hypervisor developers to identify bottlenecks with deep call nesting.

Hardware structure limitations

The RSB size is fixed and ranges from 12 to 32 entries depending on the microarchitecture, imposing a strict limit on predictable return depth: as soon as call nesting exceeds the buffer capacity, the oldest entries are irretrievably lost, and on the corresponding RET instructions the processor either resorts to the less accurate indirect BTB predictor or stalls the pipeline until address resolution. Moreover, the RSB does not distinguish privilege levels in older microarchitectures, creating a security limitation. With Software Guard Extensions (SGX) enabled, the enclave must operate with a truncated shadow return stack, which increases the miss rate and reduces protected code performance.

Development history and microarchitectural evolution

First introduced in Intel Pentium Pro processors as a small four-entry return stack to reduce the RET prediction penalty, the RSB evolved through depth increases and the emergence of overflow reconstruction logic in the Nehalem and Sandy Bridge microarchitectures. Then, in response to Spectre attacks, the Cascade Lake and Cooper Lake microarchitectures added automatic buffer shadowing on thread switches. AMD processors, starting with Zen 3, split the RSB into two independent blocks for different prediction types, simultaneously introducing hardware protection against overflow attacks through an entry invalidation mechanism upon entering system management mode. This eliminated the need for software barriers and reduced overhead to negligible values.