Indirectly Calling Nested Functions in GCC: Avoiding Executable Stacks

Author: Martin Uecker
Date: 2026-08-29
Contact: [Blog/Impressum]

---

Introduction

In previous discussions, we explored the utilization of nested functions within Clang and GCC 17 for callbacks without the prerequisite of an executable stack.

While some developers simply accept an executable stack—arguing it isn't as catastrophic as critics suggest—it is entirely possible to bypass this requirement through a specific technical "hack."

GCC: The Mechanics of Nested Functions and Trampolines

To understand the workaround, we must first examine how GCC handles the process of taking the address of a nested function. Consider the following "toy" example (which avoids the use of new macros):

typedef int cb_f(int y);

int baz(cb_f p, int x) { 
    return p(x); 
}

int foo(int k) {
    int bar(int x) { 
        return k + x; 
    }
    return baz(bar, 2 * k);
}

On an x86_64 architecture, the compiler generates the following assembly:

bar.0: 
    movl %edi, %eax 
    addl (%r10), %eax 
    ret 
foo: 
    subq $56, %rsp 
    leaq 64(%rsp), %rax 
    movq %rax, 32(%rsp) 
    movl %edi, (%rsp) 
    leaq 4(%rsp), %rax 
    movw $-17591, 4(%rsp) 
    movabsq $bar.0, %rcx 
    movq %rcx, 6(%rsp) 
    movw $-17847, 14(%rsp) 
    movq %rsp, 16(%rsp) 
    movl $-1864106167, 24(%rsp) 
    addl %edi, %edi 
    call *%rax 
    addq $56, %rsp 
    ret 

Understanding the Trampoline

The code above creates a trampoline on the stack and invokes it via the inlined baz function. A trampoline is essentially a miniature code sequence that:

1. Loads the static frame register.
2. Points to a stack structure containing captured variables from the parent function.
3. Jumps to the actual local function.

If we translate the immediate constants used in the assembly back into instructions, we see the following logic:

The Workaround: Direct Calling

Rather than executing the trampoline, we can manually extract the code address and the static chain from the trampoline's memory. We then use the __builtin_call_with_static_chain built-in to invoke the function directly.

Using the peek and array_slice macros from the noplate library (targeting x64_64 with a large memory model), the extraction looks like this:

unsigned char (*tramp)[24] = (void*)bar;
void *code = peek(uint64_t, &array_slice(tramp, 2, 10));
void *chain = peek(uint64_t, &array_slice(tramp, 12, 20));

These pointers provide the same data that the upcoming GCC 17 provides via __builtin_call_static_chain and __builtin_call_code_adress. The final call is performed as follows:

__builtin_call_with_static_chain(((typeof(bar)*)code)(arg), chain);

Trade-offs and Security

This method serves as a fallback for older GCC versions, but it has drawbacks:

• A trampoline is still generated by the compiler.
• The compiler cannot perform devirtualization on the indirect call.
• The stack is still initially marked as executable.

However, since we never actually execute the trampoline code, we can strip the executable stack requirement using patchelf:

• Generate binary
• Run: patchelf --clear-execstack program $\rightarrow$ Executable Stack $\rightarrow$ Non-Executable Stack

This approach is integrated into the experimental noplate library, which constructs a "wide pointer" from the chain and code address.

Trampolines as Function Descriptors

Another intriguing possibility is treating the trampoline itself as a function descriptor.

Instead of extracting the pointers at the moment of creation, we pass the trampoline address normally. At the call site, we check if the pointer refers to a trampoline; if it does, we extract the code and chain to call the nested function via __builtin_call_with_static_chain.

Essentially, we are using a "super simple interpreter" at the call site to read the trampoline's instructions rather than executing them.

Logic Flow

Literature

• GCC, Nested Functions
• GCC, Constructing Function Calls