The Mystery of the Missing DLL: Not Unloaded, Yet Gone (Part 1)

Originally published on "The Old New Thing" by Raymond Chen

The engineering team in charge of shell32.dll was recently notified of a bug: a specific third-party application was experiencing a high volume of crashes, and the evidence pointed toward shell32.dll as the culprit.

🔍 Analyzing the Crash Dumps

Upon inspecting the crash dumps, the diagnosis was immediate. The system was suffering from a stack overflow. The stack trace looked like this:

# Child-SP RetAddr Call Site
00 000000ba`92851098 00007ff9`fed521c1 ntdll!_chkstk+0x37
01 000000ba`928510b0 00007ff9`feea5ace ntdll!RtlDispatchException+0x2d1
02 000000ba`92851300 00007ff9`fed4e02d ntdll!KiUserExceptionDispatch+0x2e
03 000000ba`92852060 00007ff9`fed5222f ntdll!RtlLookupFunctionEntry+0x8d
04 000000ba`928520b0 00007ff9`feea5ace ntdll!RtlDispatchException+0x33f
05 000000ba`92852800 00007ff9`fed4e02d ntdll!KiUserExceptionDispatch+0x2e
06 000000ba`92853560 00007ff9`fed5222f ntdll!RtlLookupFunctionEntry+0x8d
07 000000ba`928535b0 00007ff9`feea5ace ntdll!RtlDispatchException+0x33f
08 000000ba`92853d00 00007ff9`fed4e02d ntdll!KiUserExceptionDispatch+0x2e
09 000000ba`92854a60 00007ff9`fed5222f ntdll!RtlLookupFunctionEntry+0x8d
0a 000000ba`92854ab0 00007ff9`feea5ace ntdll!RtlDispatchException+0x33f
0b 000000ba`92855200 00007ff9`fed51f29 ntdll!KiUserExceptionDispatch+0x2e
0c 000000ba`92855f70 00007ff9`feea5ace ntdll!RtlLookupFunctionEntry+0x8d
0d 000000ba`928561c0 00007ff9`fed4e02d ntdll!RtlDispatchException+0x33f
...

The pattern is obvious: the sequence from RtlLookupFunctionEntry $\rightarrow$ KiUserExceptionDispatch repeats indefinitely.

The "Exception Death Spiral"

We have entered a recursive loop of failure. The logic follows this path:

1. An initial exception occurs.
2. The kernel determines the exception cannot be handled in kernel mode, so it reflects it back to user mode via KiUserExceptionDispatch.
3. While the system is attempting to locate the appropriate exception handler using RtlLookupFunctionEntry, another exception is triggered.
4. This second exception restarts the entire process.

The Result: The stack is consumed by these recursive calls until it is completely exhausted, leading to a final stack overflow exception that kills the process.

Mathematically, the stack depth $S$ grows as: $S_{total} = \sum_{i=1}^{n} (\text{frames per loop}) \approx n \times 3$ where $n$ is the number of recursive failures until $S_{total} > \text{StackLimit}$.

🕵️ Tracing the Root Cause

The bug was blamed on shell32 because it appeared to be the source of the first exception. To verify this, we have to look at the very bottom of the stack:

The sequence shows that the process was shutting down. The crash happened during the DLL detachment phase, specifically when shell32 was cleaning up via CoTaskMemFree.

🛠️ Next Steps in the Investigation

To understand why CoTaskMemFree failed, we need to examine the exception record. Since these records are passed to RtlDispatchException, we can inspect the arguments provided by KiUserExceptionDispatch.

• Identify the recursive loop.
• Trace the stack to the original faulting instruction.
• Analyze the exception record in RtlDispatchException.
• Determine why the memory was inaccessible.

Relevant Data Point:

# Child-SP RetAddr Call Site
243 000000ba`9294db70 00007ff9`feea5ace ntdll!RtlDispatchException+0x33f
244 000000ba`9294e2c0 00007ff9`fcba0af0 ntdll!KiUserExceptionDispatch+0x2e

---

Images from the original report: