Complete Fuzzing Series - Blog 4 of N
| Part | Title | Status |
|---|---|---|
| 1 | What Is Fuzzing and Why Does It Actually Work? |  |
| 2 | How a Fuzzer Works: Mutations, Seeds, and Corpus Explained | |
| 3 | Types of Fuzzing Explained: The Complete Fuzz Testing Guide | |
| 4 - You are here | What Happens Inside a Program When It Crashes | |
| 5 | Sanitizers: Your Fuzzing Superpowers |  |
| 6+ | More parts coming… | |
→ Next: Blog 5 - Sanitizers: Your Fuzzing Superpowers ← Previous: Blog 3 - Types of Fuzzing Explained: The Complete Fuzz Testing Guide
Part of the Complete Fuzzing Series on IoTSec.in
Before We Start - I Was Confused Here Too
In Blog 1, I told you that a crash is a signal. That a fuzzer’s job is to find the situations a developer never designed for. That every crash is worth looking at.
But I never explained what actually happens inside the program when it crashes.
Like, what does “the program crashed” even mean at a technical level? Where exactly does it break? Who notices? Who shuts it down? And when you see that wall of red text in your terminal - what is it actually telling you?
When I first started seeing crash outputs from AFL, I had no idea how to read them. I could see something went wrong. I could not tell what. I could not tell where. I definitely could not tell how bad.
That changes today.
By the end of this blog you will be able to look at a real crash output - the same kind researchers see when fuzzing actual CVEs - and read it like you wrote the program yourself.
Let’s build up to that from the ground.
Where Does a Program Actually Live When It Runs?
Before we can talk about crashes, we need to talk about where a program lives when it is running.
When a program is just sitting on your disk, it is dead. It is bytes in a file. The moment you run it, the operating system loads it into RAM and gives it a private chunk of space to work in. That space is called memory.
Think of RAM like a giant hotel. Billions of rooms, each one numbered. Those numbers are called addresses. Every piece of data your program needs - variables, function calls, temporary values - gets a room. The program reads and writes to those rooms constantly while it is running.
Now, not all rooms are used the same way. A running program’s memory is divided into regions, and each region has a specific job:
High addresses
┌─────────────────┐
│ Stack │ ← function calls, local variables, return addresses
├─────────────────┤
│ ↓ │ grows downward
│ │
│ ↑ │ grows upward
├─────────────────┤
│ Heap │ ← dynamically allocated memory
├─────────────────┤
│ Global / BSS │ ← variables that exist for the whole program
├─────────────────┤
│ Code (Text) │ ← the actual instructions of the program
└─────────────────┘
Low addresses
Two of these regions are where almost every interesting crash happens - the stack and the heap. Let’s talk about both.
The Stack - You Learned This in College. Here Is What Nobody Told You.
If you went through a CS or engineering degree, you saw this in your data structures class. Stack is LIFO - Last In, First Out. Like a stack of plates. You put plates on top, you take plates from the top. You memorized it, passed the exam, and probably never thought about it again.
Here is what nobody told you about why it actually matters.
When your program calls a function, the computer needs to remember three things:
- What values were passed into the function?
- What local variables did the function create?
- When the function finishes, where should execution go back to?
All of that gets packed together and pushed onto the stack as one unit. That unit is called a stack frame.
void greet(char *name) {
char message[50]; // local variable - lives on the stack
// ... do something
}
int main() {
greet("Alice");
// after greet() finishes, execution comes back HERE
}
When greet() is called, a stack frame is created for it. message[50] gets space right there on the stack. And right next to message[50], also on the stack, is the return address - the address of the instruction in main() that execution should go back to once greet() is done.
That detail is everything.
The return address sits on the stack right next to the local variables. Now ask yourself - what happens if someone writes more than 50 bytes into message[50]?
They overflow past the boundary. And right past that boundary is the return address.
Overwrite the return address with something you control - you have just told the program to jump somewhere you chose when the function returns. That is a stack buffer overflow. That is why the stack matters for security research. Not because of LIFO. Because of what lives next to what.
The Heap - Where Dynamic Memory Lives and Where It Gets Messy
The stack is automatic. You declare a variable inside a function, it appears. The function ends, it is gone. You manage nothing.
The heap is different. The heap is memory you ask for manually, at runtime.
char *buffer = malloc(100); // give me 100 bytes right now
// ... use buffer ...
free(buffer); // I'm done, give it back
You asked for it. You use it. You free it when done. The heap is for situations where you do not know at compile time how much memory you need - you only know after reading a file, after a user gives you input, after parsing a network packet.
The key difference from the stack:
| Stack | Heap |
|---|---|
| Automatic - the program manages it | Manual - you manage it |
| Fixed size, predictable | Flexible, grows as needed |
| Function ends → memory gone | Memory stays until you call free() |
Manual memory management means humans make mistakes. Two of the most dangerous bugs in security research live here:
Heap overflow - you write past the end of what you allocated. You corrupt whatever allocation sits next to yours in memory.
Use-after-free - you call free() on the memory, but then accidentally use the pointer again. The memory is gone. Something else might have taken that space by now. You are reading or writing into someone else’s data without knowing it.
Both of these can be exploitable. And both of them show up in the crash output we are going to read at the end of this blog.
Registers - The CPU’s Scratch Pad
RAM is large but relatively slow. The CPU cannot work out of RAM for every single operation. So the CPU has its own tiny, ultra-fast storage built directly into the chip. These are called registers.
Think of it like this. RAM is the notebook on your desk. Registers are what you are holding in your hand right now. Much faster to work with, but only a few at a time.
The registers that matter most for reading crash output:
| Register | What it does |
|---|---|
RSP |
Stack Pointer - points to the top of the stack right now |
RBP |
Base Pointer - points to the bottom of the current stack frame |
RIP |
Instruction Pointer - the next instruction the CPU will execute |
RAX, RBX… |
General purpose - calculations, temporary values |
The one that matters most for crashes is RIP.
RIP is always, at every moment, pointing to the next instruction the CPU will execute. The entire execution of your program is just RIP moving forward one instruction at a time. The CPU reads RIP, executes that instruction, moves RIP forward, repeat.
Now connect this back to the stack. When a function finishes, the return address stored on the stack gets loaded into RIP. That is how the program knows where to go back to.
So if an attacker overwrites the return address on the stack - they are not just corrupting a number. They are controlling what gets loaded into RIP. And controlling RIP means controlling what the CPU executes next.
That is the endgame of a stack buffer overflow. The whole goal is: control RIP, control the program.
One more thing - on 32-bit systems you will see
ESP,EBP,EIPinstead. Same registers, just 32-bit versions. Same concept. On 64-bit it is the R-prefix versions. You will see both in the real world so now you know.
What Happens at the OS Level When a Program Crashes
So the program is running. RIP is moving forward. Stack and heap are being used. Everything is normal.
Then the program tries to access a memory address it does not own. Maybe it tried to read address 0x00000000. Maybe it wrote past the end of a buffer and hit memory that belongs to no one.
The moment that happens, the hardware catches it.
Every process in Linux lives inside its own virtual address space. Your program thinks it owns a huge range of addresses all to itself. The browser running next to it thinks the same thing. They are both living in private illusions. Real physical RAM is shared underneath, but each process sees only its own world.
The component that manages this is called the MMU - the Memory Management Unit. It is hardware built into the CPU. Every single memory access your program makes goes through the MMU. The MMU checks: does this process have a valid mapping for this address?
If yes - fine, translate the virtual address to real physical RAM and continue.
If no - the MMU raises a page fault.
The OS kernel receives that page fault. Most page faults are completely normal - the kernel handles them silently and execution continues. But when the page fault is caused by a genuinely illegal access - an address that belongs to no one, or a write to a read-only region - the kernel decides this process did something it was not supposed to do.
The kernel then sends a signal to the process:
SIGSEGV - Signal 11 - Segmentation Fault
The process receives SIGSEGV. If the program has not written special code to handle that signal - and most programs have not - the process terminates immediately.
The full chain every time a crash happens:
Program accesses invalid address
→ MMU checks its mapping table, finds nothing valid
→ MMU raises a page fault
→ OS kernel receives the page fault
→ Kernel identifies it as an illegal access
→ Kernel sends SIGSEGV to the process
→ Process dies
→ Core dump written to disk
This entire chain happens in microseconds. From the bad memory access to the process being dead - faster than you can blink.
One thing I want you to notice - the OS does not crash. The kernel does not crash. Only the process dies. The OS gives every process its own private virtual address space specifically so that one bad program cannot take down the whole system.
That isolation is also why fuzzing is safe to run. You crash the target program hundreds of thousands of times and your machine stays up the entire time. The kernel just keeps collecting the signals and moving on.
Types of Crashes - Not All Crashes Are the Same
A crash is not just a crash. Different types tell you different things about what went wrong - and how serious it might be.
Segmentation Fault
The classic. The program accessed memory it was not allowed to touch. MMU caught it. OS sent SIGSEGV.
int *ptr = NULL;
*ptr = 5; // writing to address zero - instant segfault
Signal: SIGSEGV - signal 11
Stack Overflow
The stack has a size limit. It is not infinite.
Every function call pushes a new stack frame onto the stack. If your program calls functions too deeply - the most common cause is infinite recursion - the stack keeps growing until it spills into memory it does not own.
void infinite() {
infinite(); // calls itself forever - stack explodes
}
You learned recursion in college. This is what actually happens underneath when it goes wrong.
Signal: also SIGSEGV - because the stack overflowed into invalid memory
Heap Corruption
The program did something illegal with heap memory - wrote past an allocation, used memory after freeing it, freed the same pointer twice.
The dangerous thing about heap corruption: it often does not crash immediately. The corruption happens silently. The program keeps running. Then much later, when it tries to use that corrupted memory somewhere completely different, it crashes.
The crash location and the actual bug location can be far apart. That is what makes heap bugs hard to debug and very interesting to analyze.
Signal: SIGSEGV or SIGABRT depending on how it is caught
Null Pointer Dereference
The program tried to read or write through a pointer that was never set - it is still pointing at address zero, which is NULL.
char *ptr = NULL;
printf("%s", ptr); // trying to read from address 0
Usually not directly exploitable. But always a sign that the program’s logic is broken somewhere - a pointer that should have been initialized was not.
Signal: SIGSEGV - signal 11
Quick Reference
| Crash Type | What happened | Signal |
|---|---|---|
| Segmentation fault | Accessed invalid memory | SIGSEGV (11) |
| Stack overflow | Stack grew past its limit | SIGSEGV (11) |
| Heap corruption | Heap memory mismanaged | SIGSEGV or SIGABRT |
| Null pointer dereference | Read/write through NULL pointer | SIGSEGV (11) |
Notice something - most of these produce the same signal. SIGSEGV tells you that something went wrong. It does not tell you what or how bad. That is why you read the full crash output. The signal is the alarm. The crash output is the actual story.
Reading a Real Crash - CVE-2021-3156
This is where everything comes together.
In 2021, a security researcher fuzzed sudo - the program that almost every Linux system uses to run commands as root. What they found became CVE-2021-3156, also known as Baron Samedit. A vulnerability that existed undetected in sudo for 10 years and gave an attacker full root access on almost every Linux system in the world.
The crash that revealed it looked like this:
root:/pwd/sudo-1.8.31p2# /pwd/sudoedit -s '00000000\'
================================================================
==144165==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60200000157a
at pc 0x55b78a14a674 bp 0x7ffe25b32c40 sp 0x7ffe25b32c38
WRITE of size 1 at 0x60200000157a thread T0
#0 0x55b78a14a673 in set_cmnd /pwd/sudo-1.8.31p2/plugins/sudoers/./sudoers.c:868:10
#1 0x55b78a13ea47 in sudoers_policy_main /pwd/sudo-1.8.31p2/plugins/sudoers/./sudoers.c:306:19
#2 0x55b78a1126f5 in sudoers_policy_check /pwd/sudo-1.8.31p2/plugins/sudoers/./policy.c:872:11
#3 0x55b78a04c31b in policy_check /pwd/sudo-1.8.31p2/src/./sudo.c:1138:11
#4 0x55b78a0405d3 in main /pwd/sudo-1.8.31p2/src/./sudo.c:253:11
#5 0x7f52b843a0b2 in __libc_start_main /build/glibc-eX1tMB/glibc-2.31/csu/../csu/libc-start.c:308:16
#6 0x55b789f1fa1d in _start (/pwd/sudo-1.8.31p2/src/sudo+0x1bea1d)
0x60200000157a is located 0 bytes to the right of 10-byte region [0x602000001570,0x60200000157a)
allocated by thread T0 here:
#0 0x55b789f9815d in malloc (/pwd/sudo-1.8.31p2/src/sudo+0x23715d)
#1 0x55b78a149c3c in set_cmnd /pwd/sudo-1.8.31p2/plugins/sudoers/./sudoers.c:854:36
#2 0x55b78a13ea47 in sudoers_policy_main /pwd/sudo-1.8.31p2/plugins/sudoers/./sudoers.c:306:19
#3 0x55b78a1126f5 in sudoers_policy_check /pwd/sudo-1.8.31p2/plugins/sudoers/./policy.c:872:11
#4 0x55b78a04c31b in policy_check /pwd/sudo-1.8.31p2/src/./sudo.c:1138:11
#5 0x55b78a0405d3 in main /pwd/sudo-1.8.31p2/src/./sudo.c:253:11
#6 0x7f52b843a0b2 in __libc_start_main /build/glibc-eX1tMB/glibc-2.31/csu/../csu/libc-start.c:308:16
Let’s read it. Line by line. Like a researcher.
Line 1 - The Input That Triggered the Crash
root:/pwd/sudo-1.8.31p2# /pwd/sudoedit -s '00000000\'
This is the exact command that was run. A call to sudoedit with a specially crafted argument - a string ending with a backslash \. That trailing backslash is the trigger. The fuzzer found that this specific input causes the overflow. This is your crash-triggering input - the equivalent of the file AFL saves inside output/crashes/.
Line 2 - The Sanitizer Header
==144165==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60200000157a
at pc 0x55b78a14a674 bp 0x7ffe25b32c40 sp 0x7ffe25b32c38
Break this down piece by piece:
144165- the process ID of the sudo process that crashed. Nothing special, just an identifier.AddressSanitizer- ASan caught this. Remember - ASan is one of those tools that makes silent bugs loud. Without ASan this crash might have gone completely unnoticed. We will cover sanitizers in depth in Blog 5.heap-buffer-overflow- the type of crash. Not a segfault. Not a null pointer. A heap buffer overflow. Memory was written past the end of a heap allocation.0x60200000157a- the exact memory address where the illegal write happened.pc 0x55b78a14a674- program counter. This is RIP. The instruction that caused the overflow was sitting at this address.bp 0x7ffe25b32c40- base pointer. This is RBP. Where the current stack frame starts.sp 0x7ffe25b32c38- stack pointer. This is RSP. The top of the stack at the moment of crash.
You already know what RSP, RBP, and RIP are. Now you are seeing them in a real crash output from a real CVE.
Line 3 - What Exactly Happened
WRITE of size 1 at 0x60200000157a thread T0
WRITE- the program was writing, not reading. It tried to write data into memory it does not own.size 1- just one byte. One single byte written past the boundary. This is a classic off-by-one overflow. The allocated buffer ended at0x602000001579. The write went to0x60200000157a- exactly one byte too far.thread T0- happened on the main thread.
One byte. That is all it took. One byte written to the wrong address in sudo - a program that runs as root on virtually every Linux system - opened the door to full privilege escalation. That is why crash output matters even when it looks tiny.
The First Backtrace - Where the Crash Happened
#0 0x55b78a14a673 in set_cmnd sudoers.c:868:10
#1 0x55b78a13ea47 in sudoers_policy_main sudoers.c:306:19
#2 0x55b78a1126f5 in sudoers_policy_check policy.c:872:11
#3 0x55b78a04c31b in policy_check sudo.c:1138:11
#4 0x55b78a0405d3 in main sudo.c:253:11
#5 0x7f52b843a0b2 in __libc_start_main libc-start.c:308:16
#6 0x55b789f1fa1d in _start
The # number is printed right there in the output. I am not calculating anything - I am just reading what is on the screen.
#0 is always where the crash happened. #6 is always where the program started - _start is the very first thing any Linux program executes when the OS boots it up.
Read it bottom to top - that is the journey the program took:
_start
→ __libc_start_main
→ main (sudo.c line 253)
→ policy_check (sudo.c line 1138)
→ sudoers_policy_check (policy.c line 872)
→ sudoers_policy_main (sudoers.c line 306)
→ set_cmnd (sudoers.c line 868) ← CRASH HERE
The program started, went through main, went through a chain of policy checking functions, and landed in set_cmnd at line 868 of sudoers.c. That is the exact file and line where the overflow happened. A researcher now knows exactly where to look. No guessing.
The Second Section - Where the Memory Was Allocated
0x60200000157a is located 0 bytes to the right of 10-byte region
[0x602000001570, 0x60200000157a)
allocated by thread T0 here:
#0 malloc
#1 set_cmnd sudoers.c:854:36
This is AddressSanitizer being exceptionally helpful. It is not just telling you where the crash happened - it is telling you the complete life story of the memory that got corrupted.
- A 10-byte buffer was allocated starting at
0x602000001570, ending at0x60200000157a 0 bytes to the rightmeans the write landed exactly at the first byte past the end - the classic off-by-one- That buffer was allocated by
malloc, called fromset_cmndat line 854 of sudoers.c - The overflow happened at line 868 of the same function
So in the function set_cmnd:
- Line 854 allocates a 10-byte buffer on the heap
- Line 868 writes one byte past the end of that buffer
Fourteen lines apart. That is your bug, pinpointed to the exact source line. Without ASan, this would have been completely invisible. The program would have kept running. The corruption would have silently spread. Maybe crashed somewhere completely unrelated later. Maybe never crashed at all and just quietly leaked privileges.
This is what found a 10-year-old vulnerability hiding in plain sight.
Why Every Crash Is Worth Investigating
A crash is the program telling you: I found a situation I was not designed for.
Some crashes are boring on the surface - a null pointer in a rarely-reached error handler with no attacker-controlled data nearby. Not exploitable. Move on.
Some crashes look small and turn out to be everything - one byte written past a buffer in sudo. Ten years undetected. Root on every Linux system in the world.
You do not know which kind it is until you look.
That is why you never ignore a crash output. What looks like noise sometimes turns out to be the most important finding of the whole fuzzing campaign. What looks scary sometimes turns out to be harmless once you dig into it. The only way to know is to read it.
And now you can.
One More Thing - Crashes Are the Beginning, Not the End
The fuzzer finds crashes. Your job starts after.
Reading the output tells you what crashed and where. But it does not automatically tell you why or how bad. For that you need a debugger - GDB specifically. You take the crash-triggering input, load it in GDB, step through execution, watch the registers, inspect the stack, understand exactly what happened at every instruction.
That is a whole series on its own. If registers, stack frames, and reading program state in a live debugger interests you - that is exactly what we are building toward in the upcoming GDB debugging series on IoTSec.in. The fuzzing series finds the crashes. The debugging series teaches you how to fully tear them apart.
For now - you have everything you need to read a crash output and know what you are looking at.
What I Found Confusing (And Now Don’t)
“What is the difference between a crash and a segfault?” A segfault is one type of crash - specifically when the OS sends SIGSEGV because the program accessed invalid memory. Not every crash is a segfault. A heap corruption might trigger SIGABRT instead. The crash is the event. The signal is how the OS communicated it.
“If the signal is always SIGSEGV, how do I know what type of crash it is?” The signal just tells you the program died. The crash output - the backtrace, the ASan error, the register values - tells you the actual type. SIGSEGV for a null pointer dereference and SIGSEGV for a stack buffer overflow look identical at the signal level. You read the full output to know which one you are dealing with.
“What is AddressSanitizer? Is it always there?” No - ASan has to be compiled in deliberately. It instruments the binary to detect memory errors and report them loudly instead of silently. Without ASan, many of these crashes would never appear at all. The sudo crash above might have gone undetected forever without it. Blog 5 is entirely about sanitizers and why they are one of the most important tools in fuzzing.
“The backtrace has frame numbers - does #0 always mean the crash?” Yes. Always. #0 is the innermost frame - where execution was at the moment of the crash. The highest number is always the outermost frame - where the program started. Read top to bottom to go from crash point outward. Read bottom to top to follow the journey that led to the crash.
“Why does the second backtrace show malloc? That looks like the crash happened in malloc.” The second section in the ASan output is not where the crash happened - it is where the memory was allocated. ASan tracks every allocation and shows you its origin when that allocation is involved in a violation. It is extra context, not a second crash location.
“Is every heap overflow exploitable?” Not automatically. Exploitability depends on what lives next to the overflowed buffer in memory, how much you can control what gets written, and whether the overwritten data influences program execution. But every heap overflow is worth investigating because some of the most serious vulnerabilities - including CVE-2021-3156 - started as a single byte written one position too far.
What We Learned
Memory - When a program runs, the OS loads it into RAM and gives it a private chunk of space called the virtual address space. Every location in that space has a numbered address. The program reads and writes those addresses constantly while running.
Stack - The region of memory used for function calls. Each function call creates a stack frame containing local variables, parameters, and the return address - the location execution will go back to when the function finishes. Return addresses sitting next to local variables is what makes stack overflows dangerous.
Heap - The region of memory used for dynamic allocation. Memory you ask for manually with malloc() and release with free(). Bugs here include heap overflow (writing past the end of an allocation) and use-after-free (using memory after it has been freed).
Registers - Tiny, ultra-fast storage built directly into the CPU. RSP points to the top of the stack. RBP points to the bottom of the current stack frame. RIP points to the next instruction the CPU will execute - controlling RIP means controlling the program.
Virtual memory - Every process lives in its own private address space. The program sees virtual addresses. The MMU translates them to real physical RAM locations behind the scenes.
MMU (Memory Management Unit) - Hardware built into the CPU that translates virtual addresses to physical ones and enforces which addresses a process is allowed to access. When a process touches an address it does not own, the MMU raises a page fault.
Page fault - The signal the MMU sends to the OS kernel when a memory access cannot be resolved. Normal page faults are handled silently. Illegal ones result in the kernel sending SIGSEGV to the process.
SIGSEGV - Signal 11. Sent by the OS kernel to a process when it makes an illegal memory access. Most commonly known as a segmentation fault. The process receives this signal and terminates unless it has special handling code.
Core dump - A snapshot of the process’s memory, registers, and stack state at the exact moment of crash. Written to disk automatically. Security researchers analyze core dumps to understand what happened.
Stack overflow - A crash caused by the stack growing past its size limit, usually from infinite or excessively deep recursion. Produces SIGSEGV because the overflowing stack hits invalid memory.
Heap overflow - Writing past the end of a heap-allocated buffer, corrupting whatever memory sits next to it. Can be silent - the program might keep running after the corruption and crash much later somewhere unrelated.
Use-after-free - Using a heap pointer after the memory it points to has been freed. Frequently exploitable because the freed memory can be reallocated and filled with attacker-controlled data.
Null pointer dereference - Attempting to read or write through a pointer that is NULL (address zero). Usually produces SIGSEGV. Often not directly exploitable but always indicates broken program logic.
Off-by-one overflow - Writing exactly one byte past the end of a buffer. Looks tiny. Can be critical - CVE-2021-3156 was a one-byte heap overflow in sudo that gave root access on virtually every Linux system for 10 years.
AddressSanitizer (ASan) - A compiler-based tool that instruments a program to detect memory errors at runtime and report them loudly. Without ASan, heap overflows and use-after-free bugs often produce no visible crash at all. Blog 5 covers sanitizers in full.
Backtrace - The ordered list of function calls that were active at the moment of crash. Frame #0 is always where the crash happened. The highest frame number is always where the program started. Reading bottom to top shows you the journey that led to the crash.
→ Next: Blog 5 - Sanitizers: Your Fuzzing Superpowers
← Previous: Blog 3 - Types of Fuzzing Explained: The Complete Fuzz Testing Guide
Part of the Complete Fuzzing Series on IoTSec.in