Complete Fuzzing Series - Blog 5 of N
| Part | Title | Status |
|---|---|---|
| 1 | What Is Fuzzing and Why Does It Actually Work? |  |
| 2 | How a Fuzzer Works - Mutations, Seeds, and Corpus Explained | |
| 3 | The Taxonomy: Every Type of Fuzzing Explained | |
| 4 | What Happens Inside a Program When It Crashes | |
| 5 - You are here | Sanitizers: Your Fuzzing Superpowers | |
| 6+ | More parts coming… |  |
→ Next: Blog 6 - Coverage: The Feedback That Makes Fuzzing Smart
← Previous: Blog 4 - What Happens Inside a Program When It Crashes
Before We Start - AFL Found Crashes. But What About the Bugs It Didn’t Find?
I want to start with something that genuinely confused me after finishing Blog 4.
We spent all of Blog 4 learning about crash types. Stack overflow. Heap overflow. Use-after-free. Double free. Null pointer dereference. Integer overflow. I felt good. I understood what each one was. I understood how AFL catches them by watching for signals from the child process.
And then I hit this scenario at the end of Blog 4 and it stopped me cold.
Jack ran AFL for 24 hours. Found 3 crashes. Fixed them. Shipped his code. But there was a fourth bug. The JPEG parser was reading 10 bytes past the end of a buffer. The program kept running like nothing happened. No crash. No signal. AFL never saved that input. The bug was invisible.
And I sat there thinking wait. AFL is supposed to find bugs. That IS a bug. Why didn’t it find it?
That question is what this entire blog is about.
By the end of this blog you will understand:
- Why AFL is completely blind to a whole category of bugs
- What sanitizers are and how they work at a fundamental level
- The four main sanitizers and what each one watches for
- The tradeoffs of using them
- What to do when you don’t even have source code
Let’s go.
Part 1 - The Bug That Doesn’t Crash: Understanding Why AFL Goes Blind
Let’s go back to that fourth bug. The JPEG parser reads 10 bytes past the end of a buffer.
Before I explain anything, I want you to think about what AFL actually watches. We covered this in Blog 4. AFL watches for signals from the child process. SIGSEGV (sig:11). SIGABRT (sig:06). SIGFPE (sig:08). That is literally it. If the child process sends one of those signals, AFL saves the input. If not, AFL discards it and moves on.
So the question becomes: does reading 10 bytes past the end of a buffer send a signal?
Here is the thing that broke my brain at first.
It depends on WHERE those 10 bytes are.
If the buffer ends right at the edge of a memory page and those 10 bytes beyond it land in unmapped memory yes, the CPU tries to access memory the OS never gave to this process. The MMU fires. SIGSEGV sent. AFL catches it.
But if the buffer is in the middle of a heap allocation and those 10 bytes beyond it land inside memory that physically exists, memory that belongs to another variable or another chunk the CPU reads them just fine. No page fault. No signal. The OS has absolutely no idea anything went wrong.
SCENARIO A - buffer at edge of mapped page:
[ your buffer: 100 bytes ][ UNMAPPED MEMORY ]
↑
CPU tries to read here
MMU fires — page fault
SIGSEGV sent
AFL catches it ✅
SCENARIO B - buffer in middle of heap:
[ your buffer: 100 bytes ][ other data: belongs to something else ]
↑
CPU reads here just fine
those bytes physically exist
no page fault, no signal
program keeps running
AFL sees nothing ❌
Scenario B is what happened with Jack’s parser. The buffer was sitting in the heap. The 10 bytes past it belonged to some other variable. The CPU read them. Processed them. The program kept running with garbage data and nobody noticed.
This category of bug has a name: silent bugs.
Silent bugs are memory accesses that go out of bounds but do not crash because the memory they spill into physically exists. The CPU can reach it. The OS never fires. The program keeps running. And AFL, which is only watching for signals, sees absolutely nothing.
And here is the scary part about silent bugs beyond just “AFL misses them.”
Those 10 bytes Jack’s parser was reading past its buffer what were they? They were whatever happened to be sitting in heap memory next to that buffer. On Jack’s test machine, maybe it was zeros. Fine. But on a user’s machine, it could be anything. A password that was recently processed. An encryption key sitting in memory. Session tokens. Sensitive data from another part of the program.
The parser just read that sensitive data. Quietly. Without crashing. Without anyone knowing.
On a network-facing program, an attacker can sometimes craft input specifically designed to trigger this. The leaked memory gets included in a response. The attacker reads it. This class of vulnerability even has a famous real-world example the Heartbleed bug in OpenSSL was essentially this. An out-of-bounds read that leaked sensitive server memory over the network. No crash. Completely silent. For two years before anyone found it.
AFL would have missed Heartbleed too.
So now the question is obvious: how do we make AFL see these silent bugs?
Part 2 - The Problem: AFL Needs a Signal to See a Bug
Let me frame the problem precisely so the solution makes complete sense.
AFL needs a signal to save an input as interesting. Silent bugs produce no signal. Therefore AFL cannot see them.
The fix is simple in concept: make silent bugs produce a signal.
If we could somehow make the program scream produce a SIGSEGV the exact moment it reads byte 101 of a 100-byte buffer, then AFL would catch it. Save the input. Mark it as a crash. Bug found.
The problem is: how do you make a program scream when it does something that the OS, CPU, and hardware all consider perfectly fine?
You cannot change the OS. You cannot change the CPU. You cannot change how hardware works.
But you can change the program itself.
What if, at the moment that program was compiled, someone injected extra code into it — code that watches every single memory access and checks whether it is valid? Code that says “before you read that address, let me check if you are allowed to read there. If not, I will crash the program myself.”
That is exactly what sanitizers are.
A sanitizer is a tool that instruments your program at compile time to detect bad memory events the moment they happen even if they would not normally crash.
Part 3 - How ASAN Actually Works: Shadow Memory and Red Zones
The most important sanitizer for fuzzing is called AddressSanitizer, almost always written as ASAN.
To understand how ASAN works, I need to give you the analogy first.
Imagine every single byte of your program’s memory has a tiny notebook sitting next to it. In that notebook, one word is written: either “valid” or “poisoned”.
When your program calls malloc(100) and gets 100 bytes ASAN goes through and writes “valid” in the notebook next to each of those 100 bytes. Your program is allowed to read and write there.
But ASAN does not stop there. It looks at the bytes just before your buffer starts and just after your buffer ends. Those bytes were never given to your program. ASAN writes “poisoned” in their notebooks. These poisoned regions surrounding your buffer are called red zones. Think of them as a trip wire.
MEMORY LAYOUT WITH ASAN:
your 100-byte buffer
↓
[ REDZONE ][ byte0 ][ byte1 ]...[ byte99 ][ REDZONE ]
poisoned valid valid valid poisoned
↑ ↑
trip wire trip wire
before buffer after buffer
Now here is the key part. At every single memory access — every read, every write anywhere in the program ASAN checks the notebook for that address first.
program tries to read byte at address X
↓
ASAN checks notebook: what does it say for address X?
↓
valid? poisoned?
↓ ↓
continue normally STOP IMMEDIATELY
crash the program
print full report
tell you exactly what happened
So when Jack’s parser reads byte 101 of a 100-byte buffer:
WITHOUT ASAN:
byte 101 physically exists in heap memory
CPU reads it fine
no signal
AFL sees nothing
bug invisible ❌
WITH ASAN:
byte 101 is inside ASAN's red zone
notebook says: POISONED
ASAN immediately crashes the program
sends SIGSEGV
AFL sees signal — saves the input ✅
AND prints this:
==ERROR: AddressSanitizer: heap-buffer-overflow
READ of size 1 at address 0x602000000064
#0 parse_jpeg() jpeg_parser.c:47
#1 main() jpeg_parser.c:112
buffer of size 100 was allocated at:
#0 malloc()
#1 parse_jpeg() jpeg_parser.c:31
SUMMARY: heap-buffer-overflow on address 0x602000000064
Look at what ASAN just handed you. The exact file. The exact line number where the bad read happened. The exact line where the buffer was allocated. The type of bug. The operation read or write.
Compare that to what AFL gives you without ASAN:
id:000000,sig:11
That is a filename. You have to figure out everything else yourself.
With ASAN, the bug is basically already diagnosed.
How Does ASAN Handle Use-After-Free?
Now here is something I had to think hard about. ASAN catches more than just out-of-bounds access. It also catches use-after-free.
Remember from Blog 4 use-after-free is when you call free(buf) and then use buf again afterward. The memory no longer belongs to your program, but the pointer still holds the old address.
Without ASAN, this is a silent bug if the freed memory gets reallocated to something else. Two parts of the program share the same address without knowing. Corruption happens silently.
With ASAN the moment you call free(buf), ASAN marks all those bytes in the notebook as poisoned. Not valid, not a red zone specifically marked as freed memory.
BEFORE free():
buf → [ valid ][ valid ][ valid ]...[ valid ]
AFTER free():
buf → [ poisoned ][ poisoned ]...[ poisoned ]
"freed memory — do not touch"
Now when your code does buf[0] = 'A' after free:
ASAN checks notebook for buf[0]: POISONED (freed memory)
ASAN immediately crashes
Prints exactly:
heap-use-after-free
WRITE at line X
freed at line Y
allocated at line Z
Same mechanism. Shadow memory. Notebook. Check before every access. Poison means crash immediately.
This is why ASAN is so powerful for fuzzing. One tool. One mechanism. Catches multiple classes of silent bugs that AFL would never see.
How to Actually Use ASAN
Using ASAN is a single flag change when compiling with Clang or GCC:
# Normal compilation — no sanitizer
gcc -o jpeg_parser jpeg_parser.c
# With AddressSanitizer
clang -fsanitize=address -g -o jpeg_parser jpeg_parser.c
That -fsanitize=address flag tells the compiler: inject ASAN’s checking code into this binary. The -g flag keeps debug symbols so ASAN can report exact line numbers.
When using AFL specifically, you compile the target with AFL’s instrumentation AND ASAN together:
AFL_USE_ASAN=1 afl-clang-fast -o jpeg_parser jpeg_parser.c
That single environment variable tells AFL’s compiler wrapper to add ASAN instrumentation on top of AFL’s coverage tracking. One binary. Both tools working together.
Part 4 - The Tradeoff: ASAN Makes Everything Slower
Here is the honest truth about ASAN that I want to lay out clearly because a lot of tutorials skip it.
ASAN is checking every single memory access. Every read. Every write. Anywhere in the program. That is a massive amount of extra work on top of your actual program logic.
The real-world cost:
Normal binary:
execution speed: baseline (let's call it 1x)
memory usage: baseline
Binary with ASAN:
execution speed: ~2x slower (half the speed)
memory usage: 2x to 3x more RAM
For fuzzing, this matters. AFL runs millions of executions. If each execution takes twice as long, your fuzzing throughput gets cut in half.
Without ASAN:
1,000,000 executions per hour
finds: only bugs that produce signals on their own
misses: ALL silent bugs
With ASAN:
~500,000 executions per hour
finds: crash bugs AND silent bugs
misses: almost nothing
Is the tradeoff worth it?
For most security research yes. Half the speed but finding twice the bugs. Finding the silent bugs that would have shipped to production undetected. That trade is almost always correct.
But professional researchers do not choose one or the other. They run both simultaneously:
Machine 1: AFL + ASAN
slower
catches everything both loud and silent bugs
deep analysis mode
Machine 2: AFL without ASAN
full speed
pure coverage exploration
finds new code paths fast
feeds interesting inputs to Machine 1
Machine 1 and Machine 2 share a corpus directory.
Machine 2 explores fast. Machine 1 verifies deep.
This is called a fuzzing campaign. Multiple instances working together. Each doing what it is best at.
Part 5 - ASAN Is Not Enough: Meet the Sanitizer Family
When I first learned about ASAN I thought great, this catches everything. We are done.
I was wrong. ASAN is brilliant at memory boundary problems. Out of bounds. Use after free. Double free. But there are entire categories of bugs that ASAN would never catch.
Let me show you why.
The Bug ASAN Cannot See: Integer Overflow
Remember from Blog 4 Jack had this code:
void vulnerable(int len, char *input) {
char *buf = malloc(len + 1);
memcpy(buf, input, len);
}
When len is INT_MAX (2,147,483,647 the largest number a 32-bit signed integer can hold):
len + 1 = 2,147,483,647 + 1
= -2,147,483,648 ← wraps around to negative
malloc(-2,147,483,648) receives a garbage value. Allocates a tiny or nothing buffer. Then memcpy tries to copy 2 billion bytes into it. Crash.
Now would ASAN catch this?
Think about where ASAN sits. ASAN watches memory accesses. It checks notebooks. Valid or poisoned.
But the integer overflow happens before malloc is even called. The math wraps. The broken value goes into malloc. By the time ASAN records “buffer allocated, size = tiny broken number” ASAN genuinely believes that tiny size IS the correct buffer. Nobody told ASAN the size was calculated wrong. The math happened somewhere ASAN was not watching.
len + 1 overflows ← ASAN not watching here, this is arithmetic
↓
malloc(broken_size) ← ASAN records: "buffer of broken_size bytes"
↓ ASAN thinks this is correct
memcpy(buf, input, len) ← ASAN: "writing past end of buffer CATCH"
Actually wait ASAN does catch it eventually. But only at the memcpy stage. Not at the integer overflow itself. And there are cases where integer overflow causes wrong logic without any memory boundary violation ASAN misses those entirely.
This is where the second sanitizer comes in.
UBSan - Undefined Behavior Sanitizer
UBSan watches the math. Specifically, it watches for operations that the C standard declares “undefined behavior” things where the standard says “we don’t guarantee what happens here.”
Signed integer overflow is undefined behavior in C. Dividing by zero is undefined behavior. Reaching the end of a non-void function without returning is undefined behavior.
Why does “undefined” matter? Because when the C standard says something is undefined, the compiler is allowed to do literally anything including optimize in ways that silently break your program in ways you never expected.
UBSan catches these at the moment they happen:
int len = INT_MAX;
int result = len + 1; // UBSan: signed integer overflow on line 2
// CRASH — full report immediately
==ERROR: UndefinedBehaviorSanitizer: signed-integer-overflow
#0 vulnerable() test.c:2
2147483647 + 1 cannot be represented in type 'int'
Think of ASAN and UBSan as two specialists watching two completely different parts of your code:
Your C program:
[arithmetic operations] → [memory gets allocated] → [memory gets accessed]
↑ ↑
UBSan sits here ASAN sits here
watches the math watches boundaries
They inject watchers at completely different points. Neither overlaps with the other. You need both.
To compile with UBSan:
clang -fsanitize=undefined -g -o jpeg_parser jpeg_parser.c
# Or combine ASAN and UBSan together — very common:
clang -fsanitize=address,undefined -g -o jpeg_parser jpeg_parser.c
MSan - Memory Sanitizer
This one confused me the most when I first read about it. Let me show you the bug first.
int buf[10];
// declared — but never assigned any values
// buf contains whatever random bytes were in that memory location before
printf("%d\n", buf[5]); // reading from buf without ever writing to it
Is this an out-of-bounds access? No. buf has 10 integers. We are reading index 5. Perfectly within bounds. ASAN’s shadow memory says: valid.
Is there undefined math? No. We are just reading. UBSan has no complaint.
So what is wrong?
buf[5] contains garbage. Whatever happened to be sitting in that memory before this function ran. On different machines, different runs, different conditions it contains completely different values. The program is making decisions based on random leftover data.
This is called an uninitialized memory read. And it is dangerous for exactly this reason: an attacker can sometimes control what garbage is in that memory. Your program then makes security decisions based on attacker-controlled data without ever knowing the data came from an attacker.
This is the bug MSan catches:
int buf[10] declared
→ MSan marks all 10 slots as UNINITIALIZED
buf[5] read
→ MSan checks: was buf[5] ever written to?
→ No
→ CRASH. Uninitialized memory read. Line reported.
==ERROR: MemorySanitizer: use-of-uninitialized-value
#0 main() test.c:4
ASAN misses this. UBSan misses this. MSan catches exactly this declared but never initialized, then read.
To use MSan:
clang -fsanitize=memory -g -o jpeg_parser jpeg_parser.c
One important note: MSan cannot be combined with ASAN in the same binary they conflict with each other at the instrumentation level. You run them in separate fuzzing campaigns.
TSan - Thread Sanitizer
This one is completely different from the other three. The other sanitizers watch what happens to memory. TSan watches who is touching memory and when.
Let me explain threads briefly because this is where a lot of beginners get lost.
Modern programs can run multiple threads simultaneously multiple workers inside the same program doing different things at the same time. A web server might have one thread handling your request while another thread handles someone else’s request. They share the same memory space.
Here is where it gets dangerous.
Imagine two threads Thread A and Thread B both working with the same variable:
int counter = 0;
// Thread A does this:
counter = counter + 1;
// Thread B does this at the same time:
counter = counter + 1;
This looks simple. Both threads add 1. Counter should be 2.
But here is what actually happens at the CPU level:
Thread A:
1. Read counter from memory → gets 0
2. Add 1 → gets 1
3. Write 1 back to memory
Thread B (running simultaneously):
1. Read counter from memory → gets 0 ← reads BEFORE Thread A writes back
2. Add 1 → gets 1
3. Write 1 back to memory ← overwrites Thread A's result
Final value of counter: 1. Not 2.
One increment got silently lost. No crash. No signal. The program keeps running with wrong data.
This is a race condition. Two threads racing to access the same memory without coordination. The result depends on timing who reads and writes first. It does not happen every time. Only when the timing lines up in a specific unlucky way. Extremely hard to reproduce. Extremely hard to find manually.
TSan catches this by tracking which threads access which memory locations and flagging when two threads touch the same location without proper synchronization:
==ERROR: ThreadSanitizer: data race
Write of size 4 in Thread 1
#0 increment() test.c:8
Read of size 4 in Thread 2
#0 increment() test.c:8
Why would ASAN, UBSan, and MSan all miss this? Because it is not about memory boundaries, arithmetic, or initialization. It is about thread timing two workers colliding on the same resource. Completely separate dimension of bugs.
clang -fsanitize=thread -g -o server server.c
Part 6 - The Full Sanitizer Family: A Map
Let me lay out all four sanitizers in one place so the picture is complete:
WHAT WENT WRONG → WHICH SANITIZER CATCHES IT
─────────────────────────────────────────────────────────
Reading past end of buffer ASAN (AddressSanitizer)
Writing past end of buffer ASAN
Use-after-free ASAN
Double free ASAN
INT_MAX + 1 wrapping UBSan (Undefined Behavior Sanitizer)
Division by zero UBSan
Null pointer before checking UBSan
Signed overflow UBSan
int buf[10]; read buf[5] MSan (MemorySanitizer)
without initializing it
Reading uninitialized memory MSan
Two threads writing to TSan (ThreadSanitizer)
same variable simultaneously
Race condition TSan
Four specialists. Four completely different categories. All bugs that AFL would miss without them. All made loud they crash the program the instant the bad thing happens and print exactly what went wrong.
The typical combination for fuzzing:
Campaign 1: AFL + ASAN + UBSan ← most common starting point
Campaign 2: AFL + MSan ← separate campaign, conflicts with ASAN
Campaign 3: AFL + TSan ← for multithreaded targets specifically
Campaign 4: AFL alone ← pure speed, maximum coverage exploration
Run them in parallel if you have the machines. Let them all feed into a shared corpus.
Part 7 - What If You Don’t Have Source Code?
This is the question I kept coming back to throughout this whole topic, and I want to address it properly because if you are into IoT security and you are, that is why you are here this is your reality constantly.
You pull firmware off a router. You extract binaries. You want to fuzz them. No source code. No C files. Just compiled machine code sitting there.
You cannot use ASAN. You cannot use UBSan. You cannot use MSan or TSan. The compiler already ran. You missed your window. The sanitizers inject their watching code at compile time and compile time is long gone.
So what do you do?
Valgrind
The tool that fills this gap is called Valgrind. It does a similar job to ASAN watches memory accesses, catches out-of-bounds reads and writes, catches use-after-free but it works at execution time on an already-compiled binary.
How? Valgrind acts as a middleman between the CPU and your program. Every single instruction the program tries to execute gets intercepted by Valgrind first. Valgrind examines it. Checks it. Then allows it to execute.
WITH ASAN (compile time instrumentation):
program runs
→ watcher code already baked inside the binary
→ CPU executes program + watchers natively
→ fast — no middleman
WITH VALGRIND (runtime instrumentation):
program tries to execute instruction X
→ Valgrind intercepts it
→ Valgrind examines it: "is this a memory access? is it valid?"
→ Valgrind allows execution
→ next instruction intercepted again
→ and again
→ and again
→ every single instruction goes through Valgrind's hands
That middleman cost is brutal in terms of speed:
Normal execution: 1x speed (baseline)
With ASAN: ~0.5x speed (2x slower)
With Valgrind: ~0.05x speed (10x to 20x slower)
Valgrind makes your program 10 to 20 times slower. When fuzzing millions of executions, that difference is enormous.
But when you have no source code Valgrind is sometimes the only tool you have.
HAVE SOURCE CODE?
↓ yes
Compile with ASAN/UBSan/MSan/TSan
Fast, comprehensive, ideal
NO SOURCE CODE? Binary only?
↓
Valgrind — slow but works without recompiling
Or emulation-based fuzzing (QEMU mode in AFL)
Or full-system fuzzing
We cover all of these in Phase 8 of this series
The IoT-specific fuzzing techniques how to handle closed firmware, MIPS binaries, ARM binaries, no-MMU targets that is its own phase. Phase 8. For now understand the concept: no source code means no compile-time sanitizers, which means you either accept slower runtime tools or you find creative workarounds.
Part 8 - Putting It All Together: Jack’s Complete Setup
Let me bring everything back to Jack and show you what a real fuzzing setup looks like with sanitizers in the picture.
Jack has his JPEG parser. Source code available. He wants to fuzz it properly.
Step 1 - Compile with AFL + ASAN:
# Install AFL and clang if not already present
sudo apt install afl++ clang
# Compile with AFL instrumentation + ASAN
AFL_USE_ASAN=1 afl-clang-fast -g -o jpeg_parser_asan jpeg_parser.c
# Also compile a fast version without ASAN for coverage exploration
afl-clang-fast -g -o jpeg_parser_fast jpeg_parser.c
Step 2 - Set up corpus directory:
mkdir -p corpus crashes
# add a few small valid JPEG files as seeds
cp sample.jpg corpus/
Step 3 - Launch Campaign 1 (ASAN, deep analysis):
AFL_USE_ASAN=1 afl-fuzz -i corpus -o output_asan -- ./jpeg_parser_asan @@
Step 4 - Launch Campaign 2 (fast, coverage exploration):
afl-fuzz -i corpus -o output_fast -- ./jpeg_parser_fast @@
Now both are running. Campaign 2 explores fast. Campaign 1 catches silent bugs.
When Campaign 1 finds something what Jack sees in the crash file:
output_asan/crashes/id:000000,sig:11,src:000003,op:havoc,rep:4
And when he runs that crash file manually against the ASAN binary:
./jpeg_parser_asan output_asan/crashes/id:000000,sig:11,src:000003,op:havoc,rep:4
He gets the full ASAN report. Exact file. Exact line. Exact bug type. The silent bug that AFL alone would have completely missed diagnosed in seconds.
That fourth bug that shipped to production? With ASAN in the setup, it would have been caught in the fuzzing campaign. Before release. Before anyone could exploit it.
What I Found Confusing (And Now Don’t)
“AFL watches for signals so why doesn’t an out-of-bounds read send a signal?”
Because signals only fire when the CPU tries to access memory the OS never gave to the process. If the out-of-bounds bytes are inside memory that physically exists other variables, other heap chunks the CPU reads them just fine. No page fault. No signal. The OS has no idea anything went wrong. That is the definition of a silent bug.
“ASAN injects at compile time. But the program runs at runtime. How does compile-time code watch runtime behavior?”
The compiler injects the watching code INTO the binary. It becomes part of the binary itself. So at runtime, the program is running both the original logic AND the ASAN watchers simultaneously, natively, as one binary. There is no external tool watching — the watcher is inside.
“I thought ASAN knows every buffer size at compile time. Why can’t it catch integer overflow?”
ASAN watches memory accesses. It does not watch arithmetic. The integer overflow happens during a calculation len + 1 before any memory is touched. ASAN’s instrumentation is not injected around arithmetic operations. That is UBSan’s job. Two different specialists injecting code at two different points.
“Can I just use all sanitizers at once?”
ASAN and UBSan work together fine -fsanitize=address,undefined. But MSan conflicts with ASAN (they both try to instrument the same memory tracking and interfere with each other) so MSan needs its own separate campaign. TSan also conflicts with ASAN and needs its own campaign.
“Valgrind and ASAN do similar things. Why is Valgrind so much slower?”
ASAN is baked into the binary at compile time. It runs natively the CPU executes both the program code and the ASAN checking code as one unit. No middleman. Valgrind is a runtime middleman. It intercepts every instruction before the CPU executes it, examines it, then allows it. That interception happens for every single instruction. The overhead is enormous 10x to 20x.
“What about IoT firmware where I have no source code at all?”
Valgrind is one option slow but requires no source code. AFL also has QEMU mode (-Q flag) which can fuzz binary-only targets using dynamic translation without source code. And for embedded targets running on specific architectures, there are emulation-based approaches. All of that is Phase 8 of this series.
Glossary
Silent bug - A memory access bug that does not crash the program because the memory being accessed physically exists. The CPU reads or writes successfully. No signal is sent. AFL never detects it. The program keeps running with garbage or corrupted data.
Sanitizer - A compile-time instrumentation tool that injects checking code into a binary during compilation. The instrumented binary detects bad memory events the moment they happen and crashes with a detailed report — even for events that would not normally crash.
Compile-time instrumentation - The process of injecting extra code into a binary during compilation, before the binary is built. The injected code becomes part of the binary and runs natively alongside the original program logic.
AddressSanitizer (ASAN) - A sanitizer that catches memory boundary violations. Tracks which memory is valid and which is poisoned using shadow memory. Catches out-of-bounds reads and writes, use-after-free, and double free. Makes the program roughly 2x slower.
Shadow memory - ASAN’s internal bookkeeping system. A parallel region of memory where ASAN records the status of every byte in the program’s address space. Each byte is marked either valid or poisoned. Every memory access is checked against shadow memory before it proceeds.
Red zone - A region of poisoned bytes ASAN places immediately before and immediately after every allocated buffer. If any access lands in a red zone, ASAN crashes the program immediately. Red zones act as tripwires for out-of-bounds access.
Poisoned memory - Memory that ASAN has marked as invalid in shadow memory. This includes red zones around buffers, freed heap memory, and unallocated regions. Any access to poisoned memory triggers an immediate crash with a full report.
UBSan (Undefined Behavior Sanitizer) — A sanitizer that catches undefined behavior in C arithmetic and logic. Catches signed integer overflow, division by zero, null pointer dereference before check, and other operations the C standard declares undefined.
Undefined behavior - Operations in C where the language standard makes no guarantee about what happens. Signed integer overflow is undefined behavior — the result can be anything. The compiler is free to optimize based on the assumption that undefined behavior never occurs, which can cause subtle and dangerous bugs.
MSan (MemorySanitizer) - A sanitizer that catches uninitialized memory reads. Tracks whether each memory location has ever been written to before being read. If a program reads a variable that was declared but never assigned a value, MSan crashes immediately.
Uninitialized memory read - Reading a variable or buffer that was declared but never assigned a value. The value read is garbage — whatever happened to be in that memory location previously. Dangerous because the garbage can be attacker-controlled in some scenarios.
TSan (ThreadSanitizer) - A sanitizer that catches race conditions between threads. Tracks which threads access which memory locations and flags when two threads read and write the same location without proper synchronization.
Race condition - A bug that occurs when two threads access the same memory simultaneously without coordination. The result depends on timing — which thread reads and writes first. Does not crash every time. Only triggers when timing lines up in a specific way, making it extremely hard to find manually.
Thread - A separate worker inside a program that runs code simultaneously with other threads. Threads share the same memory space, which creates the possibility of race conditions when they access the same data at the same time.
Valgrind - A runtime instrumentation tool that watches memory accesses in already-compiled binaries. Works without source code by intercepting every CPU instruction at runtime and checking it. Catches similar bugs to ASAN but is 10x to 20x slower because it acts as a middleman for every instruction.
Runtime instrumentation - Adding watching behavior to a program at execution time, after it has already been compiled. Valgrind does this by intercepting and translating instructions on the fly. Slower than compile-time instrumentation because of the constant interception overhead.
Heartbleed — A real-world vulnerability in OpenSSL (CVE-2014-0160) caused by an out-of-bounds read — a silent bug. The program read past the end of a buffer and leaked whatever was in adjacent memory, including private keys and passwords. No crash. Silent for two years before discovery.
Fuzzing campaign - Running multiple instances of a fuzzer simultaneously, often with different configurations. A common setup is one instance with ASAN for deep bug detection and one instance without ASAN for fast coverage exploration, sharing a corpus directory.
AFL_USE_ASAN - An environment variable that tells AFL’s compiler wrapper to inject ASAN instrumentation into the target binary during compilation. The resulting binary has both AFL’s coverage tracking and ASAN’s memory checking.
-fsanitize=address - The Clang/GCC compiler flag that enables AddressSanitizer. Passed at compile time, it instructs the compiler to inject ASAN’s shadow memory tracking and access checking code into the binary.
Crash triage - The process of taking a crash file found by AFL and investigating it to understand the exact bug type, location, and exploitability. ASAN dramatically speeds up triage by providing exact file names, line numbers, and bug classifications in the crash report.
→ Next: Blog 6 - Coverage: The Feedback That Makes Fuzzing Smart
← Previous: Blog 4 - What Happens Inside a Program When It Crashes