RondoDox Incident Series - Part 1 of 2
| Part | Title | Status |
|---|---|---|
| 1 - You are here | I Got Hit by a Botnet-for-Hire: The Full RondoDox Incident |  |
| 2 | RondoDox Botnet Full Malware Analysis, Attribution, and New IOCs | |
| 3 | RondoDox Part 3: How to Actually Counter It (Even When Your VPS Is Already Choking) | |
A note before you read this: this is a rewrite. I published a first version of this story a while back. Then I sat down with the actual raw logs auth.log, bash history, journalctl, an IR session transcript instead of my own memory of that morning, and found out I’d gotten a few things wrong. This version keeps the mistakes in, on purpose, with corrections next to them. If you ever end up in this situation, I’d rather you learn from what I got wrong than read a version of the story where I quietly cleaned it up.
Before We Start
I run a small VPS. Nothing fancy a box in Mumbai, a couple of Docker containers, SSH key-only login, the kind of server you set up once and mostly forget about.
On June 13, 2026, at 6:32am, I wasn’t even looking for a hacker. My terminal AI assistant had crashed a little earlier, and I assumed it was an out-of-memory kill. So that’s what I went looking for.
I did not find an OOM kill. I found a botnet.
Part 1 - What I Was Actually Doing When I Found It
Here’s the honest version, straight from my own shell history, in the order I actually ran it:
free -h # how much RAM is free?
dmesg | grep -i "killed process" | tail -20 # did the OOM killer fire?
cat /sys/fs/cgroup/memory/memory.limit_in_bytes
opencode; echo "Exit code: $?" # confirming it actually crashed
which opencode
who
w
last | head -20
ps aux --sort=-%cpu | head -20 # <- this is where it happened
That last command was supposed to be a sanity check, not a discovery. I was looking for a runaway process eating RAM. What I got instead was a process near the top of the list I didn’t recognize, chewing CPU, with a name that looks completely legitimate at a glance:
USER PID %CPU COMMAND
root 1106 - /bin/softirq --randomx-1gb-pages -o [pool]:444 ...
ksoftirqd is a real Linux kernel thread. Every system has one. /bin/softirq, sitting in /bin/, taking command-line flags, is not. At 6am, half-asleep, hunting for an OOM issue, that’s exactly the kind of name your eyes are built to slide past.
I didn’t slide past it, mostly because the CPU number was too weird to ignore. Two lines later I’d killed it:
kill -9 1106 # the miner
kill -9 1043 # a second suspicious process
Correction I need to make here: in the original version of this post I described the miner’s live behavior in detail CPU percentage, memory footprint, the exact command line as if it came from a clean process capture. It didn’t. By the time any tool actually logged the process list, I’d already killed it by hand. Every number I quoted about the running miner came from a glance at my own terminal, relayed from memory afterward. It’s probably close to correct. It is not verified, and I shouldn’t have written it as fact. If you’re documenting an incident, the moment you see something suspicious in htop screenshot it, or ps aux --sort=-%cpu >> evidence.txt before you touch anything. I didn’t, and it cost me the ability to prove my own numbers later.
Part 2 - Going Back and Doing It Properly
I want to be upfront about something: the first version of this blog was written the same week as the incident, from memory and a quick look at logs, under some adrenaline. Months later I went back with the actual forensic artifacts full auth.log, complete bash history, the entire IR session transcript with real timestamps on every command and cross-checked what I’d written against what actually happened.
Some of it held up exactly. Some of it didn’t. This rewrite is the corrected version, and I’m calling out the corrections explicitly as I go, because I think the mistakes are as instructive as the incident itself.
Part 3 - How They Actually Got In
I’d been hunting through an open-source AI video generation tool as part of some vulnerability research, found a few real issues auth that could be bypassed, an upload endpoint with basically no checks and reported them through GitHub Security Advisories. To build the proof-of-concept, I deployed the app on my own server, in Docker, API bound to a high port.
I shared a PoC link with the maintainer as part of that report, then took the link down once the report was filed. What I didn’t do was tear down the actual server that PoC was running on it kept going in a background terminal session I lost track of, unaudited, for weeks. That forgotten box is the one this story is actually about.
On top of that, Docker does something a lot of people don’t find out until it bites them:
Docker inserts its own iptables rules that run ahead of UFW. Any container published to 0.0.0.0 is reachable from the internet regardless of what your host firewall thinks it’s blocking.
Here’s the part I have to be honest about: in the original post I wrote this as “UFW was up, only 22/80/443 were open, and Docker silently bypassed it” a clean, confident sentence. Going back through my own history, I can’t actually prove UFW was doing anything at all that night. I’d experimented with it earlier, on and off, but there’s no evidence it was actively enforcing anything at the moment of the attack, and an attempt to properly install and enable it after I found the miner failed silently and I didn’t notice for over an hour.
So the honest lesson isn’t “my firewall got bypassed.” It’s “I don’t actually know if my firewall was doing anything, and it wouldn’t have mattered anyway” because Docker’s rule insertion happens regardless of UFW’s state. That’s a worse position to be in, not a better one, and it’s the one I’d rather you learn from.
The real bug: three things stacked, not one
I went and checked the app’s code. Found this:
# router = new_router(dependencies=[Depends(base.verify_token)]) ← commented out
router = new_router() ← this is what actually ran
Authentication existed in the codebase. It was just switched off one commented-out line, zero enforcement. Combine that with an upload endpoint that only checked file extension, not content, and you get an attacker who can write arbitrary files to wherever the app is allowed to write.
Where the app was “allowed to write” turned out to be everywhere.
The part that took me the longest to actually understand: the volume mount
The docker-compose.yml had this:
volumes:
- ./:/MoneyPrinterTurbo # the whole host, writable, from inside the container
I stared at this line for a long time before it clicked, so let me try to explain it the way I eventually understood it as two doors into the same room, not two separate rooms:
Your host filesystem Inside the container
───────────────────── ───────────────────────
/root/MoneyPrinterTurbo/ <───────────► /MoneyPrinterTurbo/
Same folder. Two doors. Nothing gets copied —
whatever the container writes through its door
appears, instantly and for real, through your door.
As whatever user the container is running as.
A Docker volume isn’t a copy, and it isn’t a sandbox boundary. It’s the same directory, viewed from two different filesystem namespaces. The container wasn’t writing “into itself.” It was writing into my actual home directory, as root, because that’s what the container process was running as.
A related mix-up worth calling out on its own, because it tripped me up too: this is not the same thing as --network host. They sound similar and both start with “the container gets more access than usual,” but they open completely different doors:
| Setting | What it actually does |
|---|---|
-v ./:/app (volume mount) |
Filesystem escape. The container can read/write your host’s files directly. This is the bug that hit me. |
--network host |
Network escape. The container shares the host’s network stack instead of getting its own isolated one. No filesystem access implied at all. |
One bug lets an attacker write a file to your disk. The other lets a container’s traffic look like it’s coming from the host itself. Neither one implies the other. I had the first one, and I spent longer than I’d like to admit trying to explain it to myself in terms of the second before I realized they weren’t related.
The upload endpoint plus the mount plus the missing auth check meant: an unauthenticated request from the internet could write a file anywhere on my host filesystem, as root. The auth bypass and file-exposure issues are the ones I reported and got published as GHSA-hhqf-49vx-fv7x rated High severity, no CVE assigned as of this writing. What I didn’t fully appreciate when I filed that report: the same root cause auth disabled plus an exposed upload path was also enough, combined with how I’d deployed the container, to write arbitrary files to the host instead of just leaking existing ones. The bug I reported and the bug that got me weren’t two different incidents. They were the same hole, used two different ways.
Part 4 - The Attack Chain, and What I Can Actually Prove
Here’s where I have to slow down and be careful, because this is the part where the first version of this post overstated its own certainty.
The chain, as best I can reconstruct it, matching the known pattern of this botnet:
1. Masscan finds the exposed port
2. GET /docs → Swagger UI hands over every unauthenticated endpoint
3. A file lands on the host filesystem, unauthenticated, as root
4. A shell script runs, downloads the main dropper
5. Script disables defenses, kills competing malware
6. Dropper connects to a second C2 server
7. Dropper drops the miner binary and starts mining
8. Seven persistence mechanisms get planted, files made immutable
9. chattr, crontab, and iptables binaries get deleted to blind me
10. The botnet's C2 can now issue DDoS commands to my server at will
What the logs actually prove, once I stopped reconstructing from pattern-matching and looked only at what survived: the earliest artifact I have is a root-level command at 02:06:26 that runs the dropper directly. By the time logs pick this up, the attacker already had root. The upload step steps 1 through 3 above is consistent with the vulnerability I confirmed exists (I later proved it live, with a proof-of-concept script, months afterward) and with this botnet’s known behavior elsewhere. But the actual file that got uploaded that night, and any log of the request that put it there, didn’t survive the app’s storage directory was cleaned up during remediation before I thought to preserve it.
So: the vulnerability is real and verified. The general shape of the chain matches this botnet’s documented behavior everywhere else it’s been found. But “here’s exactly what request hit my server at exactly what second” for the first three steps is inference, not proof, and I should have said so the first time.
What is proven, second by second, starts here: 02:06:26 a root-privileged command runs the dropper. Seven seconds later, the miner binary is sitting on disk.
Step 10 is the one that actually mattered more than the mining. Cryptomining was side income. My server was, at that point, a member of a botnet that gets rented out to attack other people’s game servers and Discord communities.
Part 5 - The Persistence Was Absurd
After a reboot, the dropper reinstalled itself and planted seven separate ways to survive a wipe, within about a minute of each other:
Layer 1: systemd service
Layer 2: SysV init script + rc3.d symlink
Layer 3: /etc/crontab (@reboot)
Layer 4: /etc/cron.d/ entry (@reboot)
Layer 5: /var/spool/cron/root (@reboot)
Layer 6: /var/spool/cron/crontabs/root (@reboot)
Layer 7: /etc/rc.local (startup)
I deleted the systemd service. Killed the process. Rebooted to confirm it was gone.
It came right back because six other copies of the same instruction were still sitting there.
Assume there’s more than one persistence mechanism until you’ve actually checked all seven of these categories. One clean-looking kill is not a clean system.
Part 6 - The chattr Wall
This is the part of the story that actually held up exactly as I remembered it, because I documented it in real time as I was fighting it.
I went to delete the malware files. Permission denied as root.
$ rm /bin/softirq
rm: cannot remove '/bin/softirq': Operation not permitted
The attacker had run chattr +ia on every malware file immutable and append-only. The immutable flag overrides even root, by design. That’s the entire point of the flag existing.
So I went to clear it the normal way:
$ chattr -i /bin/softirq
bash: chattr: command not found
They’d deleted the chattr binary itself, right after using it. I tried reinstalling the package it comes from. That hung. It left a stuck package-manager lock behind, which then blocked every other attempt to fix anything through the package manager for the next several minutes. A few more attempts at clearing the flag through other tools failed with cryptic low-level errors nothing about this was going to be a one-liner.
The actual way out: chattr isn’t magic, it’s a thin wrapper around a syscall. ioctl() with FS_IOC_SETFLAGS is what clears the bit. Any language that can make syscalls can do the same thing chattr does, tool or no tool:
import ctypes, os
libc = ctypes.CDLL('libc.so.6', use_errno=True)
FS_IOC_GETFLAGS = 0x80086601
FS_IOC_SETFLAGS = 0x40086602
FS_IMMUTABLE_FL = 0x10
FS_APPEND_FL = 0x20
fd = os.open(b'/bin/softirq', os.O_RDONLY)
flags = ctypes.c_int(0)
libc.ioctl(fd, FS_IOC_GETFLAGS, ctypes.byref(flags))
flags.value &= ~(FS_IMMUTABLE_FL | FS_APPEND_FL)
libc.ioctl(fd, FS_IOC_SETFLAGS, ctypes.byref(flags))
os.close(fd)
Flags cleared instantly. Everything the attacker had “permanently” locked was deletable again, using the exact same syscall the deleted tool used to call under the hood.
They deleted the tool, not the syscall. Your defenses live in the userland tools you’re used to reaching for. The kernel API underneath them is always still there if you’re willing to go one layer lower.
I ran the same routine against every other locked file the cron entries, the crontab spool files same result each time.
Part 7 - Two More Things I Got Wrong the First Time
I want to correct these two explicitly, because they’re exactly the kind of mistake that’s easy to make under pressure and easy to repeat if nobody says it out loud.
The “SSH backdoor” wasn’t the attacker’s key
In the original post I described a planted SSH key as a quiet backdoor the attacker left for themselves. Going back through the actual key fingerprints and comparing them against my own known keys, that’s wrong. The key that got planted into a secondary account’s authorized_keys file was a copy of my own key same fingerprint, same identity, as the one I’d been using to log in for over a month before the attack even happened.
It was still added as part of the attacker’s persistence batch, deliberately, at the same moment as the other six persistence mechanisms. But there’s no log of it ever being used, and it isn’t attacker-controlled credential material it’s a copy of mine. I don’t have a clean explanation for why an attacker would plant a copy of the victim’s own key rather than one of their own, and I’m not going to pretend I do. I’m noting it as an open question rather than dressing it up as a “quiet way back in” like I did the first time.
The mystery “helper” binary wasn’t a helper
I found a small setgid binary during cleanup and originally described it, loosely, as some kind of watchdog or helper process. It isn’t. The system’s cron daemon was logging it performing crontab-style operations while the real /usr/bin/crontab binary had been deleted. It’s the attacker’s own drop-in replacement for crontab, used to manage persistence entries without needing the original tool.
One detail I still can’t resolve: its file timestamp says it was placed months before this incident closer to the previous September than to June. I don’t have an artifact that explains that gap. It might be a backdated timestamp, or it might mean this box had something on it long before June 13th that I never caught. I genuinely don’t know, and I’d rather say that than guess.
Part 8 - The Corrected Timeline
| Time (UTC) | Event | Confidence |
|---|---|---|
| 02:06 | Dropper runs as root, miner appears on disk 7 seconds later | Verified log + file timestamp |
| 02:10 | Multi-architecture payloads tested | Verified log |
| ~05:59 | Reboot; miner service starts automatically | Verified journal |
| ~06:21–06:23 | Second reboot | Verified boot record |
| 06:24 | Full redeploy all 7 persistence layers, immutable flags set, SSH key copy planted | Verified file timestamps |
| ~06:33–06:38 | I notice the compromise and kill the miner by hand | From my own shell history, no independent timestamp |
| 06:38 onward | Structured cleanup begins | Verified session log |
| ~06:47 | Immutable flags cleared via direct syscall call, malware deleted | Verified session log |
| ~07:12 | Mining pool IPs blocked at the firewall | Verified session log |
| ~07:22 | Docker breaks its own iptables binary was deleted too | Verified journal |
One correction that matters more than it looks: an earlier internal draft of my own notes had the persistence event at 06:24 down as the initial compromise. It wasn’t. The actual break-in was over four hours earlier, at 02:06. 06:24 was the attacker coming back to dig in properly, not the moment they got in. If you’re timelining your own incident, don’t assume the moment you noticed something, or the loudest event in your logs, is the moment it started. Check the file timestamps on the very first dropped binary that’s your real starting point.
What I’d Do Differently
Capture evidence before you touch anything. I killed the miner process before capturing a clean process list. I’ll never be able to independently verify what I saw with my own eyes that morning, and I have to caveat my own report because of it.
Don’t assume your firewall is doing what you think it’s doing. I believed UFW had my back. I still don’t know if it was even active that night, and it wouldn’t have mattered against this specific bug either way.
Check for deleted system binaries immediately, not after burning time wondering why a normal command stopped existing. A quick package-integrity check would have told me chattr was gone in seconds.
Separate what you’ve verified from what you’re inferring, in your own notes, as you write them not months later when you finally go back and check. I conflated “this matches the known pattern of this attack” with “this is what happened on my server” in the first draft, and it took a full forensic pass to catch it.
Rotate credentials during cleanup, not after. If an attacker still had a live session while I was cleaning up, they could have watched me work.
The Checklist, If This Happens to You
ps aux --sort=-%cpu | head -10 # what's actually eating your CPU — capture this before you kill anything
find / -type f -name "*.service" -newer /etc/hostname # recently added systemd services
lsattr /etc/crontab /etc/cron* /var/spool/cron/* # immutable flags on cron files
dpkg --verify | grep -v " c " # deleted or modified package files, chattr included
nft list ruleset # check FORWARD/OUTPUT policy, not just iptables
ss -tunap # suspicious outbound connections, especially to mining-pool-style ports
Final Thoughts
The mining was almost the least interesting part of this it’s the most common outcome of any opportunistic compromise, and by itself it’s boring. What actually mattered was the redundancy: seven persistence layers, a deliberately deleted chattr, a level of care that doesn’t look like an automated script running once.
That’s not automation. That’s someone who’s done this before.
They still made mistakes, and so did I, writing this up the first time. If you’re running a server right now check your cron files today. Capture your process list before you kill anything suspicious. And if you get something wrong in your own writeup afterward, say so it’s usually the most useful part for the next person reading it.
The full technical breakdown the miner and dropper internals, the persistence mechanics, and how this got attributed to a known botnet campaign is in Part 2.