RondoDox Part 3: How to Actually Counter It (Even When Your VPS Is Already Choking)

Research & Writeups
1

Incident Response Series, Part 3 of 3

Part Title Status
1 I Got Hit by a Botnet-for-Hire: The Full RondoDox Incident :white_check_mark:
2 RondoDox Botnet Full Malware Analysis, Attribution, and New IOCs :white_check_mark:
3 - You are here RondoDox Part 3: How to Actually Counter It (Even When Your VPS Is Already Choking) :white_check_mark:

Why This Post Exists

Part 1 was about what happened to me. Part 2 was about what the malware actually is. This one is about the part most guides skip: how do you actually fight this thing when your server is already gasping for air?

Here is the problem nobody warns you about. Most budget VPS providers, think Hostinger, Bluehost, DigitalOcean’s smallest droplets, give you one or two CPU cores and a modest amount of bandwidth. That is fine for running a small forum or a side project. It is not fine when a cryptominer decides to eat every CPU cycle you have.

When that happens, everything gets slow. Your SSH session lags so badly that every keystroke takes a second to show up. Commands that normally finish instantly hang for ten or twenty seconds. If you are on a metered bandwidth plan, the miner’s network traffic (or the traffic caused by the hosting provider auto-blocking you for abuse) can eat into your monthly limit too. I have read plenty of stories online of people who had a “VPS” but functionally could not use it while it was infected, because every action they tried to take competed with the malware for the same starved resources.

So you do not have unlimited time to sit there reading documentation while your admin panel times out for the fifth time. You need a plan you can execute fast, and you need to actually understand what you are doing, not just copy paste commands you found somewhere. That is what this post is for.

I am going to explain every concept from the ground up. What a process actually is. What persistence means. What chattr does and why it matters. What a system call is and why understanding that saved my server. And then I will give you the actual step by step workflow, the one you can follow the next time something like this happens to you, whether it is this exact malware or something completely different.

If you have not read Part 1 and Part 2 yet, I would start there. This post assumes you already know the shape of what RondoDox did. But even if you have not, everything here should still make sense on its own.

First, Understand What You Are Fighting

Before touching a single command, you need a mental model of what is actually happening on your machine. Skipping this step is why most “just run these commands” guides fail the moment your situation looks slightly different from the example.

What is a process?

Every program running on your server, whether it is your web server, your database, or a cryptominer, is a process. Linux gives every process a number called a PID (Process ID). You can see every running process with:

ps aux

Each row is one process. The columns tell you who owns it (USER), how much CPU and memory it is using, and the actual command that started it. This is your very first weapon. If something is eating your CPU, it will show up here with a suspiciously high number in the %CPU column.

What is persistence?

A single running process is fragile. Kill it, or reboot the server, and it is gone. Attackers know this, so they do not rely on just one running process. They set up persistence: a mechanism that automatically restarts their malware whenever the server reboots, or in more sophisticated cases, whenever you kill the process manually.

In Linux, there are several places persistence commonly hides. You need to know all of them, because an attacker who only sets up one is an amateur, and one who sets up seven (like the one in Part 1) is someone who has done this before.

  • Cron jobs, especially @reboot entries. Cron is Linux’s built in job scheduler. An @reboot line means “run this command every time the system boots up.” There are actually multiple places cron jobs can live: /etc/crontab, /etc/cron.d/, and each user’s personal crontab (crontab -l shows yours, but the files also live on disk at /var/spool/cron/crontabs/ on Debian based systems, or /var/spool/cron/ on some others).
  • systemd services. Modern Linux distributions use systemd to manage what starts at boot. A malicious .service file dropped in /etc/systemd/system/ and enabled will start right alongside your legitimate services.
  • SysV init scripts, the older mechanism systemd mostly replaced. These live in /etc/init.d/ and get linked into runlevel directories like /etc/rc3.d/.
  • /etc/rc.local. An old but still functional catch all script that runs at the very end of the boot process on many distributions.
  • Shell profile files, like ~/.bashrc or ~/.profile. These run every time a user opens a new shell, not just on reboot. This one is sneaky because it does not need a reboot to fire, just someone logging in again.

Why does this matter for speed? Because if you kill the process without checking all of these first, it will simply come back, and you will have wasted precious time and CPU cycles on a server that is already struggling to breathe.

The one rule that matters more than any command

Remove persistence before you kill the process.

If you kill the miner first, and it still has an active cron job or systemd service, it just restarts. Now you are fighting the same battle again, except your server has even less headroom left because you just burned time and resources on a fix that did not stick. Every guide, every checklist, every instinct you build from this point forward should follow this order: find persistence, disable persistence, then kill the process, then clean up the files.

What Is chattr, and Why Does It Matter So Much?

chattr is a Linux command that changes special file attributes, ones that go beyond the normal read, write, and execute permissions you are probably already familiar with. The two attributes that matter most in an incident like this are:

  • immutable (i): the file cannot be modified, deleted, renamed, or linked to, by anyone, including the root user, until the flag is removed.
  • append only (a): the file can only have data added to the end of it. Existing content cannot be changed or removed.

An attacker who sets chattr +ia on their malware files has just made your normal cleanup instincts useless. You will try rm -f malware_file, and it will fail with “Operation not permitted”, even though you are root. This confuses a lot of people, because we are trained to think root can do anything. Root normally can, but the immutable flag specifically blocks even root, unless you first clear the flag.

You check for these flags with lsattr:

lsattr /path/to/suspicious/file

If you see an i in the output, that file is immutable. If you see an a, it is append only.

Normally, clearing this is simple:

chattr -i /path/to/suspicious/file

But here is the twist that turned a five minute fix into a real problem for me: what if the chattr binary itself has been deleted?

That is exactly what happened. The attacker used chattr to lock their files, and then deleted the chattr binary itself, so that even if I figured out what was wrong, I would not have the tool to fix it. This is where understanding what is actually happening underneath the command becomes the difference between being stuck and being able to keep going.

What Is a System Call, and Why Should You Care?

This is the concept that actually saved me, and it is the one most guides never explain properly, so let me slow down here.

Every command line tool you use, chattr, rm, ls, all of them, is just a program that, under the hood, asks the Linux kernel to do something on its behalf. The kernel is the core part of the operating system that actually controls hardware, files, memory, and processes. Regular programs are not allowed to touch these things directly for safety reasons, so instead they make a request to the kernel called a system call, or syscall for short.

chattr is not magic. It is just a small program that, when you run chattr +i somefile, makes a specific syscall called ioctl (short for input output control) with a request code called FS_IOC_SETFLAGS, telling the kernel “please set these file attribute flags on this file.” That is it. That is the entire trick behind the command.

Here is why that matters: the chattr binary is just a convenience wrapper around that syscall. It is not the only way to make that syscall. If the binary is deleted, the underlying kernel feature it was using is still there, fully intact, waiting for any program willing to ask for it correctly.

This means you can write your own tiny program, in almost any language that can talk to the kernel directly, and get the exact same result as chattr, without chattr existing anywhere on the system. Here is the actual code I used, in Python, using a built in module called ctypes that lets Python call C library functions and, through them, make raw system calls:

import ctypes, os

libc = ctypes.CDLL('libc.so.6', use_errno=True)

FS_IOC_GETFLAGS = 0x80086601
FS_IOC_SETFLAGS = 0x40086602
FS_IMMUTABLE_FL = 0x10
FS_APPEND_FL = 0x20

for path in [b'/path/to/locked/file1', b'/path/to/locked/file2']:
    fd = os.open(path, os.O_RDONLY)
    flags = ctypes.c_int(0)
    libc.ioctl(fd, FS_IOC_GETFLAGS, ctypes.byref(flags))
    flags.value &= ~(FS_IMMUTABLE_FL | FS_APPEND_FL)
    libc.ioctl(fd, FS_IOC_SETFLAGS, ctypes.byref(flags))
    os.close(fd)

Walking through what this actually does:

  1. libc.so.6 is the C standard library, present on essentially every Linux system, even ones where chattr has been removed. It contains the low level ioctl function that talks to the kernel.
  2. FS_IOC_GETFLAGS and FS_IOC_SETFLAGS are just numeric codes the kernel recognizes as “give me the current flags” and “set these flags”, respectively. These numbers are not secret. They are documented in the Linux kernel source and header files.
  3. FS_IMMUTABLE_FL and FS_APPEND_FL are the specific bit values representing the immutable and append only attributes.
  4. For each locked file, we open it (just to get a file descriptor, a number the kernel uses to refer to that open file), read its current flags, clear the two bits we care about, and write the new flags back.

Once this ran, every file that had been “permanently” locked was instantly editable and deletable again. The attacker’s biggest trick, deleting the tool that clears the lock, turned out to be pointless, because the lock was never really about the tool. It was about a kernel feature the tool merely exposed.

The broader lesson, and this is the one I want you to actually take away: almost every command line tool on Linux is a thin wrapper around one or more system calls. When a tool is missing, broken, or deliberately sabotaged, ask yourself what syscall it would have made, and whether you can make that call yourself in a language you already have available (Python is preinstalled or easily installable almost everywhere). This mindset generalizes far beyond chattr. It is genuinely one of the most useful things you can understand about how Linux works.

The Full Workflow: What To Actually Do, In Order

This is the part you can bookmark. It is written to be general enough to apply to more than just RondoDox, because the next thing that hits your server will not look identical to this one.

Step 1: Confirm something is actually wrong

Before you panic, verify. Look at CPU usage sorted from highest to lowest:

ps aux --sort=-%cpu | head -20

Anything using an unusually high, sustained percentage of CPU that you do not recognize deserves a closer look. Pay attention to process names that look like they are trying to blend in. Real kernel threads appear in square brackets, like [ksoftirqd/0], and have no file path. If you see something that looks like a kernel thread name but is actually a real binary at a real path (check with ls -la /proc/<PID>/exe), that is a strong sign of disguise.

Step 2: Check every persistence location, not just one

Go through each of these, even if you already found something in an earlier one. Redundant persistence is common precisely because attackers expect defenders to stop looking after finding the first mechanism.

crontab -l
cat /etc/crontab
ls -la /etc/cron.d/
ls -la /var/spool/cron/crontabs/ 2>/dev/null
ls -la /var/spool/cron/ 2>/dev/null
systemctl list-units --type=service --all | grep -v "@"
find /etc/systemd/system -name "*.service" -newer /etc/hostname
ls -la /etc/init.d/
cat /etc/rc.local
grep -n "" ~/.bashrc ~/.profile 2>/dev/null | tail -40

That -newer /etc/hostname trick on the find command is genuinely useful. It shows you any systemd service file created more recently than your system was originally set up, which is a fast way to spot something new without knowing exactly what you are looking for.

Step 3: Check for locked files before you try to delete anything

lsattr /suspicious/path/here

If you see i or a flags and the chattr binary is missing or you simply want the more reliable method, use the direct syscall approach from the section above. Adjust the file paths for whatever you actually found.

Step 4: Check for a network backdoor

Attackers with root access often plant an SSH key so they can get back in even after you think you have cleaned everything.

cat ~/.ssh/authorized_keys
cat /root/.ssh/authorized_keys

Look for any key you do not recognize adding. If you find one, remove it immediately, and rotate every password and key on the box afterward, not just the compromised one. Assume anything the attacker could have touched should be treated as untrusted.

Step 5: Check what is talking to the network

ss -tunap
lsof -i -P -n

Look for connections to IP addresses or ports you do not recognize, especially high, unusual ports like 443 or 444 combined with a process name that has nothing to do with your actual applications. Cryptominers in particular tend to phone home to mining pool addresses over these kinds of connections.

Also check both firewall systems if your distribution has them, because they do not always talk to each other:

iptables -L -n -v
nft list ruleset

More than one admin has spent an hour confused about why traffic was not flowing, only to discover one firewall system was allowing it while a completely separate one was silently dropping it. Check both.

Step 6: Now, and only now, remove persistence

Go back through everything you found in Step 2 and remove it. Delete the cron entries, disable and delete the systemd service, remove the init.d script and its runlevel symlinks, clean out rc.local, and remove the suspicious lines from shell profile files.

Step 7: Kill the process

Only now, with every persistence mechanism gone, kill the actual running process.

pkill -9 -f "the_process_name"

If it comes back after this, you missed a persistence mechanism. Go back to Step 2 and look harder, particularly in the less obvious spots like the alternate cron spool directory or a systemd .timer unit instead of a .service.

Step 8: Delete the dropped files

With the immutable flags cleared and the process dead, you can now safely delete the actual malware files and any temporary files it created, commonly hiding in /tmp or /dev/shm.

Step 9: Verify with a reboot

This is the step people skip because they are tired and just want it to be over, and it is the step that actually proves you succeeded. Reboot the server, wait for it to come back up, and check everything again: process list, cron, systemd, network connections. A clean reboot with nothing reappearing is your actual confirmation, not a feeling of “I think I got it all.”

Working Fast When Your Resources Are Already Gone

If your server is so overloaded that even running these commands is painfully slow, a few things help:

  • Prioritize disabling persistence over anything else first. Even if the miner keeps running for a few more minutes while you work through the cron and systemd checks, that is fine. What you cannot afford is killing it, having it respawn, and burning through your limited CPU budget doing the same investigation twice.
  • Use nice and ionice on your own commands if the shell itself feels sluggish, to make sure your cleanup commands get scheduled ahead of the miner’s own processes where the scheduler allows it.
  • Watch your hosting provider’s abuse detection. Providers like Hostinger will often automatically firewall your ports if they detect outbound abuse traffic like cryptomining. If your site suddenly becomes completely unreachable from outside during an incident, check your hosting control panel for an abuse notice before assuming you broke something further. You may need to contact support to have the restriction lifted once you have actually cleaned the server.
  • Do not restart the whole server hoping it fixes itself before you have found and removed persistence. A reboot with active persistence just gives the malware a clean, fresh, less contested environment to relaunch in.

Preventing the Next One

None of this matters as much if the same door stays open. Once you are clean:

  1. Rotate every credential on the box. Root password, every user password, database passwords, application secrets. Assume the attacker saw everything.
  2. Disable SSH password authentication entirely, and use key based authentication only:
    sed -i 's/^#*PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
systemctl restart ssh
  3. Disable direct root SSH login, forcing a regular user plus sudo instead:
    sed -i 's/^PermitRootLogin yes/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
  4. Install a host based firewall and default to denying incoming traffic you have not explicitly allowed:
    apt install ufw
ufw default deny incoming
ufw allow ssh
ufw allow http
ufw allow https
ufw enable
  5. Install fail2ban to automatically block IPs hammering your SSH port with failed login attempts.
  6. Turn on automatic security updates so known vulnerabilities get patched without you needing to remember to do it manually.
  7. Actually read your logs occasionally. /var/log/auth.log will often show the early warning signs, repeated failed logins from the same address, long before an attacker succeeds.

Try It Yourself, Safely

Reading a checklist is one thing. Actually finding a disguised process, tracing every persistence mechanism, and getting the order of operations right under a bit of real pressure is a completely different skill, and it is the one that actually sticks.

So we built a harmless, fully contained clone of this exact malware. It behaves the same way (drops files, hides a process, sets up multiple persistence mechanisms, and even fights back if you remove things in the wrong order) but it cannot mine anything, cannot reach the network, and cannot touch anything outside its own sandbox. You get a real terminal, a real infected environment, and thirty minutes to find it and clean it properly.

Launch Machine

If you get it clean on the first try, congratulations, you already understand more than most people who have never had to deal with this. If it comes back on you once or twice before you figure out the right order, that is normal too. That is exactly how I learned it.

Glossary

Process - A running instance of a program, identified by a unique PID.

Persistence - Any mechanism configured to automatically restart a program, typically after a reboot or a kill attempt.

Syscall (system call) - A request a program makes to the operating system kernel to perform a privileged operation, like reading a file’s attributes or opening a network connection.

ioctl - A general purpose system call used for device and file specific control operations that do not fit neatly into standard read or write calls.

Immutable flag - A file attribute, set via chattr +i or the equivalent syscall, that prevents a file from being modified, deleted, or renamed by anyone until the flag is cleared.

Kernel space vs user space - Kernel space is where the operating system’s core code runs, with full hardware access. User space is where regular programs, including chattr, ps, and your web server, run with restricted access, communicating with the kernel through system calls.

Cron - Linux’s built in time based job scheduler.

systemd - The modern init system used by most current Linux distributions to manage services and what starts at boot.

Final Thoughts

The point of this post was never to make you memorize a list of commands. Command lists get outdated the moment the next piece of malware does something slightly different. What does not get outdated is understanding that a locked file is just a kernel flag, that a command is just a wrapper around a syscall, and that persistence has to be removed before the process, every single time, on every single kind of malware you will ever run into.

If you take one thing from this whole series, take that. The rest you can look up when you need it.

Series so far:

  • Part 1: What happened, and the seven backup plans
  • Part 2: What the malware actually is, dropper analysis and IOCs
  • Part 3: This post, how to counter it and practice it yourself
RondoDox Botnet Attack: How Hackers Rooted My VPS in Under 5 Minutes
RondoDox Botnet Malware Analysis: Dropper Reverse Engineering, Attribution, and New IOCs