Basic Steps Taken to Diagnose and Fix a Compromised or Heavy Resource Consuming Server

Recently one of my servers most probably got hacked (compromised). As I logged into my linode manager it said 200% bandwidth usage and it was growing at about 1gb per 1-2 mins (checked with iftop). So I opened a support ticket and asked a couple of linode fellows for help.

Just wanted to jot down all the steps I took to diagnose and fix.

What's the one thing every developer wants? More screens! Enhance your coding experience with an external monitor to increase screen real estate.

Web Server Logs

The first thing that’d probably come to one’s mind is to check the web server logs (in my case, thats apache logs). So I quickly take a look at the access and error logs but didn’t find anything fishy. For more information on the log formats check apache docs.

Examine the /tmp directory

This directory is often used by attackers to store their files in. Again, nothing suspicious found.

Contemplating Account Logins

Its quite possible that I might have fallen victim to an SSH brute force attack. So in order to work on this suspicion I went on auditing the file where all the authentication attempts are logged – /var/log/auth.log. This is where I found something dubious:

<datetime> <host> sshd[11223]: reverse mapping checking getaddrinfo for [] failed - POSSIBLE BREAK-IN ATTEMPT!
<datetime> <host> sshd[11223]: Invalid user admin from
<datetime> <host> sshd[11223]: input_userauth_request: invalid user admin [preauth]
<datetime> <host> sshd[11223]: pam_unix(sshd:auth): check pass; user unknown
<datetime> <host> sshd[11223]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=
<datetime> <host> sshd[11221]: Failed password for root from port 1675 ssh2
<datetime> <host> sshd[11221]: Disconnecting: Too many authentication failures for root [preauth]
<datetime> <host> sshd[11221]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=  user=root
<datetime> <host> sshd[11221]: PAM service(sshd) ignoring max retries; 6 > 3
<datetime> <host> sshd[11223]: Failed password for invalid user admin from port 2285 ssh2
<datetime> <host> sshd[11223]: pam_unix(sshd:auth): check pass; user unknown

Umpteen authentication failures like those that didn’t seem right.

It was also suggested to execute the last command that displays all user logged in (and out) by searching /var/log/wtmp and cross reference the data with the login attempts from the authentication log. Another command, lastlog, is also useful to display the most recent login of all the users on the system.

$ last
median   pts/0    Sun Mar 23 10:25   still logged in
median   pts/0    Sun Mar 23 08:52 - 10:22  (01:30)
reboot   system boot  3.12.9-x86_64-li Sun Mar 23 08:52 - 10:39  (01:47)
median   pts/0  Fri Mar  7 06:29 - down   (02:14)
median   pts/0  Fri Mar  7 06:02 - 06:25  (00:22)
root     pts/0     Fri Mar  7 02:02 - 02:23  (00:21)
reboot   system boot  3.12.9-x86_64-li Thu Mar  6 13:52 - 08:43  (18:50)

$ lastlog
Username         Port     From             Latest
root             pts/0     Fri Mar  7 02:02:12 +0000 2014
daemon                                     **Never logged in**
bin                                        **Never logged in**
sys                                        **Never logged in**
sync                                       **Never logged in**
games                                      **Never logged in**
man                                        **Never logged in**

Note: It is quite possible that the records in authentication or web server or any sort of log has been falsified by the hacker, i.e., fake entries have been made into logs. The bad guy can even replace a lot of the commonly used programs with his own nasty version.

Check for Foreign Processes

Finally, reviewing the processes running on the server with commands like ps aux and htop to check for any service that might have gone astray helped out the most. I figured that a strange iTunes process was running, which is what was leading to so much of outbound traffic. This file was residing in /root.

$ sudo ls /root
dlcfg  fake.cfg  iTunes  QQiPPro

There were a couple of unknown binaries lying there that was scary.


Since it was quite difficult for me to determine the full scope of this attack, especially how my box was broken into, I decided to quickly setup a new server with all my services.

This time I disabled ssh password authentication and use SSH keys instead to massively improve security.


I didn’t notice any discrepancy with my data which could probably mean that the bad guy somehow wasn’t into the system but could just upload a couple of executable files (although I could be completely wrong with this assumption).

The reason why I did this writeup is to basically have a list of quick steps that can be performed in similar situations in future.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download

Author: Rishabh

Rishabh is a full stack web and mobile developer from India. Follow me on Twitter.

Leave a Reply

Your email address will not be published. Required fields are marked *