Recently one of my servers most probably got hacked (compromised). As I logged into my linode manager it said 200% bandwidth usage and it was growing at about 1gb per 1-2 mins (checked with iftop). So I opened a support ticket and asked a couple of linode fellows for help.
Just wanted to jot down all the steps I took to diagnose and fix.
What's the one thing every developer wants? More screens! Enhance your coding experience with an external monitor to increase screen real estate.
Web Server Logs
The first thing that’d probably come to one’s mind is to check the web server logs (in my case, thats apache logs). So I quickly take a look at the access and error logs but didn’t find anything fishy. For more information on the log formats check apache docs.
Examine the /tmp directory
This directory is often used by attackers to store their files in. Again, nothing suspicious found.
Contemplating Account Logins
Its quite possible that I might have fallen victim to an SSH brute force attack. So in order to work on this suspicion I went on auditing the file where all the authentication attempts are logged – /var/log/auth.log. This is where I found something dubious:
<datetime> <host> sshd[11223]: reverse mapping checking getaddrinfo for 197.51.174.61.dial.wz.zj.dynamic.163data.com.cn [61.174.51.197] failed - POSSIBLE BREAK-IN ATTEMPT! <datetime> <host> sshd[11223]: Invalid user admin from 61.174.51.197 <datetime> <host> sshd[11223]: input_userauth_request: invalid user admin [preauth] <datetime> <host> sshd[11223]: pam_unix(sshd:auth): check pass; user unknown <datetime> <host> sshd[11223]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.174.51.197 <datetime> <host> sshd[11221]: Failed password for root from 61.174.51.197 port 1675 ssh2 <datetime> <host> sshd[11221]: Disconnecting: Too many authentication failures for root [preauth] <datetime> <host> sshd[11221]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.174.51.197 user=root <datetime> <host> sshd[11221]: PAM service(sshd) ignoring max retries; 6 > 3 <datetime> <host> sshd[11223]: Failed password for invalid user admin from 61.174.51.197 port 2285 ssh2 <datetime> <host> sshd[11223]: pam_unix(sshd:auth): check pass; user unknown
Umpteen authentication failures like those that didn’t seem right.
It was also suggested to execute the last command that displays all user logged in (and out) by searching /var/log/wtmp and cross reference the data with the login attempts from the authentication log. Another command, lastlog, is also useful to display the most recent login of all the users on the system.
$ last median pts/0 122.178.204.9 Sun Mar 23 10:25 still logged in median pts/0 122.178.204.9 Sun Mar 23 08:52 - 10:22 (01:30) reboot system boot 3.12.9-x86_64-li Sun Mar 23 08:52 - 10:39 (01:47) median pts/0 122.172.146.173 Fri Mar 7 06:29 - down (02:14) median pts/0 122.172.146.173 Fri Mar 7 06:02 - 06:25 (00:22) root pts/0 27.40.69.157 Fri Mar 7 02:02 - 02:23 (00:21) reboot system boot 3.12.9-x86_64-li Thu Mar 6 13:52 - 08:43 (18:50) $ lastlog Username Port From Latest root pts/0 27.40.69.157 Fri Mar 7 02:02:12 +0000 2014 daemon **Never logged in** bin **Never logged in** sys **Never logged in** sync **Never logged in** games **Never logged in** man **Never logged in**
Note: It is quite possible that the records in authentication or web server or any sort of log has been falsified by the hacker, i.e., fake entries have been made into logs. The bad guy can even replace a lot of the commonly used programs with his own nasty version.
Check for Foreign Processes
Finally, reviewing the processes running on the server with commands like ps aux and htop to check for any service that might have gone astray helped out the most. I figured that a strange iTunes process was running, which is what was leading to so much of outbound traffic. This file was residing in /root.
$ sudo ls /root dlcfg fake.cfg iTunes QQiPPro
There were a couple of unknown binaries lying there that was scary.
Redeploying
Since it was quite difficult for me to determine the full scope of this attack, especially how my box was broken into, I decided to quickly setup a new server with all my services.
This time I disabled ssh password authentication and use SSH keys instead to massively improve security.
Conclusion
I didn’t notice any discrepancy with my data which could probably mean that the bad guy somehow wasn’t into the system but could just upload a couple of executable files (although I could be completely wrong with this assumption).
The reason why I did this writeup is to basically have a list of quick steps that can be performed in similar situations in future.
