If you’re involved in web hosting, you’re probably also involved in cleaning up hacked websites; especially if you have users running WordPress, Drupal or Joomla, which are all notorious for having endless security issues year after year, often due to the use of poorly written third party add-ons from people who have no business writing code.

In any case, the typical scenario is once someone hacks into a site, they add their own php files amongst the site’s legit files so they can get back in later even after the hole is patched.  These can often be very difficult to find.  However, if you still have a running process on the server, you may be able to figure out how it was started which will lead you back to the vulnerable, or malicious, file.

Here’s an example of finding a rogue php file that was used to spawn a rogue perl process, meaning the traditional technique of using lsof to find what file php was executing did not work.  ps just shows crap like this:

100    29530 1  0 14:47 ?        00:00:08 [perl]

The code was not in a file, so lsof just showed normal files in use:

perl      29530    username  txt       REG                8,2     7184       3938 /usr/bin/perl

However, some more useful arguments to the ps command will let you find the exact start time of the process, down to the second:

# ps -ax -o pid,comm,cmd,lstart|grep 29530
29530 username           [perl]             Wed Nov 26 14:47:42 2014

Okay, so let’s take that to the web server log and see what we find:

# grep 14:47:42 access_log
46.165.240.89 - - [26/Nov/2014:14:47:42 -0500] "GET /wp-content/rss.php HTTP/1.0" 200 200 "-" "Mozilla/5.0 (compatible; Goooglebot/2.1; +http://www.google.com/bot.html)"

So there we have it, someone managed to upload a malicious file via wordpress.  You would of course still need to secure wordpress, but at least you found how the rogue process was started.