A Forensic Overview of a Linux perlbot

It's fairly old news that exploit attempts against PHP, ColdFusion, and Content Management Systems are quite common these days.  Most of these attempts target old vulnerabilities, hoping to hit enough neglected servers to make it worthwhile.   In some cases, a particular exploit may succeed against a high value server, leading to some pretty significant data leakage.  In others, the attacker is going for volume, where they can make use of a large number of compromised hosts. 

I recently noticed a PHP exploit attempt coming from several HK and CN IP addresses, so I allowed the exploit on my honeypot and documented the results. The resulting activity was very reminiscent of the the early botnet days, with IRC Command and Control, as well as the use of the compromised host as a scanning tool.  

I thought it might be interesting to show this example of a successful PHP exploit against a linux server, what it does, and how you might go about some basic analysis. I'll conclude by showing how some basic memory forensics can be performed on this type of incident using Volatility. 

In this example, the remote attacker attempts the following PHP exploit:

PHP exploit attempt against CVE-2012-1823

This exploit attempt is against an old vulnerability (CVE-2012-1823) which allows a remote attacker to inject arbitrary code via command line options within the HTTP query string.
When decoded, the attack string is:

Decoded PHP exploit string

The rest of the HTTP POST reveals the attempted commands to be executed if the exploit was successful:

Attempted command injection string

This string of commands instructs the compromised server to do the following:

• Change working directory to /var/tmp
• Remove any file named "a.pdf" from that directory
• Download file "a.pdf" from a domain under the attacker's control and save it to /var/tmp. The "a.pdf" file is actually a perl script, which is given a "pdf" file extension.
• Execute the perl script "a.pdf"
• Finally, delete the perl script, "a.pdf"

To ensure downloading success of the remote script, the attacker repeats these instructions using 'curl', 'fetch', and 'lwp-get'.

Network packet capture reveals the overall activity of the injected script.  Upon execution, the script sleeps for awhile, then connects to the IRC C2 on 'vafel.pexit.cu.cc', port 45129.

Perl script connecting to remote IRC server on vafel.pexit.cu.cc

A short time later, the compromised host is instructed to fetch another script, "ins_h.sh" this time from m1.pexit.cu.cc

Compromised server instructed to download bash script 'ins_h.sh'

The contents of the "ins_h.sh" script shows among other things, the attacker creating hidden directories on the linux server, fetching source code of another tool (h.c), compiling it, and modifying the crontab.

Not long after this, a fair amount of assorted files were downloaded to the compromised host.  Among these include mining software, development libraries, and compiling tools. Also downloaded were a large collection of linux local privilege escalation exploits. 

After the attacker had the box configured the way he wanted it, activity related to Bitcoin and Primecoin mining soon began.   Of particular note was the use of the Stratum Mining Protocol connecting to a pool server on 37.251.139.161.

Bitcoin Stratum mining protocol traffic

Most servers that are injected with these various scripts are then used for a variety of tasks, including DDoS, vulnerability scanning, and exploiting.  The mining of virtual currency is now often seen running in the background during the attacker's "downtime".  The Internet Storm Center recently posted an entry, "The case of Minerd", which discusses the increasing use of compromised servers being used to mine virtual currency.  

Volatility

Let's now do a bit of memory forensics on the RAM image of this particular compromised host.  We'll use Volatility version 2.3.1 which has the ability to analyze Linux images.  In order to do this, you'll have to supply the appropriate Linux profile.  Creating a profile is very easy, but it needs to be done for the appropriate distribution and kernel in use.  I'd recommend bookmarking Ken Pryor's Github site, where Ken is building a repository of Linux Volatility profiles.

The system under analysis is an Ubuntu 10.04 server, with kernel version 2.6.32-33-generic-pae.

We'll first take a look at the active processes on the system via the 'linux_pslist' plugin.

'linux_pslist' output

Note the timestamps for PID 1517 (httpds) and PID 27157 (rsyslogd) are much later than those processes listed above it.  Since the earlier processes and their relative timestamps resemble a Linux startup routine, the two later processes are of interest.   Note also that these processes show a User ID and Group ID of 1002, which Ubuntu allocates to a user account.

The Volatility plugin, 'linux_psaux' behaves similar to the Linux command 'ps aux', so it is able to show  the command line arguments used at process invocation.  Running this against our image shows:

'linux_psaux' output

So the process named 'httpds', with PID 1517 was started with "/usr/bin/httpd" and process 'rsyslogd', with PID 27157 was invoked with command line arguments "-B -c".  We'll note to check 'httpd' in /usr/bin if we obtain a disk image of the compromised server.  Researching the 'rsyslogd' command, we learn that "-B" is not a valid option, so this process remains suspicious. 

While the pslist command finds running processes by walking the linked list of Linux task_struct-> tasks, the  'linux_pidhashtable' command walks the hashtable, which hashed by process identifier (pid).  This may help in finding hidden processes.

'linux_pidhashtable' output

Note the four additional  'rsyslogd' processes discovered by the 'linux_pidhashtable' plugin.

In cases like this, it's always a good practice to check the server's networking information.  Volatility has a number of networking plugins that will help identify remote connections and the processes that initiated them.  For example, the 'linux_netstat' plugin behaves just like the Linux 'netstat'  command, and will list active network connections, as well as listening sockets.  In this case, we see suspicious processes, PID 1517, and PID 27157 as the processes associated with established network connections to remote IP addresses.  

  

'linux_netstat' output

It's also a good idea to run the 'linux_route_cache' plugin, to display the routing table cache.  This may reveal any old connections that may not be seen via the 'linux_netstat' plugin.

'linux_route_cache' output

Now that we've established PIDs 1517 and 27157 as suspicious processes,  let's get a list of all the open files and paths associated with each.  Similar to using 'lsof' on a live, running Linux system, we'll use the Volatility plugin 'linux_lsof' here.  

'linux_lsof' output for PID 1517	'linux_lsof' output for PID 27157

Note that both PID 1517 and 27157 have two open files in common.  One is socket: [7916], and the other is the httpd.pid file in the hidden directory "/tmp/.ICE-unix/-log/".

Another useful Volatility plugin is 'linux_proc_maps'. This plugin will display the details of process memory, including shared libraries.  Details include the start and end location, inode and flag for each section.  This is very valuable information to be gleaned from memory during a forensic investigation.  For example, on PID 1517 we get the following:

'linux_proc_maps' output for PID 1517

We see the same hidden directory mentioned earlier, but now referring to filename 'httpds'.  The inode for this file is seen as 405961.  The 'linux_proc_maps' plugin has an output switch that allows for the dumping of the listed segments.  However to recover the complete, intact file, we need to recover it from the page cache, which holds in memory all the pages pertaining to a file.   We can do this via the 'linux_find_file' plugin.  This plugin will find the address to the file's inode, then allow you to dump the cached file contents from memory.  So for the 'httpds' file at inode 405961:

python vol.py -f /home/abc/pexit.vmem --profile=LinuxUbuntu1004_pae32-33x86 linux_find_file -F "/tmp/.ICE-unix/-log/httpds"

'linux_find_file' output for 'httpds' file

We then use the 'linux_find_file' plugin with the -O option, passing the address of the inode in order to extract the file.  After extraction, we can run 'strings' on it and see contents of significance:

'strings' of recovered 'httpds' file from hidden directory

These are just a few examples of how Volatility can assist in your forensic analysis of a compromised host.  A good deal of what is gleaned from Volatility can then be correlated to the disk image examination.  

I've posted the Linux memory image for this perlbot at the bottom of this post. Be sure to get the Linux profile, "Ubuntu1004_pae32-33.zip" from Ken Pryor's site.  You'll need it to run Volatility against this image. 

I hope this post shed some light on one example of a successful PHP exploit and the resulting script injection.  Besides ensuring that Internet facing servers are properly patched and hardened, knowing how to quickly track such a compromise should be part of best practices.  Across my honeypots, I'll see dozens of these a day, including Linux ELF files, perlbots, and vintage shells.   

While these injected perl and shell scripts are typically considered the patio gnats of the Internet, more annoying than anything else, they do have the potential to cause considerable harm. 

Memory image of pexit perlbot

Update - January 2, 2014

It looks like this particular perlbot has moved to "sdatdauy. [.]cu.cc"
The script download is at t1.sdatdauy. [.]cu.cc - port 35489
IRC C2 is on s1.sdatdauy. [.]cu.cc - port 35174