Collecting Volatile Data with AWK

On a recent forensics case, a coworker and I noticed some interesting logs on a Linux web server. TCPDump showed some strange traffic from a handful of IPs, but the access logs were not showing any visits from the offending addresses. The traffic was encrypted so it wasn’t possible to see what was being sent, so we needed to do some additional digging. A lot is required to take a memory image of a Linux machine, so we decided to go with something faster and more targeted: collect volatile data with AWK.

AWK Primer

If you are already an AWK guru, this section is not for you. Head on down to the next one!

Below is a simple AWK script that explains some of the basic features of the tool. The script explains itself pretty well. Above is the output when I run it against my apache2 logs. AWK can search like grep, cut like cut, and even can run system commands!

Collecting Volatile Data

The ability to run system commands allows us to capture snapshots when we find certain text (such as a specific error message). In the forensics case I mentioned at the beginning, we were looking for outbound connections, the sources, and and the specific running processes so that we could (theoretically) record the malicious actions in real time. Several useful Linux-based tools that fulfilled our needs were netstat (to see current network connections), ps (to view the processes), and lsof to view open files and the processes that were connected to them. We used the network connections to correlate processes that had open files and then tried to determine what was causing the strange logs.

The goal, of course, was to identify sufficient evidence of a breach. In this case, we monitored the error_log using tail and search for our error. Every time it occurs, it appends the new volatile data to the specified files.

Here’s an example test run.

The files appear after we echo “Malicious” to our fake error log. I ran netstat with the flags -anp to give us all sockets, what program is running, and the numeric addresses. Ps is run with the aux flag so that we’re able to see all user processes, list processes without controlling ttys, and user oriented format.

Since we were working on a potentially compromised system, our tools might not be trusted. BusyBox turned out to be a great option, as I didn’t have any prepared safe binaries. BusyBox was created for embedded Linux systems so it is small and had most of the utilities that we need.

What else can we do with AWK? How about monitoring a file for users that access it? Or alerting when a specific user logs in? It’s definitely a useful tool. I recommend exploring it further. Keep hacking!