Identifying Credit Card Skimmers Using Linux’s “strace” Command

RSM US LLP’s (RSM’s) digital forensics and incident response (DFIR) team recently worked a case where a client was informed that their website’s payment platform was suffering from an ongoing attack. Based on customer complaints and common point-of-purchase (CPP) notifications from issuing banks, the client feared that credit card information was being scraped from purchases made on their platform.

After initial analysis of the file system, including CPP notifications and logs, the server and payment application had identified multiple instances of previously exploited vulnerabilities, unauthorized access and web shells. However, full review of this evidence did not provide us with any obvious patterns or focus for how credit card data was being stolen. While we are not going to get into the details of the web shells found and their capabilities within this blog, our analysis and monitoring of these files did not show any evidence of data exfiltration or automated staging or capturing of data. This blog will describe how we identified these credit card skimmers using Linux’s “strace” command.

Initial Observations

Our first objective was to try to identify some pattern or trigger for when credit cards would be impacted. We observed that those affected could either be guests or users with valid accounts within the payment application, purchasing any item at any time and frequency, while using an assortment of different card types (credit/debit), issuers and payment gateways. The suspected scrapper activity also appeared to be active in bunches, seemingly running and then appearing to cease for a while before resuming. From the customer perspective, everything seemed normal during the shopping and checkout processes. We did perform customer-side analysis of the shipping and payment processes and debugging through the browser to inspect elements, source files, parameters and network activity. The only anomaly we identified was the amount of time for credit card authorization after a purchase was made; in our observations, sometimes up to a minute for approval.

Methodology

With available log data and client-side analysis providing no conclusive results, we obtained permissions to perform system-level analysis, or dynamic inspection of the running web server processes, using provided root-access to the client’s web server. To start, we used the Linux strace command on the live web server to intercept and record all system calls called by a process and the resulting signals received by the process, allowing for the analysis of all sorts of valuable information, such as files opened, data written, network connections opened and associated parameters. For the purpose of this case, it is very important to capture the entire context of the payment application, which would involve running strace to capture all system calls for the HTTPD process (Apache web requests and responses) and the Hypertext Preprocessor (PHP) processes (which will handle actual code processing, content and execution). Together, by tracing both Apache and PHP, we can capture the entire life cycle of the payment actions, as well as everything that occurs behind the scenes and at the system level.

strace Command to Capture HTTPD Processes

ps ax | grep -i httpd | grep -v grep | awk ‘{ print "strace -f -s500 -o strace."$1".log -p "$1"&" }’ | sh

strace Command to Capture PHP Processes

ps ax | grep -i php | grep -v grep | awk ‘{ print "strace -f -s500 -o strace."$1".log -p "$1"&" }’ | sh

We gathered a listing of all HTTPD and PHP processes and piped that as input for the strace command. This command will hook onto each parent process ID (PPID) listed from the “ps ax” output, follow any forks or child processes that may spawn during our trace, set a string size value (to limit trace output), and output to a log file labeled by process. It should be noted than on a typical system running a web server, an analyst can expect 20–40 unique processes running that support the web server, so in effect it is common to have 20–40 strace output files relating to the unique Apache and PHP processes. It should also be noted that, because an analyst will not know the exact PPID or child process ID (PID) that will handle a specific web action or checkout process, all web processes will have to be included and hooked.

That’s a lot of information an analyst will have to investigate, so our next action was to scope down the trace results and focus our efforts. Because the amount of time to authorize a payment was anomalous and excessive, we had a good assumption that either data was being intercepted, parsed and then sent to the payment gateway for authorization, or the entire communication was was being relayed first before being allowed to continue legitimately. We next implemented a watch loop to monitor both inbound and outbound network connections based on these assumptions, specifically observing connections during the checkout process. We simply initiated a trusted “netstat” command with the “-c” option to continually refresh.

Using the output of this command, we identified a suspicious outbound connection over Transmission Control Protocol (TCP) Port 80 to 185.180.198.XX, which appeared to be initated during every checkout process.

We can now leverage our strace output to figure out what was initiating this connection and where in the checkout process this was called. We searched for this IP across all strace output files and identified Domain Name Service (DNS) record requests for “trueXXX.com” within several of the PHP output files. We observed from the output below that we captured network-related system calls, such as “socket()”, “connect()”, “sendmsg()”, and “recvfrom()”, collectively representing the DNS request to resolve trueXXX.com, and processing the return record of “185.180.198.XX.”

A breakdown of the relevant system calls seen in the above screenshot is provided below:

• sendmmsg() – allows the caller to transmit multiple messages on a socket using a single system call (In this case, the contents of this call contain the URL used as the basis for the DNS query that resolves to the 185.180.198.XX IP address seen within the netstat command output.)
• recvfrom() – receives a message from a socket and captures the address from which the data was sent (In this case, it was used to receive the DNS query for trueXXX.com to resolve to 185.180.198.XX.)
• socket() – creates an endpoint for communication and returns a file descriptor that refers to that endpoint
• connect() – connects the socket to the address specified in the file descriptor

We needed to work our way backwards to figure out why the resulting URL (trueXXX.com) was called. To find this, we concentrated on locating “open()” calls up the stack in close proximity to our DNS requests. We were working under the assumption that a specific PHP file was opened and contained an “include” statement, or similar, to initiate the DNS request. The open() system call opens a file specified by a pathname. The return value of open() is a file descriptor, which is a small, nonnegative integer that is an index to an entry in the process’ table of open file descriptors. We attempted to see if this file descriptor was used in subsequent system calls, such as “lsat().”

We identified a potential candidate file that was opened immediately before the DNS request called “secure-mysql.php” located within the “/public-html/include” directory on the web server.

The contents of the /public-html/include/secure-mysql.php file included a method to send data using the “curl()” function and a Base64 encoded parameter. A snippet of this file is shown in the screenshot below.

A breakdown of this line of code seen within the /public-html/include/secure-mysql.php file is outlined below:

• curl_init() – initializes a new cURL session that uses URL syntax to transfer data to and from servers
• curl_setopt() – sets an option on a given cURL session handle (In this case, it is setting up the option containing the Base64-encoded parameter.)
• curlopt_url() – provides the URL to use in the cURL request (In this case, the Base64-encoded parameter most likely decodes to a URL.)
• http_build_query() – generates a URL-encoded query string from the associative array provided (In this case, this is used to generate the HTTP query based on the contents of the “c” array. This array was very interesting to understand.)

We decoded this Base64 parameter to determine what URL the cURL command was using and saw the same URL (trueXXX.com) from the DNS query resolving to 185.180.198.XX, the IP address from the netstat command we ran earlier. This confirmed to us that this PHP file, /public-html/include/secure-mysql.php, was responsible for initiating this outbound connection over TCP Port 80, which was further validated by the presence of the “http_build_query” function.

The decoded string also showed us that the “/livechat/stat.gif” file accepted a URL variable after the “&” character, which would correspond to the “c” variable identified in the http_build_query() function.

Revisiting our strace output, we searched for entries related to “stat.gif” and came across the “c” variable equal to another Base64 encoded string. Therefore, our function would now look like this: (http_build_query(array(‘c’=>base64_encode))).

The “sendto()” function in the screenshot above was sending this data from the http_build_query function in the /public-html/include/secure-mysql.php file as a URL variable, attempting to mask itself as a standard HTTP GET request to the remote “http[:]//trueXXX.com/livestat/stat.gif” file. Therefore, while the destination server does not need to respond or even have to contain a stat.gif file, it is still obtaining data sent over HTTP within that encoded URL variable.

When we decoded the URL variable, we noticed the purchase details from our checkout processes, including all relevant user and credit card information. In fact, for every purchase made within the payment application, a network session was established that sent the contents of the payment method and cards being used.

We next created several YARA rules to search the file system for indicators of compromise similar to the cURL functionality and broad enough to identify various encoding schemes and obfuscation techniques. Additionally, we specifically created rules for the URL and IP address identified. Below is an example of one of these rules.

The YARA output identified a file on the system directly referencing the malicious secure-mysql.php file; /public-html/admin/config.php. Within this file is a line of code using the include_once() function targeting the malicious secure-mysql.php file. This function is used by PHP to include and evaluate a specified file during the execution of a script. In this case, whenever “config.php” was loaded and executed, which was a normal and legitimate file loaded within the checkout process, it was told to include and execute code within the malicious secure-mysql.php file. This line of code is highlighted in the screenshot below.

Additionally, we verified that the config.php file is the base configuration file for the client’s payment platform on their website, directly called upon using the require_once() function which is also used by PHP to load a target file for code execution. Based on this functionality, we had proven when a payment transaction began on the client’s website the config.php file, including the malicious code within secure-mysql.php, was loaded and executed.

Conclusions

Based on these findings, an attacker had successfully performed the following:

1. Gained access to the client’s environment.
2. Manipulated the client’s payment platform (config.php) to load malicious code found in a separate file on the system (secure-mysql.php).
3. Captured payment information using this malicious code and sent the results outbound to a remote address as an encoded URL parameter (stat.gif) over TCP Port 80.

Lessons Learned

Based on our findings, it became clear that this basic, yet stealthy, attack would have left little forensic artifacts within the log data we had initially analyzed. The strace command run on the live web server provided the greatest level of insight into what was going on within the client’s environment, allowing us to analyze system calls directly to narrow down how credit card information was being scraped from the client’s website, then work backwards to remove the threat.

While effective, this attack could have been identified and potentially stopped much earlier with a few simple security measures:

1. File Integrity Monitoring – Having a system in place to routinely check the integrity of configuration files within an environment can help mitigate the risk of a malicious alteration, such the inclusion of the secure-mysql.php file and its malicious code within one of the payment platform’s core configuration files, config.php.
2. Restrict Outbound Connections from Internal Systems – Preventing systems from establishing and persistently reaching out to unauthorized remote addresses, such as 185.180.198.XX over TCP Port 80, would have prevented the successful exfiltration of data from the environment.
3. Active Network Monitoring – Ensuring that you have a solid methodology for capturing and monitoring network activity to and from an environment is key. Active monitoring can identify anomalous connections, such as an active connection to an unauthorized external IP address, and allow for a quick response to address and potentially terminate this connection.