War Room

Shells from above

RSM logo

Home > Defense > Netscaler Still in the Wild

Netscaler Still in the Wild

By

It has been two months since Cirtix released details about CVE-2019-19781, a vulnerability found in their NetScaler product. In that time, we here at RSM have been working with several of our clients to help mitigate this vulnerability and remediate the effects of any successful compromises on their systems. Unfortunately, it appears that many more networks are affected by this vulnerability and we feel that it is important to continue to spread the word about this vulnerability, the indicators of compromise, as well as the mitigation and remediation steps necessary to protect vulnerable networks. The following is a summary of the current information available as it pertains to CVE-2019-19781.

The Vulnerability

CVE-2019-19781 is a directory traversal vulnerability which can lead to remote code execution. It is specifically the result of improper sanitization of pathnames referenced in HTTP requests to NetScaler devices. This improper sanitization practice allowed for attackers to perform a relative (“../”) directory traversal attack to access other directories outside of the web root directory. This vulnerability also allows attackers to refer to the absolute pathname of the target system that they are attempting to compromise. Several systems are affected by this vulnerability and should be immediately patched, including; ADC and Gateway versions 10.5, 11.1, 12.0, 12.1, and 13.0 as well as SD-WAN versions 10.2.6b, and 11.0.3b.

High Level Attack Chain

Most of the attackers follow a similar attack chain while exploiting NetScaler devices that are vulnerable to CVE-2019-19781. They begin by identifying a vulnerable system, typically through a GET request which has the characteristics of a directory traversal attack, “../”. to the vulnerable device’s template directory using the newbm.pl script, which uses the filewrite function. Finally, the attacker to the uploaded file using the directory traversal vulnerability which parses the script through the template engine. For a deeper dive into the technical aspects of this exploitation chain, see MDSec’s article at in the Additional Resources section.

Detection

Regardless of the method that the attacker uses during exploitation and the subsequent compromise of the network that the NetScaler device is a part of, there are several artifacts that point to the active compromise of a network using this vulnerability. First, logs are invaluable when identifying affected systems and attack paths. The bash.log file contains the commands that were run through the Bash interpreter, even if the HISTFILE environment variable is unset, which is a common method that attackers use to hide their presence on the network. The sh.log file similarly contains the commands used through the Bourne shell. The Notice.log file contains all notice severity level alerts for the system, commonly flagging bash commands. This log can act as a second point of failure if the bash.log or the sh.log files become compromised. The httpaccess.log and httperror.log files contain the successful and unsuccessful connections to the Citrix server. These HTTP logs can specifically point to the attacker’s IP address, although it should be noted that the attacker may be hiding their IP address by using a proxy. Additionally, the presence of a POST request followed by a GET request to a is a strong indicator of compromise. This pattern persists until the attacker elevates their privileges, giving a path to follow their actions in the network. A helpful tool to follow the attacker’s actions in the network is in the /proc/[PID] file, which shows the information about a given process, defined by their process ID (PID). The last easily accessible indicator of compromise is cron jobs. Attackers have been observed utilizing cron jobs to install backdoors to the devices that they have compromised to maintain persistence. Defenders can test for this method of persistence using the command “crontab -1 –u nobody”. This command searches for cron jobs that the user “nobody” initiated, since there shouldn’t be a “nobody” user initiating cron jobs, this is also a strong indicator of compromise. Alerts for any actions taken by the user “nobody” can be set up using a SIEM and the sh.log and bash.log files.

Mitigation and Remediation

**All patches, tools, rules, and policies referenced in this section can be found at the bottom of this post**

Fortunately, Citrix was very responsive when it came to pushing out patches for affected systems. They released patches for the NetScaler ADC, Gateway and SD-Wan devices within a month of the vulnerability disclosure. These patches should serve as the first line of defense for networks that are vulnerable to CVE-2019-19781. Citrix also released a new responder policy which blocks certain malicious requests. Citrix is not the only company taking an interest in detecting and mitigating this vulnerability as Cisco Sourcefire’s Snort tool can be used to detect and reject malicious traffic associated with the NetScaler vulnerability. Snort rules are continually being written to aid in the detection of malicious actors attempting to leverage this and other vulnerabilities. Two of the simplest yet effective rules looks for and blocks attackers attempting to interact with configuration files and Perl scripts on NetScaler devices. FireEye has also released a tool in collaboration with Citrix which can identify indicators of compromise from this attack in the network. This tool identifies evidence of failed vulnerability scans, successful vulnerability scans and indicators of successful compromise. If the scanning tool , the investigator should assume that further compromise has been performed in a manner that the tool did not pick up on and should continue their investigation manually.

It should be noted that even if these tools do not find indicators of compromise of successful scanning, attack patterns change rapidly and these tools may not identify all indicators of compromise, manual investigation is still necessary. One of the common locations for backdoors to be installed and other malicious artifacts to be found are “/netscaler/portal/scripts”, and “/netscaler/portal/templates”. Additionally, investigators should be aware that attackers are appending various files to dummy files, typically in the template directory, in order to exfiltrate sensitive data.

Given these indicators of compromise, there are several options when it comes to identifying if your network has been breached and is being actively compromised. As stated previously, one of the most common indicators of compromise is a POST request, typically “POST/vpn../vpns/portal/scripts/newbm.pl HTTP 1.1”, followed by a GET request to an XML file in the same directory. Monitoring for changes within the /vpns/portal/scripts and /vpns/portal/templates directories gives defenders another method for determining if an attacker is beginning to attack their network using CVE-2019-19781.

As always, we are more than happy to help test for CVE-2019-19781 on your network, and aid in mitigation and remediation efforts.

Patches, Tools, Rules, Policies, and Resources

 

ACD / Gateway

Vulnerable Version Updated Version Patch Links
10.5 10.5.70.12 https://www.citrix.com/downloads/citrix-adc/
11.1 11.1.63.15 https://www.citrix.com/downloads/citrix-gateway/
12.0 12.0.63.13
12.1 12.1.55.18
13.0 13.0.47.24

 

SD-WAN

Vulnerable Version Updated Version Patch Links
10.2.6b 11.1.51.615 https://www.citrix.com/downloads/citrix-sd-wan/
11.0.3b 11.1.51.615

 

Snort Rules

Rule Description Rule Syntax
Identify .conf responses alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:”Potential CVE-2019-19781 vulnerable .CONF response”; flow:established,to_client; content:”HTTP/1.”; depth:7; content:”200 OK”; distance:1; content:”|0d0a|Server: Apache”; distance:0; content:”al]|0d0a|”; distance:0; content:”encrypt passwords”; distance:0; content:”name resolve order”; reference:cve,2019-19781; reference:url,https://www.fireeye.com/blog/products-and-services/2020/01/rough-patch-promise-it-will-be-200-ok.html; sid:201919781; rev:1;)
Identify .pl responses alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:”Potential CVE-2019-19781 vulnerable .PL response”; flow:established,to_client; content:”HTTP/1.”; depth:7; content:”200 OK”; distance:1; content:”|0d0a|Server: Apache”; distance:0; content:”|0d0a|Connection: Keep-Alive”;
content:”|0d0a0d0a3c48544d4c3e0a3c424f44593e0a3c53
4352495054206c616e67756167653d6 a617661736372697
07420747970653d746578742f6a6176617363726970743e
0a2f2f706172656e74 2e77696e646f772e6e735f72656c
6f616428293b0a77696e646f772e636c6f736528293b0a3
c2f534 3524950543e0a3c2f424f44593e0a3c2f48544d4
c3e0a|”; reference:cve,2019-19781; reference:url,https://www.fireeye.com/blog/products-and-services/2020/01/rough-patch-promise-it-will-be-200-ok.html; sid:201919781; rev:1;)

 

Citrix Responder Policy

Policy Description Policy Syntax
Respond with 403 error if /vpns/ is found in the request add responder policy ctx267027

“HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\”/vpns/\”) &&

(!CLIENT.SSLVPN.IS_SSLVPN || HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\”/../\”))”

respondwith403

 

Tools

Tool Name Tool Description Link to Tool
Citrixmash Scanner Uses HEAD requests to determine device vulnerability https://github.com/x1sec/citrixmash_scanner/
FireEye IoC Scanner Leverages knowledge gained from incident response to identify indicators of compromise on a network or identify the presence of CVE-2019-19781 https://github.com/fireeye/ioc-scanner-CVE-2019-19781/

 

Additional Resources