It has been two months since Cirtix released details about CVE-2019-19781, a vulnerability found in their NetScaler product. In that time, we here at RSM have been working with several of our clients to help mitigate this vulnerability and remediate the effects of any successful compromises on their systems. Unfortunately, it appears that many more networks are affected by this vulnerability and we feel that it is important to continue to spread the word about this vulnerability, the indicators of compromise, as well as the mitigation and remediation steps necessary to protect vulnerable networks. The following is a summary of the current information available as it pertains to CVE-2019-19781.
CVE-2019-19781 is a directory traversal vulnerability which can lead to remote code execution. It is specifically the result of improper sanitization of pathnames referenced in HTTP requests to NetScaler devices. This improper sanitization practice allowed for attackers to perform a relative (“../”) directory traversal attack to access other directories outside of the web root directory. This vulnerability also allows attackers to refer to the absolute pathname of the target system that they are attempting to compromise. Several systems are affected by this vulnerability and should be immediately patched, including; ADC and Gateway versions 10.5, 11.1, 12.0, 12.1, and 13.0 as well as SD-WAN versions 10.2.6b, and 11.0.3b.
High Level Attack Chain
Most of the attackers follow a similar attack chain while exploiting NetScaler devices that are vulnerable to CVE-2019-19781. They begin by identifying a vulnerable system, typically through a GET request which has the characteristics of a directory traversal attack, “../”. to the vulnerable device’s template directory using the newbm.pl script, which uses the filewrite function. Finally, the attacker to the uploaded file using the directory traversal vulnerability which parses the script through the template engine. For a deeper dive into the technical aspects of this exploitation chain, see MDSec’s article at in the Additional Resources section.
Regardless of the method that the attacker uses during exploitation and the subsequent compromise of the network that the NetScaler device is a part of, there are several artifacts that point to the active compromise of a network using this vulnerability. First, logs are invaluable when identifying affected systems and attack paths. The bash.log file contains the commands that were run through the Bash interpreter, even if the HISTFILE environment variable is unset, which is a common method that attackers use to hide their presence on the network. The sh.log file similarly contains the commands used through the Bourne shell. The Notice.log file contains all notice severity level alerts for the system, commonly flagging bash commands. This log can act as a second point of failure if the bash.log or the sh.log files become compromised. The httpaccess.log and httperror.log files contain the successful and unsuccessful connections to the Citrix server. These HTTP logs can specifically point to the attacker’s IP address, although it should be noted that the attacker may be hiding their IP address by using a proxy. Additionally, the presence of a POST request followed by a GET request to a is a strong indicator of compromise. This pattern persists until the attacker elevates their privileges, giving a path to follow their actions in the network. A helpful tool to follow the attacker’s actions in the network is in the /proc/[PID] file, which shows the information about a given process, defined by their process ID (PID). The last easily accessible indicator of compromise is cron jobs. Attackers have been observed utilizing cron jobs to install backdoors to the devices that they have compromised to maintain persistence. Defenders can test for this method of persistence using the command “crontab -1 –u nobody”. This command searches for cron jobs that the user “nobody” initiated, since there shouldn’t be a “nobody” user initiating cron jobs, this is also a strong indicator of compromise. Alerts for any actions taken by the user “nobody” can be set up using a SIEM and the sh.log and bash.log files.
Mitigation and Remediation
**All patches, tools, rules, and policies referenced in this section can be found at the bottom of this post**
Fortunately, Citrix was very responsive when it came to pushing out patches for affected systems. They released patches for the NetScaler ADC, Gateway and SD-Wan devices within a month of the vulnerability disclosure. These patches should serve as the first line of defense for networks that are vulnerable to CVE-2019-19781. Citrix also released a new responder policy which blocks certain malicious requests. Citrix is not the only company taking an interest in detecting and mitigating this vulnerability as Cisco Sourcefire’s Snort tool can be used to detect and reject malicious traffic associated with the NetScaler vulnerability. Snort rules are continually being written to aid in the detection of malicious actors attempting to leverage this and other vulnerabilities. Two of the simplest yet effective rules looks for and blocks attackers attempting to interact with configuration files and Perl scripts on NetScaler devices. FireEye has also released a tool in collaboration with Citrix which can identify indicators of compromise from this attack in the network. This tool identifies evidence of failed vulnerability scans, successful vulnerability scans and indicators of successful compromise. If the scanning tool , the investigator should assume that further compromise has been performed in a manner that the tool did not pick up on and should continue their investigation manually.
It should be noted that even if these tools do not find indicators of compromise of successful scanning, attack patterns change rapidly and these tools may not identify all indicators of compromise, manual investigation is still necessary. One of the common locations for backdoors to be installed and other malicious artifacts to be found are “/netscaler/portal/scripts”, and “/netscaler/portal/templates”. Additionally, investigators should be aware that attackers are appending various files to dummy files, typically in the template directory, in order to exfiltrate sensitive data.
Given these indicators of compromise, there are several options when it comes to identifying if your network has been breached and is being actively compromised. As stated previously, one of the most common indicators of compromise is a POST request, typically “POST/vpn../vpns/portal/scripts/newbm.pl HTTP 1.1”, followed by a GET request to an XML file in the same directory. Monitoring for changes within the /vpns/portal/scripts and /vpns/portal/templates directories gives defenders another method for determining if an attacker is beginning to attack their network using CVE-2019-19781.
As always, we are more than happy to help test for CVE-2019-19781 on your network, and aid in mitigation and remediation efforts.
Patches, Tools, Rules, Policies, and Resources
ACD / Gateway
|Vulnerable Version||Updated Version||Patch Links|
|Vulnerable Version||Updated Version||Patch Links|
|Rule Description||Rule Syntax|
|Identify .conf responses||alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:”Potential CVE-2019-19781 vulnerable .CONF response”; flow:established,to_client; content:”HTTP/1.”; depth:7; content:”200 OK”; distance:1; content:”|0d0a|Server: Apache”; distance:0; content:”al]|0d0a|”; distance:0; content:”encrypt passwords”; distance:0; content:”name resolve order”; reference:cve,2019-19781; reference:url,https://www.fireeye.com/blog/products-and-services/2020/01/rough-patch-promise-it-will-be-200-ok.html; sid:201919781; rev:1;)|
|Identify .pl responses||alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:”Potential CVE-2019-19781 vulnerable .PL response”; flow:established,to_client; content:”HTTP/1.”; depth:7; content:”200 OK”; distance:1; content:”|0d0a|Server: Apache”; distance:0; content:”|0d0a|Connection: Keep-Alive”;
c3e0a|”; reference:cve,2019-19781; reference:url,https://www.fireeye.com/blog/products-and-services/2020/01/rough-patch-promise-it-will-be-200-ok.html; sid:201919781; rev:1;)
Citrix Responder Policy
|Policy Description||Policy Syntax|
|Respond with 403 error if /vpns/ is found in the request||add responder policy ctx267027
(!CLIENT.SSLVPN.IS_SSLVPN || HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\”/../\”))”
|Tool Name||Tool Description||Link to Tool|
|Citrixmash Scanner||Uses HEAD requests to determine device vulnerability||https://github.com/x1sec/citrixmash_scanner/|
|FireEye IoC Scanner||Leverages knowledge gained from incident response to identify indicators of compromise on a network or identify the presence of CVE-2019-19781||https://github.com/fireeye/ioc-scanner-CVE-2019-19781/|
- Citrix mitigation steps: https://support.citrix.com/article/CTX267679
- TrustedSec vulnerability overview: https://www.trustedsec.com/blog/critical-exposure-in-citrix-adc-netscaler-unauthenticated-remote-code-execution/
- TrustedSec forensic overview: https://www.trustedsec.com/blog/netscaler-remote-code-execution-forensics/
- FireEye IoCs and Mitigation: https://www.fireeye.com/blog/products-and-services/2020/01/fireeye-and-citrix-tool-scans-for-iocs-related-to-vulnerability.html
- TripWire overview: https://www.tripwire.com/state-of-security/vert/citrix-netscaler-cve-2019-19781-what-you-need-to-know/
- MDSEC technical break down of exploit chain: https://www.mdsec.co.uk/2020/01/deep-dive-to-citrix-adc-remote-code-execution-cve-2019-19781/
- Citrix Log Guide: https://support.citrix.com/article/CTX227560