Updated 6 July 2022: Wallarm launched – GoTestWAF: Free Online WAF testerhttps://www.wallarm.com/gotestwaf/overview

Updated 18 January 2023: added gRPC protocol detection and other features, more details in the article.

Since 1991, Web Application Firewall, commonly referred to as WAF, has become one of the most common application security technologies available on the market. Since the last century, WAFs have evolved by incorporating the cloud and using Machine Learning instead of RegExp.

Currently, few technologies, such as NG-WAF, RASP, WAAP, and a few others, have internal WAF capabilities, which prevent web applications and API threats.

Majority of the fintech, health tech, and e-commerce companies have had WAFs installed for years to protect their APIs, but also due to PCI DSS, SOC2, and HIPAA compliance requirements, bot mitigation, and OWASP Top-10 attack prevention needs.

How good is my WAF?

WAF-like technologies have already been in place for a while, but how good are they? Given the notorious issue of false positives that WAFs have always been known for, people often focus on evaluating false positive rates while ignoring the testing of false negative rates. In addition, it is not easy to test and check the actual level of WAF or RASP protection. 

So how can we really test how good your current WAF is? What attacks can it really stop and where can application and API attacks still hit even with a WAF in place? Where are the blind spots? There isn’t a simple tool that any developer, QA, or security engineer can run to get a PDF report on your WAF coverage.

Meet GoTestWAF

To address the issue, we are open-sourcing a project called GoTestWAF

GoTestWAF generates requests with predefined, basic payloads as well as attacks specific to different APIs (REST, SOAP, XMLRPC, gRPC). Afterwards, it sends them to the application and analyzes the responses to generate a detailed report in the console output or as a PDF.

The usage is pretty simple. It only requires that you run this tool against your WAF protected application. 

docker run -v /tmp:/tmp/report gotestwaf --url=http://the-waf-you-wanna-test/

It gives clear results and indicates which of the attacks are detected with your existing appsec solution and what ways attackers can still hit your apps:

owasp   path-traversal   6/18  (0.33)
owasp   sql-injection    4/16  (0.25)
owasp   ss-include       2/8   (0.25)
owasp   xml-injection    6/8   (0.75)
owasp   xss-scripting    4/12  (0.33)
owasp   ldap-injection   0/8   (0.00)
owasp   mail-injection   3/12  (0.25)
owasp   nosql-injection  0/18  (0.00)
owasp   shell-injection  3/8   (0.38)
owasp   sst-injection    5/20  (0.25)

It’s a simple yet powerful tool that supports not only pre-defined payloads but also API requests, such as REST, SOAP, and XMLRPC, and custom encodings like Base64, which can be extended easily.

In conclusion, the GoTestWAF is a tool to test WAFs, RASPs, and WAAP for application and API attacks, not just CGI payloads from 90th. 

How it works?

The main idea is to encode and place attack payloads in different parts of an HTTP request: its body, headers, URL parameters, etc. To simplify things, we decided to implement the following logic:

Payload -> Encoder -> Placeholder

It means that every payload sample (malicious attack sample such as an XSS string like “<script>alert(1)</script>”) will be first encoded in some way then placed into an HTTP request. There is also an option to use a plain encoder that keeps the string as-is.

To make tests readable, we introduced a YAML DSL with an equal structure (payload->encoder->placeholder), where all the fields are arrays and make a test by permutations between them. 

Take a look at this SQL-injection test readable:

payload:
  - '"union select -7431.1, name, @aaa from u_base--w-'
  - "'or 123.22=123.22"
  - "' waitfor delay '00:00:10'--"
  - "')) or pg_sleep(5)--"
encoder:
  - Base64Flat
  - Url
placeholder:
  - UrlPath
  - UrlParam
  - JsonBody
  - Header

As a result of the permutation of each 4 payloads, 2 encoders, and 4 placeholders, this test will send 4*2*4=32 requests.

Note: be careful with the YAML. If you need to send a few binary payloads, then it is best to use !!binary attribute on the payload fields.

How to run

There is an Alpine-based Docker image available. It is ready to start out-of-the-box with a single command-line right after “git clone” and “docker build”:

$ git clone https://github.com/wallarm/gotestwaf
$ docker build . -t gotestwaf
$ docker run -v /tmp:/tmp/report gotestwaf --url=http://the-waf-you-wanna-test/

The only parameter here is a directory where the PDF report will be placed. If you need the report, you are required to map it from your host machine via -v docker attribute. Otherwise, all the stats will be available in your console only.

If you need to configure some more parameters, here are the command-line arguments and config.yaml: 

Usage of /go/src/gotestwaf/gotestwaf:
  -block_regexp string
     Regular Expression to detect blocking page with the same HTTP response status code as not blocked request
  -block_statuscode int
     HTTP response status code that WAF uses while blocking requests. 403 by default (default 403)
  -check_cert
     Check SSL/TLS certificates, turned off by default
  -config string
     Config file to use. Attention, if you are using the config, all the are flags will be avoided. (default "config.yaml")
  -proxy string
     Proxy to use
  -report string
     Report filename to export results. (default "/tmp/report/waf-test-report2020-April-10.pdf")
  -testcases string
     Folder with test cases (default "./testcases/")
  -threads int
     Number of concurrent HTTP requests (default 2)
  -url string
     URL with a WAF to check (default "http://localhost")

Command-line flags have priority over configuration files. As you can see, the most important thing here is how to detect a blocking page. Currently, you can do it with a response status or RegExp that will be applied to the request body. By default, the 403 response status code will be used for this.

 You can also choose test cases between two embedded: OWASP Top-10, OWASP-API, or your own. 

The report sample

Again, the main goal of this tool is to easily generate readable reports to check the current state of protection according to OWASP guidelines. 

To see what it looks like, we can install mod_security WAF and run it on the 8080 port:

git clone https://github.com/SpiderLabs/owasp-modsecurity-crs
cd owasp-modsecurity-crs/util/docker
docker build -t modsec_crs --file Dockerfile-3.0-nginx .
docker run --rm -p 8080:80 -e PARANOIA=1 modsec_crs

Next, we can launch the GoTestWAF utility:

docker run -v /tmp:/tmp/report gotestwaf --url=http://the-waf-you-wanna-test/

The results will be printed in your console as such:

owasp   path-traversal   6/18  (0.33)
owasp   sql-injection    4/16  (0.25)
owasp   ss-include       2/8   (0.25)
owasp   xml-injection    6/8   (0.75)
owasp   xss-scripting    4/12  (0.33)
owasp   ldap-injection   0/8   (0.00)
owasp   mail-injection   3/12  (0.25)
owasp   nosql-injection  0/18  (0.00)
owasp   shell-injection  3/8   (0.38)
owasp   sst-injection    5/20  (0.25)

The PDF version will also contain bupasses with details, screenshot below:

bypasses detail

We updated the 2023 GoTestWAF tool with the following features:

  1. Sending a report by email
  2. Ability to perform testing using OpenAPI specification
  3. Ability to add debug header (request from client), specify multiple status codes (blockStatusCodes, passStatusCodes).
  4. Since there were some modifications for sessions, service can detect some (2 or 3) vendors automatically and indicate them in report.
  5. The report view has changed + now you can get 2 types of report:
    1. full report with all bypassed vectors 
    2. The short version of the report with graphs and statistics.
  6. We also have the ability to generate json report and json output, so you can embed GTW somewhere, for example in the pipeline
  7. We also improved passRegex, blockRegex – now not only the response body, but the entire HTTP-response header is analyzed.
  8. Added normal gRPC protocol detection and ability to detect gRPC on a particular port

The new report looks like this:

report screenshot

Graphs from the new 2023 report:

report screenshot app security

For the technical details about payload, encoders, and responses, you can check the CSV file that will be generated in the same folder as PDF report.

The new report has a comparison with other solutions:

report screenshot comparison

Future improvements

We invite everyone to join our mission and commit to the project, so that we can collectively improve the task at hand. The current roadmap includes support of WebSockets and .NET specific attack improvements, as well as some bots-emulations and other new ideas recommended by current users. Any pull requests and improvement suggestions are greatly appreciated.