Wallarm — Network Security — Testing ModSecurity for false positives by books texts
In Network Security

Testing ModSecurity for false positives by books texts

3 Mins Read

The main things that prevent enabling security solutions like WAF/RASP/IDS/IPS in a blocking mode are false positives. Probably the second one is their inline performance and additional latency, but still.

As a cloud-native WAF vendor,  we at Wallarm are actively checking our products for false positives to continuously deliver better detection quality for our customers. 

One of the ways to address false positives is to detect it early before the real customer will be blocked. To check obvious false positives for ModSecurity, an open-source WAF based on regular expression and libinjection library, we decided to download and split by lines 899 books from Gutenberg library. 

To download it, we can run a simple bash one-liner: 

$ for i in `seq 100 999`; do wget https://www.gutenberg.org/files/$i/$i-0.txt; done

Then we need some script to take lines from books and send against the WAF, check if it’s blocked and collect results. This simple PHP script can do this:

$ cat test.php 
<?php
$file = $argv[1];
$fb = fopen("false-positives.db", "a");
foreach(file($file) as $line) {
  file_get_contents('http://localhost:8014/?test='.urlencode($line), false, stream_context_create(['http' => ['ignore_errors' => true]]));
  preg_match('{HTTP\/\S*\s(\d{3})}', $http_response_header[0], $match);
  $status = $match[1];
  if($status=="403"){
    echo "[!] False positive found! $line \n";
    fwrite($fb, $line);
  }
}

Running the docker container with ModSecurity is the next step. You can see the lowest level of paranoia parameter here since no one is using 2 and more because of false positives:

$ docker run -p 8014:80 -ti -e PARANOIA=1 --rm owasp/modsecurity-crs

Everything is ready to launch it and save the results:

$ find . -name "*.txt" -exec php test.php {} \;

Prior to the analysis of the results, it’s better to sort it and keep only unique strings:

$ sort -u false-positives.db | uniq | wc -l
9846

So, we got 9846 unique lines and 10680 non-unique of 3629525 lines in total from book texts that trigger false positives. It means, that 0.29% of all the strings trigger false positives. That’s a really big number if we will count, that these data are completely normal books’ texts.

Let’s look inside to find some funny string that caused false positives. Here are they, among others:

head of horses.
repeat it.
would go away, but when they asked me if he was not there, of course
time they sat down to a meal. Horrible! Horrible!

That’s it. You can download an entire list of false positives we found here: 

github.com/wallarm/gotestwaf

Some of the tests are already integrated into an open-source tool GoTestWAF we actively contributing last month. This tool now tests not only false negatives but false positives also. You can find how to run it in this article: https://lab.wallarm.com/test-your-waf-before-hackers/

If you are looking for an upgrade your ModSecurity to enterprise-grade WAF with API protection capabilities, feel free to schedule a demo with Brandon:


Related Posts