If you have been paying attention to the recent information security news, then you know web application technologies, and WAFs more specifically, have been a culprit in several of the incidents. WAF-based attacks are frightening because WAF tech is supposed to protect. As APIs, web applications, and microservices become more central to our lives, how can we trust WAFs to protect us against Layer-7 cyberattacks? Clearly, we need WAFs. Just like a good incident response follow up, this article aims to explore recent breaches to shed light on the dangers of various WAFs, what comes next, and where our security tools are heading in the coming decade.
Recent WAF-based breaches with CapitalOne, Imperva, and Cloudflare offer essential lessons we can learn from on how WAF technology is failing us and what can we do to improve our security health.
What is a WAF
really?
One of the top security tools used by companies, a web application firewall (WAF) helps protect web applications by filtering and monitoring issue TB traffic between a web application and the Internet. It typically protects web apps from attacks such as cross-site forgery, cross-site scripting, XXS, file inclusion, SQL injections, and many others. WAF is a protocol layer-7 in the OSI model. But, it's not designed to defend against all type of cyberattacks.
A
strong WAF compliments other tools in a security toolbox, but should not be mistaken
for a complete security solution. It eliminates many cyberattacks—those it is designed
to handle. By deploying the WAF in front of the application, the WAF acts as a
shield placed between the web application and the Internet. A proxy server
protects the client machine's identity by using an intermediary WAF. Meanwhile,
the WAF acts as a type of reverse proxy that protects the server from exposure
by having clients pass through the WAF before reaching the server.
Modern cybersecurity attacks and WAFs
Internet-based
cyberattacks have become increasingly sophisticated over the last twenty years.
Just like any technology, WAFs have evolved.
Original WAFs could only protect us from what we knew was coming. The first generation of web application firewalls used a signature-based approach. These were oriented to protect from things like server-side RCE, path traversal, SQL injection. They essentially ran as a signature-based and blacklist-based technology.

The
next evolution of WAFs remains where many enterprises still are—where the technology
stack, including Ajax, and explosive growth has enabled this next generation of
WAFs. At the same time, the number of critical web applications has exploded,
driving the deployment of what we’ll call WAF 2.0.
With
WAF 2.0, dynamic profiling and a sort of optimized signature list were introduced.
However, the processes have still remained time consuming because you need
human administration and oversight. This has meant having the resources dedicated
to the WAF or risking low implementation or high misuse.
As
development cycles have accelerated and applications have exponentially increased,
the need for WAFs to function without heavy manual tuning and admin is
essential to providing robust security. This leads us to what we can think of
as WAF 3.0—the next necessary evolution of the WAF.
Hackers
have evolved to match security technology. They are increasingly sophisticated,
opportunistic, and very good with evasion. Hackers quickly figured out that
people are deploying WAFs, so they started to create evasion techniques to
evade signature-based analysis. Hackers shifted their attention to zero days.
Fully automated security methods have traditionally come up short due to the absence of clean traffic as a source material for machine learning. So, what WAF 3.0 has to focus on is the business logic that is used with mixed traffic. This eliminates the problem of requiring clean traffic for machine learning and also protects against zero days by preventing bypass.
Machine learning plays an important part in the rapidly changing cybersecurity ecosystem. The more you use it, the better it gets because it evolves and learns from its own mistakes at nearly the speed of light. It allows us to mature into a near zero false positive rate. Machine learning and AI are the only ways to really stay afloat in the growing swell of big data.
Cloud-based problems with traditional WAFs
Traditional
WAFs have been proven to be inadequate, specifically for cloud native-technology.
In the common case where applications move to the cloud or become containerized
data, application security and API security become even more important than any
other areas of security.
One of the biggest challenges heard from enterprise CISOs is lies in the human factor. Traditional WAFs use up almost about forty-five hours per week just to process WAF fillers. Sometimes this time suck deepens when additional hours are required to write new rules. When application security teams have write these rules, it a tedious job. No one wants to do it.
Thankfully, new technology and research are cleaning up these problems.
Learning from WAF-based attacks
The Imperva breach: Who watches the watchers?
Detected on August 20th, the Imperva breach revealed that any Imperva customer using their cloud WAF product back to September 15th of 2017 were affected.
Alissa Knight, Sr. Cybersecurity Analyst, Aite Group
Imperva
has a cloud WAF product that offers a web application firewall as a service. Being
a customer of that service, you would upload your SSL certificates to Imperva
and you would forward your web traffic through Imperva’s cloud WAF product. In order
for that to happen, the SSL termination, you need to upload your actual
certificates to them and also your API keys.
In the breach, attackers hacked Imperva and gained access to the encapsulated database where the customer’s extremely sensitive information was being stored. There were breached e-mail addresses, hash and salt passwords, custom API keys, and SSL certificates that were all breached. A cybersecurity offering security as a service lost customer trust in one foul breach.
When
you're trusting a cybersecurity company to hold the keys to your kingdom, safeguarding
your sensitive data, make sure they're practicing what they preach. Unfortunately,
a lot of cybersecurity companies don't do that. Imperva didn't disclose many
details around the breach, like how it happened. So, we can only imagine how
many customers were affected who were trusting Imperva to provide their web
service.
The
very technologies we trust can scarily become a culprit or source of a data
breach. But, customers are not helpless. They still need data protection.
Fortunately,
it's really easy for customers to change providers and even change those keys. It
was very difficult to unseat a managed security service provider (MSSP) fifteen
years ago. They had their equipment on premises, making it costly to replace. Now,
given virtualization and microservices within the cloud, it's very easy to
switch MSSPs if you have a WAF as a service.
The most important lesson we can take from the Imperva breach is not to feel hostage to a MSSP. Are they tasting their own food or serving up your data to potential hackers, treating you as a poison tester? Before trusting their security as a service, ask security vendors tough questions. Ask if they are taking the right measures, like doing their own penetration testing. Are they employing either static and dynamic code analysis? Do you have any certifications are you doing SOC to Type 2 audits? Have a security expert do a thorough interrogation of a potential MSSP. Just because they are a cybersecurity company, doesn't necessarily mean that they're doing what they should be doing when it comes to the security of your data.
The most important lesson we can take from the Imperva breach is not to feel hostage to a MSSP. Are they tasting their own food or serving up your data to potential hackers, treating you as a poison tester?
Tweet
The
danger is that a lot of the enterprises want to trust an MSSP and assume they
are applying their own security because “security” is right there in the title.
They are assuming a vigilant security expert when they engage with any third
party doing the security tooling.
Cloudflare’s 27 Minutes
Approximately
67% of all internet traffic goes through Cloudflare. On July 2nd of 2019,
Cloudflare experienced an outage for about twenty-seven minutes.
If you’re doing regular expressions—for whatever that means to you, like a filter, app, checking emails, or a user’s emails—you will probably have to content with a regular expression denial of service attack. You can put sophisticated regx somewhere and run this regx against a really huge amount of data or long string.
First,
we need to explain how regular expressions work. Whenever you build a regular
expression engine, that engine should parse this regular expression and apply
it to some kind of string. Sometimes, it's impossible to make sure a string
matches that regular expression’s expression without running a reading
operation from time to time, starting at the early stages. It means that if you
put a very sophisticated filter based on regular expression somewhere and then put
some data there, it probably going be required by this engine to run this
filter against your string multiple times, not only once.
Sometimes it's possible to run the read operation only one time, but in a lot of cases it is not. The quantity of reruns can be quite huge. If that occurs, we can identify that this is a regular expression denial of service (DoS) attack wherein your regular expression engine will use all the CPU and potentially a lot of memory. That's exactly what happened with the Cloudflare.
Cloudflare blamed bad software deployment for the outage. They actually put on the production some sort of signature base on regular expression that actually was able to match pretty much every single string but with a lot of reruns on that string. That regular expression was broken by some kind of mistake because they never tested for these issues. So they used all the CPUs they have and the load balancers to just serve that regular expression. As a result, they were unable to even log in, even by a Secure Shell (SSH), to fix the problem. This was because using SSH on the same machine would require you to use the same CPU, which in this case was totally used up by the WAF. That's what we need to understand about the technical reasons underlying the Cloudflare attack.
When
we want to run any kind of service, as businesses, we need to learn how to
manage that service. The Cloudflare outage could have been avoided if the CPU
had been organized differently.
Ivan Novikov, CEO, Wallarm
This sort of problem with Regular Expression Denial of Service (ReDoS) attacks is not unique to Cloudflare. For example, the same thing occurs for Apache ModSecurity or other WAFs. Frighteningly, it happens all the time and this incident is just another of the same story.
The
takeaway from the Cloudflare incident is that we must understand that regex, as
an engine, is not as good protection for an IP or IDS. It is simply not as good
as more modern solutions for large scale and highly skilled productions.
Unfortunately, these regex engines are too often relied on by SOC analysts. Regex is commonly the only thing used to pattern match whatever attack pattern is coming and then potentially deploy that. That is really tantamount to creating like a single point of failure. So, we ought to ask where we go from here. Because clearly regex is not really working out.
Erratic hacker: The threat inside
Companies tend to hoard the details of breaches, for obvious reasons. But, when a hacker goes online and brags about their attack, it’s not long before it’s front page news. That’s what happened when a former Amazon employee started her day on July 29th of 2019, boasting she had hacked 106M CapitalOne customers’ data.
Paige
Thompson, a hacker who goes by handle “erratic”, was an Amazon employee who
gained access to CapitalOne’s database using WAF rules and roles. What is
incredible is how easily she was able to elevate privileges and exfiltrate data
to her local server. The mass media is attributing the attack to Server Side
Request Forgery (SSRF) as one of the reasons why she was able to exploit. What
we have learned, though, is that “erratic” potentially gained access though poor
firewall configuration. In other words, the WAF was the potential culprit that
allowed Paige Thompson access to the very customer databases the WAF was
supposed to protect. Configuration never went so sideways.
Erratic
seems to have accessed an obvious credential via a WAF configuration role. She
then elevated some privileges, copied some tokens, and essentially was able to
gain access to over 703 buckets.
Sometimes
we falsely assume that having an established cloud where all your data is
requires so much responsibility on the part of the cloud provider that they
must be secure. In truth, even a cloud service provider (CSP) or an incredibly
large enterprise is susceptible to these sorts of attacks and threats. We call
it insider threat. However, even considering the insider threat profile, there
are other things that went wrong associated with web application firewall rule.
Questions
remain. How did the WAF rules allow Paige Thompson to elevate that many
privileges and why was the exploration so easy?
The lessons learned from breaches
A quick recap on the main takeaways from the preceding three
attacks:
- WATCH THE WATCHERS. Thoroughly vet and ask hard questions of MSSPs
- BE CAREFUL WITH CONFIGURATION. WAFs are designed to protect you, but mishandled, they can backfire.
- REJECT REGEX. Putting regex in a critical security tooling mechanism is really not a good idea.
Lessons learned:
machine learning to the rescue
Now,
we have technologies that employ machine learning and artificial intelligence
to do things like learn from your existing traffic, set baselines, and detect
abnormalities. We have more targeted protection, like API-level monitors. We have
cloud-native solutions. And, we have automation to help your testing and
protection run at the same rate as your DevOps lifecycles.
Machine learning is especially promising. We are sort of working towards this machine learning-based model that could help us get away from regex functionality.
Real
protection will rely on applying a behavior-based model. Applying statistical
approaches, including machine learning and other tools, will be increasingly necessary
for protecting users against attacks.
Machine
learning is not a magic formula. It has to learn the business logic of your app
to understand things like how many APIs do we have, what kind of data are we
dealing with, and what kind of parameters should have a mandatory ID and which
are optional. You can use machine learning to determine a lot of these rules and
complete tasks.
New
tech tackles building around a user behavior base, to help us to identify
normal behavior. We need to identify users and abnormal behaviors in addition
to anomalies.
Cybersecurity as a
service.
Moving
into this next decade, we are seeing more and more things as a service. We're
seeing managed detection and response now where outside service providers like
MSSPs are handling Internet responses and response actions. Organizations straining
to keep up, especially given the global talent shortage in cybersecurity. They are
increasingly relying on outside companies to provide their security as a
service.
Whether
it's Imperva’s cloud WAF products or any other service, it is imperative that in
relying on external service providers we do forfeit our own role in security. Even
with a cybersecurity company as a service, there is zero guarantee that they're
more secure than you. Watch the watchers. Ask your vendors what steps are they
taking to secure their data. Ask for copies of penetration test reports that
can be protected under an NDA. You can request their most recent SOC report.
These are all things that you need to be asking.
We've
had numerous cybersecurity companies breached. Hackers are opportunists, not
terribly unlike people who walk through a parking lot checking doors to see if
they’re unlocked. Cyber-attackers don't differentiate where the windows and doors
are open. They try to get a foothold and elevate permissions and do all this
fun stuff that they love to do to cause havoc. Even with strong protections,
there could be an insider threat.
When
talking about security with a provider, go beyond what kind of a compliance
documentation they have and into deep detail into things like configuration.
Third parties are just all over. We just can't depend on these traditional technologies. We need to evolve. We need to let go of the legacy solutions that require manual tuning. We need to embrace newer technology, move to advanced WAFs that use machine learning-based models that allow them to process and learn from tons of ton of data.
Hackers are opportunists, not terribly unlike people who walk through a parking lot checking doors to see if they’re unlocked.
Tweet
No one can afford to spend 40 to 80 plus hours a month on manual tuning and heavy administration. It’s not even as effective as it is costly. Evolving technology will be about saving time and using human resources more effectively. Artificial intelligence is an assistant that sops up all the tedious stuff.
Advanced firewalls will allow you to:
- scale horizontally;
- apply policies that are centrally
managed; - REST API for full data access and
integration; - use a central dashboard; and
- allow you to protect against cyberattacks
and layer seven cyberattacks.
With this rise of integration third party applications, this is exactly where we need to go. We need to move towards these advanced solutions, like cloud-native WAFs. The problem is only going to continue, integrating more APIs and microservices at deeper layers, diversifying technology sets, and our business ecosystems evolving to be even richer, more complex, and teeming with huge volumes of vibrantly moving data. Security will continue to be in the hands of outside parties, which means your knowledge base needs to be stronger. Your security needs to be your responsibility, even when it's not in your house. Think of your data like a child you really love who needs to go to science camp. Ask all the right questions and be sure that your data is safe.
Your security needs to be your responsibility, even when it's not in your house. Think of your data like a child you really love who needs to go to science camp. Ask all the right questions and be sure that your data is safe.
Tweet



