Discovering Shadow APIs with Wallarm API firewall

Shadow APIs can be defined as active endpoints that you are not aware of. Some APIs are deployed but never documented. Others are services that don’t have an owner anymore. Some are even old v2 versions that have been deprecated for years, yet still exposed. Long story short: these APIs are not documented and not in the API catalog. Yet, they pose a real threat as they can be vulnerable. 
What we see as a trend is the need to automate the discovery of shadow or zombie APIs. We have a lot of happy customers of the API Discovery feature of our End-to-end API Security platform. But even more importantly, we recently added new capabilities into our free API Firewall (500M+ installed in Docker Hub) that also helps in discovering Shadow APIs.

New feature: Shadow API Discovery

As a reminder: the API firewall is a completely free tool that works like a proxy and performs OpenAPI requests and responds with validation mechanisms inside.  It’s really fast and even overperforms Nginx by 60%. 

Since the latest version, the API firewall has added a shadow API discovery feature. When enabled, the API firewall logs requests to those API endpoints that are not described in an OpenAPI specification but return 2xx or 5xx status codes (so we can assume that they exist and are exposed).

Trying it out

Let’s see it in action by using the Docker demo from the official Github repository of the API firewall. First of all, we need to clone the demo git.

$ git clone https://github.com/wallarm/api-firewall.git
$ cd api-firewall/demo/docker-compose

Then we need to change the docker-compose file to set the request validation mode to LOG_ONLY (change it from BLOCK to LOG_ONLY). This is important. The shadow API security feature only works in a LOG_ONLY mode for a clear reason. The API firewall needs to pass requests through to the backend -- otherwise, it won’t see response status codes (remember that we only log those requests that actually hit the backend and get 2xx or 5xx).

APIFW_REQUEST_VALIDATION: "LOG_ONLY"

The next step is to build and run a bunch of two containers:

$ make start
[+] Running 2/2
⠿ Container api-firewall          Started                                                                                         
⠿ Container api-firewall-backend  Started

Once everything is up and running, let’s test it out and send a few API requests. One to the root endpoint / and the second one to the unexisting endpoint /sss:

$ curl https://localhost:8080/
$ curl https://localhost:8080/sss

• The root / endpoint is playing the role of a shadow API endpoint. It is exposed and returns some content with a 200 status code in response. However, it’s not described in the demo OpenAPI specification.
• Meanwhile, the /sss endpoint doesn’t exist; the backend will return a 404 status code.

Let’s see those requests in logs:

$ docker logs api-firewall

time=2021-11-04T18:50:58Z level=error msg=#0000000100000001 : Shadow API : 172.19.0.1:44662 -> GET / : 200 (response length: 9593)

time=2021-11-04T18:51:00Z level=info msg=(404) : #0000000200000001 : GET /sss -> 172.19.0.1:44684 (28.023735ms)

That’s it! There is an alert on the Shadow API endpoint right there. So you can set up proper alerting by integrating logs with your alerting/incident responses toolchain and DevSecOps pipeline.
By the way, with the APIFW_SHADOW_API_EXCLUDE_LIST variable, you can configure which status codes to ignore. By default, it only includes 404 code and thus it doesn’t log any hits to the endpoints that responded with 404. But feel free to add 403 and any other codes.

Any feedback

Have ideas of how to make this feature even more exciting for your use case? We encourage everyone to join the API firewall community on GitHub and create issues for any additional feature you need.