Finding application software security bugs is a difficult process. Engineering teams often turn to security scanners to perform tests against their running applications in an attempt to find security bugs early. This is a great idea… with some caveats! We use security scanners during security assessments as a “force multiplier” to test for simple issues and get broad test coverage. A good security scanner, used correctly, can be a useful tool. I’d guess a good 10% of our security findings are a direct result of security scanners. That’s a meaningful amount but not enough for me to rely on a security scanner alone.

We also work with clients who have used security scanners before but come away disappointed. Most often this disappointment stems from misperceptions about what security scanners can do. Whether you use an open source tool, such as OWASP Zap, or a commercial tool such as Burp Suite, don’t be fooled by these common misperceptions.

Misperception: Scanners Find All Security Bugs

Scanners won’t find all of your application security bugs. Wish they could, but they can’t. This misperception is most commonly held by the less-technical members of a product organization. The misperception that an automated tool can find all security bugs is dangerous because it can give a false sense of protection.

I experienced this most vividly with a client who had used a commercial security scanner on their product and fixed all the bugs it reported. We performed a security assessment and delivered a pile of very serious findings. We had to walk the client past this misperception and help him to realize that just having a tool doesn’t mean you’ve found all the bugs.

Reality: Software security scanners will find some security bugs.

Misperception: Scanners Find Consequential Security Bugs

Security scanners are a product category and there are a lot of vendors marketing to application developers. Given all the marketing words flying around it might be easy to think that these tools, even if they aren’t perfect, will at least find all the consequential security bugs that are out there. SQL Injection, for example, has been discussed since 1998. Surely security scanners can find every instance of a consequential bug like this!

Experience has shown that even well understood bugs can get missed by a security scanner. Maybe the scanner failed to test the right route or parameters. There could be mitigating protections that hide the vulnerability from the scanner. I’ve even seen a bug triggered by a scanner’s probe, but the scanner fails to recognize the response as indicating a vulnerability. Scanners are great at finding the bugs that have known tests given that the test is applied in the correct location and the response is recognized as the scanner as vulnerable. As it turns out, some actual security bugs don’t fit these criteria and hence aren’t scannable.

Reality: Software security scanners will find scannable security bugs.

Misperception: Scanners Will Find All Instances of a Bug They Know How to Scan

I do like to use security scanners to help find additional instances of a bug once I’ve found it. Why stop after finding one vulnerable entry point when there may be many more? This can be a great application of a security scanner, but I never use a scanner as a substitute for figuring out the source code pattern that manifests the defect.

Why? Scanners can still miss bugs even if they are are scannable. This is typically due to just not using the scanner to reach every possible path through an application. Route, method, and request parameter coverage are all critically important even when using automated tools.

Reality: Scanners will find some instances of scannable security bugs.

Misperception: Scanners Can Run Automatically

Sure, most security scanners can be triggered to run automatically. This is typical usage for a continuous integration, continuous delivery (CI/CD) pipeline. What is not automatic, however, are the steps required to get the most out of a security scanner. Just running a tool “out of the box” without customization can leave lots of potential benefits unrealized. Scanners also require careful monitoring to ensure that route, method, parameter, and code-coverage objectives are being met and improving over time. Lastly, interpreting findings isn’t always easy. Sometimes even low-severity defects can be chained to make something much worse happen.

Every security scanner requires a support structure to be most effective. These support structures require continual effort to maintain and monitor.

Reality: Scanners require customization, monitoring, and training.

How To Do Good Security Work With Security Scanners

Software security scanners can be a valuable part of any application security program. However, deploying one naively can give false assurances and waste developer effort on ineffectual tasks. The investment to launch a security scanner into a CI/CD pipeline can pay off if the right steps are taken first. We group these tasks as “security engineering” and have worked with many customers to make sure that their security investment dollars go towards the most valuable efforts.