CandyShop: Security Tool Benchmark Results
Real scan results from 15+ security tools tested against intentionally vulnerable applications. Compare SAST, DAST, SCA, and container scanners with actual data.
What is CandyShop?
CandyShop aggregates scan results from popular security tools applied to well-known vulnerable applications. The goal is simple: give security professionals real benchmark data to compare tools.
Every tool was run against the same targets with default configurations. Scan duration, finding counts by severity, and detection categories are all tracked.
This matters because every vendor claims high detection rates. But when you run tools side by side against the same targets, differences become obvious.
Test Environments
The benchmark uses intentionally vulnerable applications that are widely recognized in the security community:
OWASP Juice Shop
Node.js/Express/Angular. 100+ security challenges covering OWASP Top 10 and beyond.
Broken Crystals
Node.js/TypeScript by Bright Security. 22+ vulnerability types including JWT flaws and XXE.
Altoro Mutual
J2EE banking application by HCL. Classic web vulnerabilities in an enterprise-style app.
Vulnerable Flask App
Python/Flask. Simple target for testing Python-specific security scanners.
Scanning Tools
The benchmark includes tools across four categories:
| Category | Tools Tested |
|---|---|
| SAST | Semgrep, CodeQL, NodeJsScan, Gosec, Brakeman, Bandit, Find Security Bugs, Security Code Scan, Gitleaks, ESLint, Psalm |
| DAST | OWASP ZAP, Nuclei |
| SCA | Dependabot, OWASP Dependency-Check, Nancy |
| Container | Trivy, Grype |
OWASP Juice Shop Results
Scan results against OWASP Juice Shop, a Node.js application with 100+ intentional vulnerabilities.
| Tool | Type | Duration | Critical | High | Medium | Low |
|---|---|---|---|---|---|---|
| NodeJsScan | SAST | 1:50 min | 0 | 2 | 0 | 0 |
| Semgrep | SAST | 2:24 min | 0 | 2 | 42 | 0 |
| CodeQL | SAST | 0:20 min | 19 | 148 | 18 | 0 |
| Nuclei | DAST | 1:09 min | 0 | 0 | 0 | 1 |
| OWASP ZAP | DAST | 6:59 min | 0 | 0 | 32 | 32 |
| Dependabot | SCA | 0:01 min | 0 | 1 | 3 | 0 |
| Dependency-Check | SCA | 3:18 min | 0 | 0 | 0 | 0 |
| Trivy | Container | 1:24 min | 8 | 15 | 16 | 11 |
| Grype | Container | 1:50 min | 25 | 57 | 77 | 2 |
Key observations:
- CodeQL found the most code-level vulnerabilities (185 total) in 20 seconds—the fastest SAST scan.
- Grype reported more container vulnerabilities than Trivy (161 vs 50), though some may overlap or be classified differently.
- ZAP took nearly 7 minutes but found 64 runtime issues that SAST tools missed.
- Dependency-Check found zero issues—unusual given Juice Shop’s known vulnerable dependencies.
Broken Crystals Results
Scan results against Broken Crystals, a modern Node.js/TypeScript application with 22+ vulnerability types.
| Tool | Type | Duration | Critical | High | Medium | Low |
|---|---|---|---|---|---|---|
| Semgrep | SAST | 0:28 min | 0 | 0 | 239 | 0 |
| CodeQL | SAST | 0:20 min | 4 | 11 | 37 | 0 |
| Nuclei | DAST | 1:09 min | 0 | 0 | 1 | 0 |
| OWASP ZAP | DAST | 6:59 min | 0 | 1 | 112 | 136 |
| Dependabot | SCA | 0:21 min | 4 | 16 | 6 | 2 |
| Dependency-Check | SCA | 9:15 min | 4 | 30 | 23 | 2 |
| Trivy | Container | 3:06 min | 45 | 458 | 443 | 1144 |
| Grype | Container | 2:56 min | 48 | 478 | 445 | 386 |
Key observations:
- Semgrep found 239 medium-severity issues in under 30 seconds—fast and thorough.
- ZAP found the most DAST issues (249 total) including a high-severity finding.
- Container scanners found significantly more vulnerabilities here than in Juice Shop, reflecting the different base images used.
- Dependency-Check performed better on Broken Crystals, finding 59 issues compared to zero on Juice Shop.
About the Applications
OWASP Juice Shop
The most popular intentionally vulnerable web application. Created in 2014 by Björn Kimminich, it runs on Node.js, Express, and Angular.
Juice Shop contains 100+ security challenges covering OWASP Top 10 categories: broken access control, authentication flaws, XSS, injection attacks, security misconfigurations, and more.
It’s the go-to benchmark for security tool testing because of its breadth and active maintenance.
GitHub: juice-shop/juice-shop
Broken Crystals
A modern intentionally vulnerable application developed and maintained by Bright Security.
Built with Node.js and TypeScript, it features 22+ vulnerability types including JWT authentication flaws, CSRF, LDAP injection, XXE attacks, and business logic vulnerabilities.
Broken Crystals is particularly useful for testing tools against modern JavaScript/TypeScript codebases.
GitHub: NeuraLegion/brokencrystals
Altoro Mutual
A vulnerable J2EE banking application developed in 2008 and still maintained by HCL Technology.
It simulates a banking site with classic web vulnerabilities: SQL injection, XSS, path traversal, and authentication bypasses.
Altoro Mutual is useful for testing tools against enterprise Java applications.
Demo: demo.testfire.net
Related guides:
- 19 DevSecOps Tools for a Budget-Friendly AppSec Program
- DAST Benchmark Project
- 129 Application Security Tools Compared
Suphi Cankurt works at Invicti Security and has spent over 10 years in application security. He reviews and compares AppSec tools across 10 categories on AppSec Santa. Learn more.