Measuring Application Security

We report on a qualitative study of application security (AppSec) program management. We sought to establish the boundaries used to define program scope, the goals of AppSec practitioners, and the metrics and tools used to measure performance. We find that the overarching goal of AppSec groups is to ensure the security of software systems; this is a process of risk management. AppSec boundaries varied, but almost always excluded infrastructure-level system components. Seven top-level questions guide practitioner efforts; those receiving the most attention are Where are the application vulnerabilities in my software?, Where are my blind spots?, How do I communicate & demonstrate AppSec's value to my management?, and Are we getting better at building in security over time?. Many metrics are used to successfully answer these questions, but one challenge stood out: there is no good way to measure AppSec risk. No one metric system dominated observed usage.