Security for Continuous Integration.

Automated testing in a world of "evil users."

Ian Bartholomew
March 9, 2015

Right now, we in technology are witnessing the convergence of two competing forces: on one hand, an increasing need for security (as demonstrated by these events last year); on the other, an increasing number of organizations adopting Continuous Integration (CI). In a CI model, code is integrated regularly — usually several times a day — and checked against automated tests.  Coupled with Continuous Delivery and Continuous Deployment, CI is getting code into live applications faster.  While Continuous Integration is improving the agility and speed-to-market of software organizations, that speed can leave developers more vulnerable to security breaches. 

Why unit and functional tests aren’t enough.

From a security standpoint, unit and functional tests don’t offer much protection. As part of the CI toolchain, they run against the application during integration. When developers write the unit tests, they focus on the smallest testable piece of the application. For integration tests, they focus on how the integrated code performs against expectations (for example, did the controller return a 200 based on our request to a known route?) or based on functional requirements. The problem with relying solely on unit and integration tests is that they are focused on the functional aspects of the application, not on security. 

Don’t forget the evil user.

If a team is agile, then it most likely has user stories that define what users can do. Equally important, however, is the converse of the user story: the evil user story. These describe threats from bad actors that we want to test for. For example, “As a hacker, I can read and modify any data that is input and output by the application.”

Know what you're testing.

In order to make CI secure, developers should incorporate automated, security-focused tests and scans into the integration pipeline. A good starting place  is OWASP, the Open Web Application Security Project. OWASP is a great (albeit disorganized) repository of knowledge that can lay a good foundation for testing security vulnerabilities in an application, such as XSS or SQL injection. OWASP provides many great resources, including the OWASP Top Ten, the OWASP Testing Guide and the Web Application Security Cheat Sheet

The lists from OWASP are a great foundation for evil user stories, as they very clearly enumerate the ways in which a bad actor can compromise the security of an application. 

Automating security tests.

Once developers understand what they need to test for, they need to automate that process and bring it into the CI pipeline.  There are a number of tools that will run scans against an application using automated spider crawls to look for common vulnerabilities, such as OWASP Zap, Burp Suite and Arachni ScannerOWASP-OWTF and Minion do not run scans themselves but, similarly to Jenkins and other continuous integration servers, have a plugin architecture that allows them to run any number of other programs, kicking off scans and gathering the results for presentation.  Minon and OWTF gather the output from other programs, like OWASP ZAP, into a readable and presentable format. 

BDD-based frameworks including BDD-Security, Gauntlt and Mittn enable developers to write  tests in natural language. BDD-Security uses Selenium WebDriver to drive the interactions described in the stories, and as a backend it uses OWASP ZAP.  Gauntlt takes a different approach, and uses the stories to drive individual tools like curl or nmap.  With all of these tools, it’s up to the user to write tests that are specific to the application at hand, though there are some boilerplate stories provided in each. Here is example of tests written in natural language (Gherkin in this case):

This specific test is written for Gauntlt, but it closely resembles tests written for any of the BDD-style frameworks. It lists the feature, scenario, assertions, and expectations. This test does a curl attack against a host, checking that different HTTP verbs are not allowed and sending back the correct error code. 

During test development, it's important to integrate these frameworks into the CI build pipeline. BDD-Security can run from Jenkins.  Mittn is based on the Behave framework, so it can integrate into Jenkins via it JUnit XML test output.  Gauntlt can integrate via StdOut format and HTML output (JUnit output is possible, though not by default). 

Choose the right tool.

Choosing which tool is right for any particular team is something that may come down to taste and the workflow. If a BDD workflow is already in place and the team has strong security knowledge to leverage, something like Gauntlt or BDD-Security might be best.  A team just starting out, however, might get more out of OWASP-OWTF or Minion. Most of the tools, such as Gauntlt’s Virtual Box starter kit, make it very easy to experiment, so try a few tools to determine which is most effective.

Regardless of specific product choices, integrating one or more of these testing tools into the CI pipeline is a critical step for any organization concerned with security.