Authoritative Containers — Part 2

HOW TO DEFINITIVELY TRUST THAT THE CONTAINERS YOU ARE DEPLOYING INTO PRODUCTION CONTAIN WHAT YOU THOUGHT THEY CONTAINED

Mark Argent
5 min readJul 14, 2021

In part 1 of this series I shared some thoughts on the types of assurance needed to ensure container images are safe to be deployed to an enterprise’s production environment.

In this article I share potential ways to implement that assurance, and I provide my recommendations for capturing the provenance of your container images and generating the guarantees over their contents.

I have captured my recommendations in the following picture to show the different measures that can be undertaken.

Picture of pipeline of measures for building an authoritative container image
Pipeline for building authoritative container images

Only use trusted Base Images

A recent article on work undertaken by security firm Prevasio shows that 51% of Docker Hub container images have exploitable vulnerabilities.

Be explicit about the base image for your container image. Put in place measures to whitelist public container images as part of the security controls in your software development life cycle (SDLC).

Consider pulling your approved base images into a private repository and only allowing your container image to be built from one of those in your repository. Even when you’re working with container images from reliable sources, it’s crucial to check the security of each image and its layers.

Undertake Code Analysis of the container’s contents

Perform automatic code quality reviews and static analysis of your code to detect bugs, code smells, and security vulnerabilities.

Vulnerability scanning is, at the simplest level, the use of software tools to identify and report on security issues that affect the code running in your container.

Ensure you include static application security testing (SAST) — there are several tools for analysing your application source code, byte code and binaries for coding and design conditions that are indicative of security vulnerabilities.

Perform Composition Analysis of the container’s contents

75% of codebases contain open source components with known security vulnerabilities (HelpNetSecurity — Open Source Risks)

Perform and record vulnerability scans of your application code’s dependencies to assure the container image.

There are several tools that detect publicly disclosed vulnerabilities contained within your code’s dependencies. These tools scan your applications (and their dependent libraries) to identify any known vulnerable components.

Log the use of dependancies for continuous tracking of vulnerabilities.

Your code’s dependancy information should be held in both the source and destination environments to provide the foundation for alerting when new vulnerabilities are identified. So put in place measures to continuing check the dependancies you know you have in your application code.

Too often I’ve seen the knowledge of an application’s dependencies not being stored in a way that means it is capable of being queried — therefore when new vulnerabilities are identified it takes too long to identify which containers in a production environment are susceptible to the vulnerability.

Conduct Static Container Analysis

Scan your container image by doing static analysis, which means it analyzes images without a need to run the container.

Performing Static Container Analysis of your image will allow you to detect OS and non-OS vulnerabilities. Use a recognised image scanning technology (eg. docker security scanning using Sysdig). These technologies scan each container layer and provide a notification of vulnerabilities that may be a threat, based on the Common Vulnerabilities and Exposure (CVE) database.

According to Sergei Schevchenko, as quoted in an article from The New Stack, “Container images need to be continuously rebuilt and retested to make sure they don’t rely on vulnerable dependencies.”

Conduct Application Security Scanning

Implement an automated tool to scan your applications, from the outside (ie. whilst it is running), to look for security vulnerabilities such as Cross-site scripting, SQL Injection, Command Injection, Path Traversal and insecure hosting configuration.

This category is frequently referred to as Dynamic Application Security Testing (DAST). Ensure you test your application or software product in an operating state — deploy and start your application and use the tools to find vulnerabilities that are not specifically linked to a code module, but happen during use.

Implement Dynamic Container Analysis

Currently, there’s no practical method to verify what a container image will do when activated, short of actually running it, which can be risky without properly isolating it from the internal network.

Perform scanning of your container image at runtime

There are capabilities ( Aqua DTA, Sysdig, StackRox) for dynamically assessing the risks that a container images pose before they are run as containers in your production environment. These run your container images in a safe and isolated sandbox environment, and analyse the image for several minutes, determining the potential risk level that the image presents if allowed to run in an open, networked environment.

Sign the container image

Sign your image to provide immutable assurance on the production environment that the container image was built using an approved pipeline and hasn’t been tampered/changed since.

Signing container images provides a means of validating where a container image came from. It facilitates checking that the image has not been tampered with, and the setting of policies to determine which validated images you will allow to be pulled to your production environment.

Store the provenance for your container image

Record the outcomes of all the measures you’ve undertaken, and create the Software Bill of Materials (SBOM) for your container image

Implement a repository to record all the evidence that is generated during the process of building and scanning your container. This evidence should be collected by your build pipeline during its execution. The repository should store the details of the artefacts that have been built and create the covering Software Bill of Materials (SBOM) to provide fully traceability of the build and container image’s provenance. This SBOM can be provided to any Security Operations Centre(SOC) your enterprise may have.

Build your container image on a segregated environment

Create a separate environment to host a centrally controlled “trusted pipeline” to build your container images.

The most secure approach is if you can perform all these measures and generate the provenance information for your container image in an environment managed independently of your development team.

This segregated environment must be secured to minimise the threat of attack through a compromised environment and should be on network segregated from other functions to protect against network attack.

Building an Authoritative Container Image

The final article (part 3) in this series provides an example of implementing the measures set out above. The ‘Authoritative Container Builder’ captures and collates the assurances needed to transfer a container image across environments of different security classification. You can read about the ‘Authoritative Container Builder’ in Part 3.

--

--

Mark Argent

IBM Distinguished Engineer. Experienced hands-on Chief Architect with over 20 years experience in the design of large complex systems and leading agile delivery