Authoritative Containers — Part 1

BUILDING SECURE, “ASSURED” CONTAINER IMAGES FOR TRANSFER ACROSS HYBRID CLOUD ENVIRONMENTS AND BETWEEN MULTI-SECURITY ZONES

Enterprises are increasingly using hybrid cloud to develop and run their business-critical workloads, often using public cloud to develop the solution and private cloud or on-premise infrastructure to host the production workload. Federal and Defence enterprises often have the need to secure their production workloads on a segregated, sometimes air-gapped, environment, but are gaining significant advantages when the development can take place on an alternative public/private cloud of lower security classification.

These shifts present the challenge of how to provide assurances that artifacts produced on a public cloud development environment are safe to be deployed to the enterprise’s private environment.

One approach for providing the required level of assurance is for the development teams to transfer a copy of the source code from their repository to the enterprise’s private repository. This provides an opportunity to inspect the code and oversee its build on the destination environment, however this process comes with several drawbacks. Most notably that it can introduce inconsistencies between the application running on the production environment compared with the one in the development team’s representative test and development environments. It also comes with the overhead of duplicating the automated build pipelines and the testing of the release.

Enabling a development team to deliver container images from their environment to the enterprise’s production environment solves issues associated with shipping of source code. It provides greater confidence that what is deployed to the enterprise’s environment is entirely the same as that in the development team’s representative environment. However, assurance is needed that the container images being produced on the development environment (eg. public cloud) are safe to deploy to the enterprise’s environment (public/private cloud).

So what level of assurance is required to ship a container image between multi-security zone environments ?

1) Assurance that the built container has not been tampered with

Concern: If the contents of the container is amended between building and shipping to the destination environment and there is no way to detect or reconcile this, then a malicious change could be present in the container and run in its destination environment.

Full traceability is required to prove that the code being produced to build the application and the container image it is built into has not been tampered with.

2) Record the pre-requisites required to build the application and container

Concern: Without the destination environment knowing what third-party libraries and their versions being utilised, a Threat Intelligence(TI) function will be unable to easily identify when an application is susceptible to a vulnerability affecting one of its pre-requisites.

The third-party resources used to build the container along with the versions of those components needs to be evidenced. This list can be provided to any Security Operations Centre(SOC) the destination environment may have. The security function can then identify containers that may be affected by known vulnerabilities with a third-party resource and what the effect could be to others within their environment.

3) Perform security testing in your pipeline

Concern: If we do not check for policy violations such as an unexpected container format or we allow common vulnerabilities into the hosting destination then we are increasing the surface area of attack in our custom containers.

Pipelines to build the container image need to contain security testing to look for deviations in any security policies or standards as well as any deviations in coding standards that could indicate an entity has tried to push a change though the pipeline. Testing the application itself and the container that wraps it is also important in the security testing phase of the pipeline.

4) Generate reports from phases in the pipeline and the environments health

Concern: We cannot be sure that the container has not been tampered with during its lifecycle and we cannot trust the environment it is being built and tested has not been compromised.

Validation of the creation of the container images needs to be carried out prior to the image being used in the destination environment. The origin and integrity of the reporting of the container image build is important, where the report was originally produced should be validated. The integrity of the report should also be validated in case the pipeline environment itself has been compromised and the report is altered between leaving the build environment and entering the reporting environment. As well as reporting on the pipeline process there should be reporting on the health of the environment carrying out the build and testing.

5) Ensure a separation of duties between the reporting infrastructure and the pipeline

Concern: We cannot trust that an environment building or testing the container if compromised is not also manipulating the reports being produced.

If someone controlled the infrastructure used by the pipeline as well as the infrastructure used by the reporting function, then they could manipulate the reports and cover their tracks. A separation of duties between the infrastructure and administration of both the pipeline and reporting environment is important to reduce the risk of both environments being compromised.

6) Container image and the reporting sent to alternate infrastructure for validation

Concern: If the container image and reporting are not sent independently from one another, then an adversary that has permission to modify the container could also amend the report and cover tracks.

To prevent anyone who can modify the container image after it has left the pipeline also amending the reporting, the two should be sent independently from another to the entity that will be conducting the validation.

Building an Authoritative Container Image

In part 2 of this series I share potential ways to implement the assurance detailed above.

I have been working with IBM colleagues on creating set of DevOps pipelines (the ‘Authoritative Container Builder’) which captures and collates the assurances needed to transfer a container image across environments of different security classification. I provide the details of that in part 3 of this series.