London, UK 
6 Snow Hill, London, EC1A 2AY, UK

+442081331119

The Hague, The Netherlands 
WTC The Hague, 
Prinses Beatrixlaan 582 
2595 BM Den Haag

+31 (0) 70 240 0021

Sofia, Bulgaria 
141 Tsarigradsko Shose Blvd, VIP Security Building, Floor 2, Sofia 1784, Bulgaria

  • YouTube
  • Black Facebook Icon
  • Black Twitter Icon

© 2019 HeleCloud™

Container Vulnerability Scanning on AWS with Clair and AWS Fargate

March 13, 2019

Introduction

 

Security and compliance of container images is a trending hot topic. The challenge discussed in this article is how to integrate a container vulnerability scanning solution within a CI/CD process. Ideally, the outcome of such a pipeline would be a secure golden image that meets the security and compliance requirements of your company and at the same time contains all the software packages that the application teams need. 

 

We took a look at several options on the market and picked Clair for the following reasons: 

 

  • Clair was developed by the CoreOS team, now part of RedHat, thus backed by a major open source contributor and vendor.

  • Clair has been out there for a while and there is a rich eco-system of tools that use the Clair API.

  • It is quite easy to be integrated within a CI/CD process.
     

Solution Overview

Figure 1 Clair Container Scanner

 

The solution is shown in Figure 1. It is worth mentioning that Clair follows a traditional client-server architecture, where the Clair server is an API hosted on a container and the clients are third party tools that send commands to the API. The solution contains the following key components: [NB1] 

  • Clair container – this is a container that hosts the scanning API. Clients need to make API requests to initiate any action (including image scanning). In our implementation,the Clair container has been slightly modified and hosted on AWS Fargate. It is a separate task in an autoscaling group. Details about the implementation are given below. 

  • PostgreSQLDB[NB2] – this is a vulnerability database that is being regularly updated with vulnerabilities for both OS and non-OS components of the container. 

  • Clients – these are tools that implement simple CLI commands to use the Clair functionality. The CLI commands trigger API requests to the Clair API. While there is a variety of options for client solutions – Clair Klar has been chosen for its simplicity and functionality. In the post, we execute Klar CLI commands from a machine in the VPC. However, it is possible to install the tool as part of the CI/CD process – as an AWS CodeBuild project for example.  

  • Application [DR3] Load Balancer – an ELB is used to present a consistent DNS record in front of the Clair container hosted on AWS Fargate, it is also easy to integrate the ELB with AWS Certificate Manager, and apply a free SSL certificate to make encrypted calls to the service. 

Solution Overview

 

  1. Create an RDS database

Figure 2 Clair DB deployment

 

Clair requires a functioning database instance. While the example presented in the official page creates a PostgreSQL container, it is generally recommended to have a separate PostgreSQL instance installed. In this example, we make use of a managed RDS instance. 

 

In our experience, a small t2 instance with 10 GB of storage is sufficient for smooth operation. The database should be however monitored.

Figure 3 Clair DB Settings

 

2. Prepare the container image

 

The Clair container image is public. We recommend to pull the latest STABLE release. 

 

NOTE: If you specify "image:latest" there is no guarantee that the release will be stable. The exact release must be explicitly specified - in our case this is v2.0.7. In the example, we use an ec2 instance (a medium t2 as some RAM is needed to test) with IAM permissions to upload an image to an ECR. 

 

So, here's what should be done: 

 

2.1. Install Clair

 

First, let's install docker simply by issuing the following command: 

------------------------------------------------------------------------------------------------------------------------------

$ yum install docker 
------------------------------------------------------------------------------------------------------------------------------

 

Next, we download the Clair container image. At the time of writing this article, the latest stable release is v2.0.7. Note that if you download the latest development release, the tools might not work.

------------------------------------------------------------------------------------------------------------------------------

$docker pull quay.io/coreos/clair:v2.0.7

$yum install python-pip

------------------------------------------------------------------------------------------------------------------------------

In order to build an image that will work on AWS Fargate, we should download and modify the docker compose file:

------------------------------------------------------------------------------------------------------------------------------

$ curl -L https://raw.githubusercontent.com/coreos/clair/master/contrib/compose/docker-compose.yml -o $HOME/docker-compose.yml

$ mkdir $HOME/clair_config

------------------------------------------------------------------------------------------------------------------------------

 

Then we download the configuration file:

------------------------------------------------------------------------------------------------------------------------------

curl -L https://raw.githubusercontent.com/coreos/clair/master/config.yaml.sample -o $HOME/clair_config/config.yaml

------------------------------------------------------------------------------------------------------------------------------

 

Edit the configuration file:

------------------------------------------------------------------------------------------------------------------------------

$nano $HOME/clair_config/config.yaml 

------------------------------------------------------------------------------------------------------------------------------

 

The important part is to update the file with the created PostgreSQL DB string. In our case this would be: 

------------------------------------------------------------------------------------------------------------------------------

  options:

      # PostgreSQLSQL Connection string

      # https://www.PostgreSQLsql.org/docs/current/static/libpq-connect.html#LIBPQ-CONNSTRING

      source: PostgreSQLsql://clairdb:Password123@clair_db_uri/clairdb     #clairdb:Password123 - this is username and password of the     PostgreSQL database 

     #clairdb after the '/' is the name of the database schema.  

------------------------------------------------------------------------------------------------------------------------------

NOTE: In this example for simplicity, we are hardcoding the password in the configuration file which is generally a bad practice. To solve this, when implementing in a production environment, the sensitive information can be retrieved from the SSM Parameter Store and passed to the container in Fargate as an environment variable.

 

Next, we need to install docker compose through the following PIP command:

------------------------------------------------------------------------------------------------------------------------------

$ pip install docker-compose

------------------------------------------------------------------------------------------------------------------------------

 

Before running docker compose, we should modify the docker compose file we downloaded in the follow manner:

------------------------------------------------------------------------------------------------------------------------------

##########DOCKER COMPOSE############

 

version: '2'

services:

  clair:

    container_name: clair_clair

    image: quay.io/coreos/clair:v2.0.7

    restart: unless-stopped

    ports:

      - "6060-6061:6060-6061"

    volumes:

      - /tmp:/tmp

      - ./clair_config:/config

    command: [-config, /config/config.yaml]

 

#######################################

------------------------------------------------------------------------------------------------------------------------------

Finally, we launch docker compose:

------------------------------------------------------------------------------------------------------------------------------

$ docker-compose -f $HOME/docker-compose.yml up -d

------------------------------------------------------------------------------------------------------------------------------

 

…and check whether the container is running:

------------------------------------------------------------------------------------------------------------------------------

$ docker container ls

CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS              PORTS                              NAMES

42616dda4906        quay.io/coreos/clair:v2.0.7   "/clair -config /con…"   23 hours ago        Up 6 seconds        0.0.0.0:6060-6061->6060-6061/tcp   clair_clair

------------------------------------------------------------------------------------------------------------------------------

 

2.2. Install and test Clair Klar

------------------------------------------------------------------------------------------------------------------------------

$ yum install go

$ go get github.com/optiopay/klar

$ find -name "*klar*" ====> ./go/bin/klar

$ export PATH=$PATH:/root/go/bin/

$ export CLAIR_ADDR=localhost

$ export CLAIR_OUTPUT=Low

------------------------------------------------------------------------------------------------------------------------------

 

Now, we can test if the Clair API works:

------------------------------------------------------------------------------------------------------------------------------

$ klar PostgreSQLs:9.5.1

------------------------------------------------------------------------------------------------------------------------------

NOTE: In this case we pull an image from docker.io. In reality, any container repository can be used such as Amazon ECR. 

 

EXAMPLE OUTPUT:

------------------------------------------------------------------------------------------------------------------------------

-----------------------------------------

CVE-2018-7169: [Medium]

Found in: shadow [1:4.2-3+deb8u1]

Fixed By:

An issue was discovered in shadow 4.5. newgidmap (in shadow-utils) is setuid and allows an unprivileged user to be placed in a user namespace where setgroups(2) is permitted. This allows an attacker to remove themselves from a supplementary group, which may allow access to certain filesystem paths if the administrator has used "group blacklisting" (e.g., chmod g-rwx) to restrict access to paths. This flaw effectively reverts a security feature in the kernel (in particular, the /proc/self/setgroups knob) to prevent this sort of privilege escalation.

https://security-tracker.debian.org/tracker/CVE-2018-7169

-----------------------------------------

CVE-2016-9841: [High]

Found in: zlib [1:1.2.8.dfsg-2]

Fixed By:

inffast.c in zlib 1.2.8 might allow context-dependent attackers to have unspecified impact by leveraging improper pointer arithmetic.

https://security-tracker.debian.org/tracker/CVE-2016-9841

-----------------------------------------

------------------------------------------------------------------------------------------------------------------------------

NOTE: The severity of the Common Vulnerabilities and Exposures

(CVEs) that Clair reports are configurable. A user can also define passing criteria and whitelists. More information about the configuration options can be found here.

 

2.3. Prepare the image for AWS Fargate

 

First, we create a new image based on the running container: 

------------------------------------------------------------------------------------------------------------------------------

$ docker container ls

CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS              PORTS                              NAMES

42616dda4906        quay.io/coreos/clair:v2.0.7   "/clair -config /con…"   4 minutes ago       Up 4 minutes        0.0.0.0:6060-6061->6060-6061/tcp   clair_clair

 

$ docker commit 42616dda4906 fargate_clair_image:latest

------------------------------------------------------------------------------------------------------------------------------

 

Then we can verify whether the image is there:

------------------------------------------------------------------------------------------------------------------------------

$ docker image ls

REPOSITORY                 TAG                 IMAGE ID            CREATED             SIZE

fargate_clair_image        latest              67c1aac60aeb        4 seconds ago       353MB

klar                       latest              abb9667bcc83        4 days ago          353MB

quay.io/coreos/clair-git   latest              be456957ff54        6 days ago          52.6MB

quay.io/coreos/clair       v2.0.7              7968a46189ae        2 months ago        353MB

------------------------------------------------------------------------------------------------------------------------------

In order to prepare the image to run in a standalone manner on Fargate, we need to create a new image that contains the configuration file. This can be achieved in many ways. For example: 

------------------------------------------------------------------------------------------------------------------------------

$ cd ~/clair_config

$ nano Dockerfile

------------------------------------------------------------------------------------------------------------------------------

------------------------------------------------------------------------------------------------------------------------------

############Dockerfile##############

FROM fargate_clair_image:latest

COPY config.yaml /config/config.yaml

####################################

------------------------------------------------------------------------------------------------------------------------------

Finally, we can build the image that we are going to use to launch the Fargate container service:

------------------------------------------------------------------------------------------------------------------------------

$ docker build -t fargate_clair:latest.

------------------------------------------------------------------------------------------------------------------------------

 

 

3. Create an ECR repo and upload the prepared working Clair image

 

Now, we are ready to create the repo:

 

 

 

Figure 4 ECR Repository

 

And … upload the image we prepared in the previous step: 

------------------------------------------------------------------------------------------------------------------------------

$ (aws ecr get-login --no-include-email --region eu-west-1)

 

$ docker tag fargate_clair:latest 389211405395.dkr.ecr.eu-west-1.amazonaws.com/vulnerability_scan:v2.0.7

 

$ docker push 389211405395.dkr.ecr.eu-west-1.amazonaws.com/vulnerability_scan:v2.0.7

------------------------------------------------------------------------------------------------------------------------------

 

4. Create a load balancer

 

 

Figure 5 Application Load Balancer

 

5. Create the AWS Fargate setup

 

We will not go through all the details. The values given below have been tested to be working smoothly:

  • Prepare the Fargate task referencing the ECR URL. In our case this is: 

  • 389211405395.dkr.ecr.eu-west-1.amazonaws.com/vulnerability_scan:v2.0.7

  • CPU Units: 1024 (or 1 vCPU)

  • Memory: 2048 MB

  • Task Execution Role: ecsTaskExecutionRole which is an AWS managed role

6. Create the Fargate cluster and service

  • Security group - must allow all the traffic out (including non-TCP). Must allow the traffic from the LB listener IP range in. 

  • Autoscaling - normally auto-healing configuration would be enough to create a service that is reliably utilized by a CI/CD process. However, if there is a need for parallel scanning of images, then the maximum number of containers should be increased.

  • ECS Autoscaling Service metric - CPU

  • Networking - in our specific setup the containers are hosted in a private security zone that is not accessible from the internet. The load balancer has a listener in the DMZ on port 6060 forwarding to port 6060 in the backend. However, the health check in the target group is on another port (by default 6061). 

7. Verify that service works

 

7.1. Check whether the task is running

 

Normally, the task should be in "RUNNING" status. 

Figure 6 Fargate Running Container

 

Indications that something is wrong include: 

  • Service is stuck in "ACTIVATING" state. This usually means that the container is being constantly restarted. In this case if would be helpful to take a look at the CloudWatch log corresponding to the respective task. If there is no log available, this typically means that the task is unable to pull the image which is an indication that the issue is related to networking or authentication with the repository. 

  • The CloudWatch logs of the task should indicate that the container successfully connects to and updates the vulnerability database which in our case is an RDS instance. If there are any sort of errors there, it is to be expected that the scanning of images would not be successful. By default, the vulnerability database is updated every 6 hours. However, this option is configurable by the user. 

7.2. Verify whether the Klar tool is still able to scan using the Fargate Clair container

 

First, define the environment variable that identifies the Clair API:

------------------------------------------------------------------------------------------------------------------------------

$ export CLAIR_ADDR=http://load_balancer_uri:6060

------------------------------------------------------------------------------------------------------------------------------

 

Second, test Klar: 

------------------------------------------------------------------------------------------------------------------------------

$ klar PostgreSQLs:9.5.1

------------------------------------------------------------------------------------------------------------------------------

The result should be the same as the result we got in section 2.2. 

 

Final Thoughts

In conclusion, Clair has been identified as a suitable open source solution that can be easily integrated within a CI/CD pipeline. The article exemplifies a simple implementation that helps to address the container security problem in an automated manner. However, there are many options and solution customizations that are outside the scope of this article. Please get in touch if we can help with AWS solution designs or any other aspect of the AWS platform.

 

Links

https://github.com/coreos/clair

https://github.com/optiopay/klar

https://docs.docker.com/

https://aws.amazon.com/rds/

https://aws.amazon.com/fargate/

 

 

 

 

 

 

 

 

Share on Facebook
Share on Twitter
Please reload

Featured Posts
Archive
Please reload