London, UK 
6 Snow Hill, London, EC1A 2AY, UK

+442081331119

The Hague, The Netherlands 
WTC The Hague, 
Prinses Beatrixlaan 582 
2595 BM Den Haag

+31 (0) 70 240 0021

Sofia, Bulgaria 
141 Tsarigradsko Shose Blvd, VIP Security Building, Floor 2, Sofia 1784, Bulgaria

  • YouTube
  • Black Facebook Icon
  • Black Twitter Icon

© 2019 HeleCloud™

How to overcome AWS Copy AMI boundaries by using Hashicorp’s Packer

March 21, 2019

By Nikolay Bunev, Cloud Services Engineer at HeleCloud

 

Introduction

 

One of the many benefits of Cloud Computing is the concept of launching as many instances as you want or need, from a single source which in the AWS world is named Amazon Machine Image (AMI), usually referred to as a “Golden Image”.

 

The Golden AMI could potentially include security hardening and application specific configurations and code, which you can then use to easily install your application.

That works pretty well if you have a single environment utilizing only one AWS region. 

However, most likely you would need multiple environments like Development, Staging and Production and potentially you may also need to deploy your application in another region to be close to your end users.

If your use case resembles the second scenario, you’ll need to share your AMI with the rest of the accounts and copy it to your secondary regions. And that’s where that may be a bit of an issue, especially if you have followed the Well Architecture Framework principles and security best practices, and your volumes are encrypted.

 

The Problem 

 

In general, there are two ways to encrypt an instance volume:

• Encrypt the volume when you create the instance for the first time

• Apply encryption while copying a snapshot

 

There are two ways to create encrypted AMIs too:

• Create an AMI by using an encrypted snapshot

• Create an encrypted AMI from an un-encrypted AMI (by using the Copy Image functionality)

 

However, creating encrypted AMIs or snapshots has some limitations as outlined in the AWS documentation:

Some Linux distributions, such as Red Hat Enterprise Linux (RHEL) and SUSE Linux Enterprise Server (SLES), use the Amazon EC2 billingProduct code associated with an AMI to verify subscription status for package updates. Creating an AMI from an EBS snapshot does not maintain this billing code, and instances launched from such an AMI are not able to connect to package update infrastructure. If you purchase a Reserved Instance offering for one of these Linux distributions and launch instances using an AMI that does not contain the required billing code, your Reserved Instance is not applied to these instances.

Similarly, although you can create a Windows AMI from a snapshot, you can't successfully launch an instance from the AMI.

 

Furthermore, Copying encrypted AMIs to other accounts and regions comes with a limitation on its own due to the billingProduct code:

• You can't copy an encrypted AMI that was shared with you from another account. Instead, if the underlying snapshot and encryption key were shared with you, you can copy the snapshot while re-encrypting it with a key of your own. You own the copied snapshot, and can register it as a new AMI.

• You can't copy an AMI with an associated billingProduct code that was shared with you from another account. This includes Windows AMIs and AMIs from the AWS Marketplace. To copy a shared AMI with a billingProduct code, launch an EC2 instance in your account using the shared AMI and then create an AMI from the instance. 

 

So, first of all, you can’t copy an encrypted AMI from one account to another due to the encryption key. That’s not a big issue, if you own both accounts, as you’ll have the source encryption key, which will make it easier for you to just re-encrypt the snapshot with the destination encrypting key in your secondary account. 

This will mostly work if you use free community-built Linux distributions like CentOS and Ubuntu.

However, if you use an enterprise provided distribution like RedHat or SUSE, or Windows then you’ll have to deal with the billingProduct code limitation too.

 

The Solution

 

As outlined in the AWS documentation:

To copy a shared AMI with a billingProduct code, launch an EC2 instance in your account using the shared AMI and then create an AMI from the instance. 

 

To sum it up, in order to have an encrypted AMI that’s using a billingProduct code which is practically the same as the original AMI in your source account, you need to:

• share the AMI with the second account (the AMI shouldn’t be encrypted)

• create an instance from the shared AMI

• create a new encrypted AMI from that instance

That approach would work, but it’s a manual process. The “Golden AMI” benefits are best used when the image creation is automated and incorporated within your CI/CD process. One possible option to automate the image distribution is to write a Lambda function that will do so by utilizing any of the AWS APIs available.

 

At HeleCloud, we use a custom-build “Golden AMI Builder” process similar to the one described in this AWS DevOps blog post which utilizes AWS CodeBuild / CodePipeline and HashCorp’s Packer.

Packer is another great tool from HashiCorp’s team and it’s worth checking if you are not familiar with it already. It uses a JSON based template in which you can describe how your AMI should look like (which is the base AMI for it, does it have encryption, instance type during build, provision scripts that you may want to execute during the build, etc.)

We already use Packer to build our Golden AMIs, so we asked ourselves, why not use the same tool to overcome the billingProduct issue within the same CI/CD we have already. Luckily, Packer supports that.

 

The picture above is a bit over-simplified as our solution has other stages as well, but it can still serve the purpose to illustrate the approach.

We have a “Build AMI” or “Initial Packer Build” stage in which we do the usual – harden the AMI security, install any application code pre-requisites, install third-party tools we need, etc. This stage will produce a ready-to-use AMI, but that AMI will NOT be encrypted.

We then have the “Distribute AMI” or “Secondary Packer Build” stage in which we encrypt the AMI with a dedicated KMS key and distribute it across accounts and regions. The Distribute AMI is basically a number of simultaneously started CodeBuild Projects (one for each destination account).

In this stage we use three EBS Packer Builder configuration options:

• ami_regions (array of strings) - A list of regions to copy the AMI to. Tags and attributes are copied along with the AMI. AMI copying takes time depending on the size of the AMI, but will generally take many minutes.

• region_kms_key_ids (map of strings) - a map of regions to copy the ami to, along with the custom kms key id (alias or arn) to use for encryption for that region. Keys must match the regions provided in ami_regions. If you just want to encrypt using a default ID, you can stick with kms_key_id and ami_regions. If you want a region to be encrypted with that region's default key ID, you can use an empty string "" instead of a key id in this map. (e.g. "us-east-1": "")

• source_ami (string) - The initial AMI used as a base for the newly created machine. source_ami_filter may be used instead to populate this automatically.

There are a few more options in use like vpc_id, subnet_id (in which vpc and subnet id to start the packer build), and of course, encrypt_boot.

Most of those configuration options are dynamically inherited in Packer as input variables from CodeBuild. The source_ami id is taken as input from the project artifacts of the initial build stage. The packer template example below closely resembles the one we have in use:

 

{
  "variables": {
    "keep_input_artifact": "true",
    "vpc": "{{env `vpc_id`}}",
    "subnet": "{{env `subnet_id`}}",
    "aws_region": "{{env `AWS_REGION`}}",
    "os_type": "{{env `os_type`}}",
    "app_type": "{{env `app_type`}}",
    "ami_name": "{{env `ami_name`}}",
    "kms_key": "{{env `kms_key`}}",
    "ami_id": "{{env `ami_id`}}"
  },
  "builders": [{
    "type": "amazon-ebs",
    "region": "{{user `aws_region`}}",
    "source_ami": "{{user `ami_id`}}",
    "ami_block_device_mappings": [
      {
        "delete_on_termination": "true",
        "device_name": "/dev/sda1"
      }
    ],
    "instance_type": "t2.medium",
    "user_data_file": "windows/bootstrap_win.txt",
    "communicator": "winrm",
    "winrm_username": "administrator",
    "ami_name": "{{user `ami_name`}}",
    "encrypt_boot": "true",
    "kms_key_id": "{{user `kms_key`}}",
    "ami_regions": [ "eu-west-1", "eu-west-2" ],
    "region_kms_key_ids": {
      "eu-west-1": "KMS_KEY_IN_EU_WEST_1",

      "eu-west-2": "KMS_KEY_IN_EU_WEST_2",
    },
    "tags": {
      "Name": "{{user `ami_name`}}",
      "ami_base_id": "{{.SourceAMI}}",
      "ami_base_name": "{{.SourceAMIName}}",
    },
    "ami_description": "{{user `os_type`}} {{user `app_type`}}",
    "associate_public_ip_address": "false",
    "vpc_id": "{{user `vpc`}}",
    "subnet_id": "{{user `subnet`}}"
  }]
}

 

As pointed out above, if you leave the region_kms_key_ids value for key empty, it will use the default kms within that account for the specified region.

It should also be noted that the regions listed as keys in region_kms_key_ids should be exact match of those you specify as a list in ami_regions as otherwise packer will fail.

As outlined previously, the template will be executed multiple times simultaneously depending on the CodeBuild projects added to the Distribute AMI stage in the CodePipeline. If we assume that we have – Development, Staging and Production the execution of the CodePipeline will produce one AMI in each region listed in ami_regions encrypted with the KMS key specified in region_kms_key_ids in each of the three accounts.

 

Final Thoughts

 

Protecting data at rest is among the key objectives in the Security Pillar of the AWS Well-Architected Framework. In order to follow that practice and create encrypted AMIs of Operating Systems that have an associated billingProduct code like RHEL, SUSE and Windows you need the right tool.

Please get in touch if we can help with AWS solution designs, or any other aspect of the AWS platform.

 

Links

 

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/creating-an-ami-ebs.html

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIEncryption.html

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/CopyingAMIs.html

https://aws.amazon.com/blogs/devops/how-to-create-an-ami-builder-with-aws-codebuild-and-hashicorp-packer/

https://www.packer.io/docs/builders/amazon-ebs.html

 

 

 

 

Share on Facebook
Share on Twitter
Please reload

Featured Posts
Archive
Please reload