Amazon Web Services

Cloud Providers

AWS Elastic Container Registry (ECR)

Cloud Services

Permits connection from only a specific public IP range

AWS ECR Resource Policy: Block Outside Specific Public IP Range

Grant n AWS accounts, any principal, to connect to ECR resource and upload images with any tag

AWS ECR: Permit Cross Account Image Upload

Grant n other accounts access to this ECR, account-wide. Use more specific principal for better security

AWS ECR, Permit Cross Account Image Download

AWS Secrets Manager

AWS KMS

Policy for principal (User, Service) to access cross-account secret and KMS CMK (decryption key)

Principal Policy - Permit Access to Cross-Account Secret and KMS Key

AWS CloudFront

For public access, permit only specific CloudFront distribution

S3: Permit Only CloudFront Specific Distribution

IAM assume role trust policy which permits assuming only from specific role(s)

Assume Role Trust Policy with Conditional to Limit to Specific Role

Amazon EC2

Permit EC2 instance to assume IAM role with this trust policy

Assume Role Trust Policy from EC2 Instance

AWS Elastic Container Service (ECS)

Trust policy on an IAM role to permit an ECS task (launched container) to assume the role

Assume Role Policy to Permit ECS Task to Assume IAM Role

Amazon S3

Don't apply this policy - it will block all console and API access, and require root user or TAC to recover

(WARNING) Block All S3 Access Except Root

Useful for dev/stage web development, where site is stored in s3. Can use many public IPs, all other are blocked

Limit S3 Web Access to Specific Public IPs

AWS documentation is comprehensive, but it sometimes feels like you're reading the encyclopedia - there's just so much data! That's why we read the encyclopedia for you, and distilled down what we think are the most important flags for resources and services, and provide example IAM resource policies to give you a running start. Keep an eye out for this series as we build it out to include the most commonly (and most commonly misconfigured) services.

The AWS Elastic Container Registry (ECR) is the native AWS service which hosts container images. These images can optionally be tagged for easy reference. The newest image pushed to the registry has the “:latest” tag, which can be referenced by other services - note this isn’t a best practice, and could include accidentally pushed or test pushes to the ECR. A best practice is to use specific versions, e.g. “ContainerName:v1.0.2”, and pin other services to those specific versions.

AWS ECR supports scan-on-push. When Scan-On-Push is enabled, ECR will scan a new image for security vulns, and provide a list of them with remediation and risk information right in the ECR.

ECR also permits encrypting images with a KMS key.


Each ECR by default has no IAM policy, and permits same-account principals to access the ECR (provided the principal’s IAM policies permit this). However, an IAM policy can be added to limit access to the ECR to specific roles, or from specific public IPs. AWS has a specific ECR and IAM page, link. Images can optionally be encrypted with a KMS controlled private key, and the IAM policy on that resource can provide another layer of security for the images in each ECR.

ECRs are built to store potentially many versions of a single image. If you have two totally different images (not just the same image with different tags) to store, put them in two different ECRs

ECRs stored container images built by any tool (usually Docker), and each image is tagged with a specific image tag that is set by the tool uploading to ECR (e.g. :v1.0.2) as well as the “:latest” tag, which changes each time a new image is uploaded

Each ECR is set to either immutable (Once tag set, image can’t be changed) or mutable (Images can be over-written by a new upload)

Images can optionally be encrypted with KMS Amazon provided or customer-provided keys. This adds an additional security step, since access to the decryption key is also required to access the images

IAM Policy Examples
Several examples are listed below. For an up to date list of all stored policies, visit this page: 

Permit account B and C access to download images from ECR in account A. This uses a broad filter “:root” which indicates the entire account 222 and 333 are trusted. If principals in those accounts are granted rights to access this ECR, they will be permitted by this ECR resource policy.

You could instead permit only specific roles by using this principal syntax:

 - An IAM JSON document governing who can access. Defaults to blank, and permits same-account access (provided principal’s role permits). Can limit access to specific roles, regions, public IPs, etc.

 For the most secure workloads, limit the roles that can access the ECR.

- Encryption used to encrypt the stored images. Defaults to AES256, and access controlled only via IAM. If KMS is selected, images encrypted with that private key, and access can also be controlled by who can access the private key for decryption.

: For the most secure workloads, use KMS CMK (customer keys) for encryption.

 Changing this setting after creation will rebuild the resource and destroy any hosted images.

- A name is required. Names must start with a letter and can only contain lowercase letters, numbers, hyphens, underscores, and forward slashes. Forward slashes can create a folder-like structure to be used for permissions granting.

 - Controls whether the image referenced by a tag can change.

 The most reliable workloads set this to Immutable, and use a different tag for each version of the image published. This is relatively inflexible, setting should be based on business needs flexible vs reliable

 - Boolean, determines if security scanning should happen automatically on new image upload.

 AWS provides both “basic” scanning on native ECR, as well as “enhanced” scanning using the Inspector tool. More information here on the differences.

 Basic scanning is both free and incredibly useful. Definitely set this to true.

 - By default, all images are retained, which can quickly cost a great deal. A lifecycle policy automatically deletes images after XX amount of days, those with/without tags, or after a number of images is satisfied. More information here.

Use this graphic and write-up to upskill on AWS ECR super fast. You'll know enough to build containers and push them live in a few minutes

Cheat Sheet: AWS Elastic Container Registry (ECR)

S3 is Amazon’s general purpose storage. S3 “buckets” are able to provide authenticated access to files both within an AWS account and between AWS accounts, as well as unauthenticated access to files (e.g., client web access). S3 buckets are very versatile, and is often used as a part of other solutions, like a logging destination. However, because of the scope and scale of this service, the configuration and IAM policies that govern it are a little more complex than with other resources. Policies govern both resource access (S3 bucket globally) as well as file-level access (S3 “objects”).

Due to this maturity, S3 is one of the most commonly misconfigured services. Misconfiguration can lead to data loss or deletion. Let’s learn what we can configure on this resource type.

S3’s IAM resource policies govern who can access the bucket globally, folder paths with children, or specific files, and what rights they have, such as read, write, and delete.

. IAM Pulse hosts our own best-practice security policies 

Bucket names are always only lower-case and hyphens, e.g. “iam-pulse-great-bucket”

Bucket names are globally unique, so once we create “iam-pulse-great-bucket”, no one else in the world can create that same bucket, even in another region

To access a bucket via your web browser, use this URL pattern: https://(bucket-name-here).s3.amazonaws.com/(file-name-here). So for our bucket "iam-pulse-great-bucket) and file “my-file.txt”, we’d navigate to: 

https://iam-pulse-great-bucket.s3.amazonaws.com/my-file.txt

You can use URL pattern to test if your files are publicly accessible

Bucket policies attach only to buckets, not to S3 objects. These bucket policies do govern file access

S3 public access was initially very easy to do - so easy some large companies accidentally leaked data. To help, AWS added several layers that govern public access.

Block Public Access - This is a single bucket-wide config flag that blocks all public access regardless of the following settings. This can also be set account-wide

Access Point policies - These resources act as a bridge to an S3 bucket, and can have different IAM policies than the bucket policy, and are often used to permit an entire VPC to get to an S3 bucket

Bucket policies - This json IAM document governs who can access the bucket directly, as well as the file objects contains within, and what rights they have to this resource

Bucket ACLs - A legacy method of controlling bucket-wide access, this defaults to “private” where only the owner has control, and no one else is granted rights

Here are a few specific policy examples. There are lots more keys and strategies, please see our 

To permit public read-only access to your bucket root listing:

To block non-secure access to all files in your bucket:

To permit a specific user ARN to read and write to a specific folder:

WARNING - Be careful not to write a policy that revokes root access to a bucket. The IAM policy on these resources is entirely possible to block console root access, and TAC will be required to recover, which can take weeks.

IAM Pulse has a huge cache of policy docs that we host here: 

 (true/false) - Prevents any item in bucket from public exposure. If S3 bucket created in console, this is enabled by default. If created via API (including Terraform), this is disabled by default.

: Unless public, unauthenticated (think web requests from a browser) are desired, leave enabled. Intra-account and inter-account access can still work with this feature enabled.

: This feature was added to help avoid leaking data. If you don’t want unauthenticated web users to access this bucket, make sure this feature is set to true.

Access Point Policies (blank) - Can connect an S3 bucket to a private VPC endpoint, and grant special access to the bucket, bypassing Bucket Policies.

Bucket Policies (blank) - These policies govern access both to the bucket globally and to specific object file paths in the bucket. Please see the examples above or the IAM Pulse policy cache.

Access Control List (private/many options) - Canned bucket-wide ACLs for access. Grants or permits access to the bucket. All other access options over-ride this setting, and AWS no longer recommends using it.

: Leave set to default of private/disabled. Modern design uses S3 bucket policies, rather than bucket-wide ACLs.

) - Controls server-side encryption at rest. Often required for compliance. Real-world compromise highly unlikely, the server(s) would need to be stolen from a physical AWS data center to reconstruct your data.

: When enabling, you can select a KMS key to encrypt with. Either use “SSE-S3” to use S3’s own key service, or select “AWS KMS key” to use the KMS service. Under that, select either “AWS Managed Key”

If you are using this bucket for cross-account storage (like a shared logging bucket), using the S3 key is easiest. It’s still possible to use your own key for cross-account logging purposes, but it’ll require granting access to that key also. If you want that, use your own AWS KMS key:

For all other use cases, enable using “AWS Key Management Service key (SSE-KMS)” → “Choose from your AWS KMS keys”. This encrypts the entire bucket contents with a private key you control, and should prevent even AWS from accessing it.

) - Stores older copies of any data objects on modify or delete. Costs can add up quickly, but great option for critical data, or for change tracking/reverting bad changes.

: Enable if extra costs justify the ability to revert files. For large data sets this cost can add up quickly. You can use the 

) - Enables detailed logs on accessing of any data stored in bucket. Compliance often requires this.

: Always enable access logging, and send the data to an S3 bucket you’ve designed for the purpose. Use a unique logging folder (e.g., logging-bucket-name/your-bucket-name) to make logs easily sorted.

, time frame) - Prevents over-writing or modifying files once created. Permits a rolling schedule (xx days from creation) to unlock files for writing. Must be set at bucket create time or with TAC’s assistance after creation.

: Don’t enable unless immutable files are required.

) - Authenticated third-parties are billed for data transfer fees. Bucket owner still pays data storage costs.

: Leave disabled unless your use case requires.

Object Ownership (ACLs disabled/ACLs enabled) - Ownership controls billing and implicit permissions to files. If S3 bucket created in console, ACLs are disabled, which AWS recommends. If created via API (including Terraform), ACLs are enabled. If enabled, ACLs are disabled.

 (blank) - CORS, or Cross Origin Resource Sharing, is a web-security technology. 

: If you are using your S3 bucket directly for web site hosting, configure this policy.

Cheat Sheet: AWS S3

Terraform

Code Frameworks

This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures. You can do it!

I was brought into DevOps in a shop that heavily utilized SparkleFormation. Sparkle (or SFN) is a ruby-based tool that constructs CloudFormation stacks. It allows DevOps folks to write minimal configurations, and then run the SFN constructor tool against them, and SFN will read your defaults (security, logging, config, etc.) and output a valid CloudFormation stack.

This has its drawbacks — obviously CloudFormation is only valid in AWS-land, but additionally, the SFN DSL (Domain Specific Language), or syntax, is painful and hard to work with. There is some security through obscurity, but primarily this difficult language operates as a tax on your teams — not good.

Terraform solves lots of these problems in an intuitive, easy way. Terraform is multi-cloud and the concept of modules can replace much of SFN’s constructor processes and gains. However, there is one process I haven’t been able to easily replicate — 

In SFN, you can set a default list of Amazon Account IDs, and utilize this list to generate IAM policies, or any other kind of json/yaml-encoded policy, in an easy way. Terraform supports this, but not in an intuitive way.

Let’s build it together and you can start using this powerful pattern.

I won’t go too deep into what AWS IAM is here — suffice to say, they are security and access control documents that control who can do what, and what actions can be taken. They’re incredibly powerful, but also complex, which means automating and process-atizing them is a huge gain for your company and your dev teams.

The AWS terraform team has helpfully created some purpose-built resources that help us build IAM Json docs and utilize them. The first is a data source called 

). Data sources generally reach out to the provider to learn about the environment. That isn’t the case here — the AWS terraform team is mis-using a data source to construct a json IAM document, which is very cool:

You’re then able to use this data source to output a json-encoded version of your policy:

This works great, but it’s again, static. We want to be able to send a list of account IDs and have IAM build us complex policies. Let’s talk about how we can build some iterative, complex policies.

First, let’s establish a local data source of account IDs. You don’t need to have this locally — you could store this in a central data store, or have the parent pass this information to the module that’s building these IAM policies. We’re using a local data store as a simple way to demonstrate.

Okay, now we have a list of account IDs. Now let’s build our json-encoded policy document. This IAM policy will only have a single statement — permitting IAM access to an entire other account by granting access to the “root” user, which means any user in that other account. However, we want it to update automatically when that passed list updates.

To do that, we set the principal as type 

 and set an identifier (required when multiple values here) of the local list of account IDs. Note that we use a 

 loop to iterate over that list and encoded it in the expected format.

Below, we set the actions — permit ECR pulling, an AWS action for an ECR (Image Repo for Container Images).

Okay, now we have a valid IAM policy document, all ready to go. However, we haven’t build the policy yet — right now this all lives within Terraform, no real resources are built.

Thankfully, we’ve done the hard part — now we are able to simply reference that data source’s output of 

, which is a json encoded policy document.

And boom, we have a policy that is able to update itself when the passed values are updated.

IAM Resources — Iterative — Multiple Statements

This is of course a limited example — what if we wanted multiple statements based on these same values?

Well, we can do that also. Rather than a single iam policy doc, we use a 

 to build several of them, one each for the passed in account IDs. The SID must be unique for each statement or IAM complains, so we append the value, which of course should be unique.

We can’t directly use these json documents — after all, a Policy expects to be passed a single json document, not several! However, the AWS Terraform team is amazing, and built this modality into the same document, so we pass each document from above into a new copy of the same resource type, as a json entry.

This resource natively combines the json docs into a single valid policy document.

And then, just as before, we’re able to utilize the combined documents as input for our compiled policy.

This pattern is flexible, and allows us to build IAM policy docs in a “constructor” pattern, similar to how SparkleFormation and other constructors do. This makes life simpler and easier for our Devs, which makes our environments more secure.

I want to thank Jamie Phillips on the Hashicorp Ambassadors group for steering me towards this solution. Jamie has a great 

 on his website that goes over a similar pattern:

An intuitive, easy way to build IAM policy docs in a "constructor" pattern using Terraform.

Terraform Dynamic IAM Policy Construction

As we’ve scaled out our CI/CD (~175 pipelines across 75 accounts) to have many builders (~25 pools, ~50 or so builders), we’ve seen pain points where the way we were doing it before just wasn’t cutting it. I’ve written extensively about how graphical, manually-managed pipelines just couldn’t scale beyond about 50 — I’d spend literally all day every day just updating Terraform versions, and double-checking store accounts, and no one wants that job.

We eventually wrote out our pipelines in YAML, and now manage them via pull requests in a git repo. That permits variables (to conclusively match storage accounts and account IDs among steps in a pipeline), and also mass updates — find/replace for terraform versions. That has been a roaring success, and we’ve adopted that model all over.

This blog focuses on our builders — as we've scaled out our AWS accounts, we've been registering a single small Amazon Linux host to be a runner in each account. That is a boon to authentication security — it can use an assumed IAM role within the account to manage it, rather than authenticating with a static IAM user, but a pain for management and inter-pipeline security. Let’s talk about each separately.

First, management. These are long-lived ec2 hosts that need to be monitored, rebooted occasionally, and patched. That overhead is an anti-pattern for the containerized, serverless models of modern applications. So it's gotta go.

Second, inter-pipeline security. Pipelines in CI/CDs often have access to privileged information — they connect to secrets stores like SSM or Vault, sometimes copying those secrets to the local disk when doing compute or ETL operations. Now image those artifacts aren’t cleaned up, and a second malicious pipeline runs on the same builder and uploads every file it can get access to to somewhere else. That's not great. There are mitigations you can build in, like each pipeline cleaning its artifacts, or making sure all teams running jobs don't store secrets on disk, but that’s very much a herding-cats model. It's gotta go.

I spent a lot of time mapping out a new model with another architect, 

, and figuring out what our goals are. Here’s what we need to satisfy:

Container or serverless driven, no servers! Containers have myriad benefits, including easy relaunch on issues, and can be frequently rebuilt to include security patches

Runners should run a single job only, and then die, so no risk of leaking secrets between jobs

Centralized repository and secret access, so there isn’t a continued need to update secrets and docker image in many locations

Easy to deploy, with a robust, standardized Terraform module

New Model: Centralized ECR and Secrets, ECS Runners

We have to solve this in both the Azure and AWS space for business reasons. I know the AWS side better, so I own that. I elected to create a “Hub” account that stores all the non-duplicated items (ECR, secret), and have many “spoke” accounts that will store their own ECS service, task definition, and scheduling.

In the single "Hub" account, I created several resources:

 to store the builder container image that spoke accounts will pull, and created an ECR policy to permit inter-account access

"" that I populated with the "PAT" token (used to authenticate to Azure DevOps (ADO) and register as a builder and a secret policy permitting inter-account access

, a customer-created and managed encryption key, which is used to encrypt the secret, and also a KMS policy which permits "spoke" accounts to access this CMK so they can decrypt the secrets manager secret

In each runner "Spoke" account we call a terraform module that creates:

 — Most AWS services optionally use logs, and need somewhere to store them

 (and policies) used to pull the ECR image and access secrets

 — Contains almost no config, but defines underlying hardware type, here "FARGATE"

 that defines how large the runner and underlying Fargate host should be, as well as container location and environment (and secret) information

 number of the above ECS task and handle replacing it after it dies

 to link autoscaling policies with the ECS service

 to spin up more runners during business hours when they are most active, and fewer after-hours, and also 0 out the runner pool each night to trigger mandatory container redeployments each morning — I’ll talk about the logic here more later in this blog

AWS has a much-maligned, and very finicky security model. It’s incredibly powerful, but also so complex it’s sometimes painful to work with.

The model basically works from both sides of access:

 a request must have an IAM policy permitting it to receive that request from the sending resource

 the request must have a policy that permits it to send the request to the target

Keep that in mind as I show the specific config below, and we’ll talk about both sides.

There’s also an interesting catch-22 here — whenever you update an IAM policy, it checks to make sure the “principal” (resource) exists. This means that you can’t proactively create the hub policy before creating the spoke IAM resources, because the hub IAM policies will say the spoke IAM roles don’t exist, so the principal is invalid. This means we need to follow an interesting deployment strategy:

First, deploy the Hub resources without policies. These need to exist so spoke IAM can be created

Second, deploy the IAM execution roles in each Spoke account, with policies to permit access to the Hub resources

Third, deploy policies in the Hub account to permit spoke access

Fourth, deploy the ECS service and task in the Spoke accounts, which should now spin up and succeed

This is a huge bummer in terms of parallel deployment — if your account management is account-based, like ours is, you’ll have issues. If you’re managing your multiple accounts with Terraform workspaces (or CloudFormation?), you could stage the deployment, and manually do each of the four steps in the proper order.

I’ll walk through the resources and policies together, but keep in mind the deployment steps above for when you build it yourself.

I’ll write another blog for how we designed, built, and tested the docker image. For now, assume the docker image works well, and accepts the following 3 environmental variables to configure itself:

 — The location of your Azure DevOps Org, e.g. 

 — The name of the existing pool on Azure DevOps to register into. NOTE: Make sure this pool exists before attempting to programmatically register to it — hosts can’t create a pool when they register.

 — a secret token permitting hosts to register to builder pools. This token is created under a specific administrator user (or service principal) and expires periodically. 

First, we need somewhere to store our docker image file. AWS permits only a single image to be stored in a single ECR — I like this model, compared with Azure’s “many images in a single ACR.” ECRs have a name, and not much else. I recommend enabling “scan_on_push” to get AWS’s built-in image security scanning and reporting.

Mutability is a topic of much discussion among the docker community. Flexibility-heavy orgs leave MUTABLE on, and this permits over-writing tags with a new version that should be used. Structured, or security-heavy orgs, turn off mutability and often use image tags to track versions, and once an image tag is registered it can’t be over-written. Totally up to you and your org!

We also need an ECR policy that permits the spoke accounts to get to this image. You can either grant access to 

 which grants access to any resource in an account to get to it, or you can grant access to something like 

, which grants access only to that specific IAM role. The image source I don’t think is very sensitive, so I use the 

 method here, but the secret and CMK policy next are very sensitive, so we are more specific there.

Next we want to create a secret and permit sharing it between accounts. However, secrets are encrypted with a key so not even AWS can decrypt them. This is a great security model, but means we need to share the encryption key between accounts first before we share the secret, or the spoke accounts will get to the secret but not be able to decrypt it. So first let’s talk KMS CMK.

KMS keys are created as a distinct resource, and their policies permit not even granting the hosting account access to the key, which isn't ideal. So first we grant access to 

 is the account ID of the Hub account. This permits the console and APIs to manage the key, something we expect and want.

The second stanza permits a specific IAM role in a spoke account to get to this resource. Remember, the IAM role must already exist in the spoke account, or this policy will be rejected by the AWS API.

Next, we create a KMS alias. This isn’t strictly required, but it’s helpful for the humans to keep track of keys and give them aliases (read: names).

Now that the KMS key is shared with the spoke account, they can decrypt stuff that is encrypted with it, like the Hub’s secret! Let’s create that secret. Note that need to call out the KMS key used to encrypt this secret, or it’ll be encrypted with a different KMS key that hasn’t been shared.

Also note that the actual PAK secret isn't provided or in terraform at all. Once this secret is created, you’d login to the AWS console by hand and populate this secret.

Open the AWS console, and find the secret there. Scroll down to "retrieve secret value" and click it. It’ll show default values.

Update the default value to the PAK you’d like your builders to use and hit save. Make sure to use the “Plaintext” type of secret, rather than json-based key/value type.

Now that our key is created and populated, we need to attach a secret policy to permit multi-account access. We use the same IAM spoke role as above.

And that's it for the Hub account — now we can work on Spoke accounts, which is where the cool stuff lives anyway. Let’s walk through the resources required there.

All the items above we’ll build in only one account, the Hub. All of the following resources we’ll build in every single “runner” or “Spoke” account. These spoke accounts will draw on the ECR, secret, and KMS CMK from the Hub account.

Every AWS resource supports extensive logging, but it often is disabled by default. For ECS services and tasks to log, we need to create a cloudwatch log group to receive the logs. No policies here — most access is within the spoke account, which will work by default.

An ECS task ties together all sorts of stuff — a container definition, IAM roles to use to access required resources, sizing, and logging. The first one we’ll build is the "Execution IAM Role" — this is used when the ECS task is launched and needs to have the ability to access all required resources.

This is also the IAM role that you'll need to permit access in the Hub account above, since it’s the resource that’ll be doing the requesting.

To permit most ECS tasks, like launching the ECS, writing logs, etc., AWS provides an IAM role that we can use and attach to. So let’s do that:

We also want to permit this requesting IAM role to get to the Hub’s secret and the KMS CMK key it is encrypted with.

Note that we don’t need to grant access to the Hub’s ECR since the above AmazonECSTaskExecutionRolePolicy already grants it.

The ECS cluster doesn’t do much — it defines who will provide the compute for any tasks or services assigned to it, and we say we’d prefer FARGATE, where AWS provides the compute for us.

Kyler Middleton

Followed Topics

Published Articles