Remediating AWS IMDSv1

- 15 minutes read - 3161 words

Compute resources in AWS (e.g. EC2 instances, ECS tasks/services, etc.) get access to AWS credentials, such as temporary instance role credentials, via the Instance Metadata Service (IMDS) . The compute resources use these credentials to access other AWS services such as SQS, DynamoDB and Secrets Manager.

Introduction: Problems with IMDSv1

There was originally only one version of IMDS, now called “v1,” which unfortunately many people still use. The technical risks and high profile incidents (the Capital One breach comes to mind) associated with v1, as well as the existence of v2 are well-documented. When an application hosted on an EC2 instance is vulnerable to SSRF , XXE or RCE , attackers can likely steal the temporary AWS credentials of the IAM role configured for the instance. This service is a particularly interesting target for attackers:

  1. It’s predictably there, unlike traditional targets like memcached or Redis.

  2. It has a trivial HTTP API, making it easier to exploit than services where you may need to smuggle a protocol inside another protocol. (Downside: because it’s read-only, you can’t do much with a blind SSRF attack.)

  3. It likely has the most interesting access anyway. Many compute resources are single-purpose: it is generally less valuable for an attacker to get elevated access within that compute instance than to get access to data or gain persistence for later access. You can do both more consistently with IAM access than local privilege escalation.

The usual classical targets for SSRF (memcached, Redis and the like) are also valuable targets, but often require a lot more fiddling. It’s hard not to appreciate the simple, ruthless effectiveness of hitting IMDSv1 instead.

Defending against this is a great example of defense-in-depth. On the one hand: SSRF, XXE and RCE are all serious bugs that your application should already be audited for. SSRF in particular is broadly underestimated by developers who aren’t familiar with sophisticated techniques (e.g. DNS rebinding, URL parser bugs, and ridiculously pernicious bugs leveraging TLS session resumption, like those then-Latacoran Joshua Maddux found) . On the other hand: it’s still valuable to mitigate the damage of one of those attacks should they happen, and allowing access to IMDSv1 creates an environment where a successful exploitation of an application vulnerability can be used to pivot into attacking the underlying infrastructure.

Abusing v1, and how v2 is different

The following figure illustrates a typical attack leveraging an application-level vulnerability to steal temporary credentials and use them to access private S3 buckets:

Figure of a hooded-attacker accessing private S3 buckets by abusing IMDSv1

IMDSv2 fixes this issue by making the user get and then repeat a value. You first get a token from the http://169.254.169.254/latest/api/token endpoint via PUT. Then, you repeat that token in every other request to IMDS via the x-aws-ec2-metadata-token header.

This may sound similar to a lot of CSRF mitigation techniques. There are a few parallels! SSRF, like CSRF, is fundamentally a problem of tricking a system into abusing its standing permissions. In CSRF, that standing permission is usually a session cookie. In SSRF that standing permission is usually privileged network access: services running on localhost, link-local, or a private network, accessible to the server but not the attacker, usually with no credentials. CSRF can similarly be used to attack local services, like a web server in your antivirus that helpfully runs arbitrary commands .

Let’s analyze what this protects against and what it leaves open. A lot of SSRF is “blind” (meaning the attacker can make requests but not see the response) and somewhat limited (e.g. only GET, no ability to modify headers). A lot of sophisticated attacks, like protocol smuggling attacks , rely on the attacker-controlled parts of the request (like SNI and TLS session tickets) to get around those limitations. IMDSv2 leverages these properties to limit exploitation: you’d need to be able to make PUT requests, see the results, and set an arbitrary header pair. While it’s possible that a particularly heinous SSRF vulnerability would still allow an attacker to exploit IMDSv2, v2 effectively prevents real-world exploitation.

XXE has similar limitations. Compared to SSRF, it’s less commonly blind but more commonly restricted in terms of headers, methods and the like. IMDSv2 hence similarly prevents exploitation via XXE.

Finally, of course, there’s little you can do against RCE. Anything you can do, attackers can do too.

The rest of this post focuses on implementing an effective remediation strategy, which ideally includes requiring the use of IMDSv2, but also provides guidance for limiting the exposure to IMDSv1 where it is required.

Detection

CLI

The EC2 describe-instances CLI call identifies instances that allow access to IMDSv1. When the metadata-options.http-tokens parameter is set to optional, then calls can be made to both the v1 and v2 versions. Setting this parameter to required enforces the use of the v2 endpoint.

For example, the following command retrieves the instance ID of all instances in the us-east-1 region that allow calls to IMDSv1:

aws ec2 describe-instances \
    --filters Name=metadata-options.http-tokens,Values=optional \
    --query 'Reservations[*].Instances[*].{Instance:InstanceId}' \
    --region us-east-1

[
  [
    {
      "Instance": "i-1234567890"
    }
  ]
]

Remediate AWS IMDSv1 Tool

To identify all the instances with IMDSv1 enabled within a given AWS account, the Remediate AWS IMDSv1 tool can be used:

python remediate-imdsv1.py --profile ${profile_name} --debug

In the below output, the instance with the ID i-1234567890 in the us-east-1 region is identified as having IMDSv1 enabled:

2021-05-31 16:46:51 w remediate-imdsv1[12107] INFO Starting
2021-05-31 16:46:51 w remediate-imdsv1[12107] INFO Identifying instances
2021-05-31 16:46:51 w remediate-imdsv1[12107] DEBUG Running against region us-west-2
2021-05-31 16:46:51 w remediate-imdsv1[12107] DEBUG Running against region eu-north-1
2021-05-31 16:46:52 w remediate-imdsv1[12107] DEBUG Running against region ap-south-1
2021-05-31 16:46:53 w remediate-imdsv1[12107] DEBUG Running against region eu-west-3
2021-05-31 16:46:53 w remediate-imdsv1[12107] DEBUG Running against region eu-west-2
2021-05-31 16:46:53 w remediate-imdsv1[12107] DEBUG Running against region eu-west-1
2021-05-31 16:46:53 w remediate-imdsv1[12107] DEBUG Running against region ap-northeast-3
2021-05-31 16:46:55 w remediate-imdsv1[12107] DEBUG Running against region ap-northeast-2
2021-05-31 16:46:56 w remediate-imdsv1[12107] DEBUG Running against region ap-northeast-1
2021-05-31 16:46:57 w remediate-imdsv1[12107] DEBUG Running against region sa-east-1
2021-05-31 16:46:59 w remediate-imdsv1[12107] DEBUG Running against region ca-central-1
2021-05-31 16:46:59 w remediate-imdsv1[12107] DEBUG Running against region ap-southeast-1
2021-05-31 16:47:00 w remediate-imdsv1[12107] DEBUG Running against region ap-southeast-2
2021-05-31 16:47:02 w remediate-imdsv1[12107] DEBUG Running against region eu-central-1
2021-05-31 16:47:02 w remediate-imdsv1[12107] DEBUG Running against region us-east-1
2021-05-31 16:47:02 w remediate-imdsv1[12107] DEBUG Identified arn:aws:ec2:us-east-1:account:instance/i-1234567890
2021-05-31 16:47:03 w remediate-imdsv1[12107] DEBUG Running against region us-east-2
2021-05-31 16:47:04 w remediate-imdsv1[12107] DEBUG Running against region us-west-1
2021-05-31 16:47:05 w remediate-imdsv1[12107] DEBUG Running against region us-west-2
2021-05-31 16:47:09 w remediate-imdsv1[12107] INFO Done

At Latacora, we do something similar. We already have tooling that captures our clients’ entire AWS environments on a daily basis. IMDS checks are just one of the checks built on top of that tooling.

CloudWatch

While the above CLI call and tool allow identifying resources with IMDSv1 enabled, the first step to implementing a remediation strategy is to assess whether the v1 endpoint is actually in use. The CloudWatch MetadataNoToken metric identifies the number of times IMDSv1 was successfully accessed by specific instances within a given timeframe.

Image of AWS web console IMDS CloudWatch metrics

As shown in the above figure, the i-0df97f0f9473909f1 instance made a number of calls to IMDSv1 in the last hour.

This information will allow defining the path forward:

Remediation

CLI

For existing instances, the EC2 modify-instance-metadata-options CLI call allows enforcing the use of IMDSv2:

aws ec2 modify-instance-metadata-options \
    --http-tokens required \
    --instance-id ${instance_id}

Note that this CLI call is only available in the AWS CLI v2 .

This command can also be run in batch via AWS Systems Manager run commands. For example, refer to this blog post by Michael Scovell . Launching new instances with the EC2 run-instances CLI call allows enforcing the use of IMDSv2 via the --metadata-options parameter:

aws ec2 run-instances \
    --image-id ami-0abcdef1234567890 \
    --metadata-options 'HttpTokens=required' \
    ...

Remediate AWS IMDSv1 Tool

To remediate all the instances with IMDSv1 enabled within a given AWS account, the Remediate AWS IMDSv1 tool can be used:

python remediate-imdsv1.py --profile ${profile_name} --debug

2021-05-31 16:46:51 w remediate-imdsv1[12107] INFO Starting
2021-05-31 16:46:51 w remediate-imdsv1[12107] INFO Identifying instances
2021-05-31 16:46:51 w remediate-imdsv1[12107] DEBUG Running against region us-west-2
2021-05-31 16:46:51 w remediate-imdsv1[12107] DEBUG Running against region eu-north-1
2021-05-31 16:46:52 w remediate-imdsv1[12107] DEBUG Running against region ap-south-1
2021-05-31 16:46:53 w remediate-imdsv1[12107] DEBUG Running against region eu-west-3
2021-05-31 16:46:53 w remediate-imdsv1[12107] DEBUG Running against region eu-west-2
2021-05-31 16:46:53 w remediate-imdsv1[12107] DEBUG Running against region eu-west-1
2021-05-31 16:46:53 w remediate-imdsv1[12107] DEBUG Running against region ap-northeast-3
2021-05-31 16:46:55 w remediate-imdsv1[12107] DEBUG Running against region ap-northeast-2
2021-05-31 16:46:56 w remediate-imdsv1[12107] DEBUG Running against region ap-northeast-1
2021-05-31 16:46:57 w remediate-imdsv1[12107] DEBUG Running against region sa-east-1
2021-05-31 16:46:59 w remediate-imdsv1[12107] DEBUG Running against region ca-central-1
2021-05-31 16:46:59 w remediate-imdsv1[12107] DEBUG Running against region ap-southeast-1
2021-05-31 16:47:00 w remediate-imdsv1[12107] DEBUG Running against region ap-southeast-2
2021-05-31 16:47:02 w remediate-imdsv1[12107] DEBUG Running against region eu-central-1
2021-05-31 16:47:02 w remediate-imdsv1[12107] DEBUG Running against region us-east-1
2021-05-31 16:47:02 w remediate-imdsv1[12107] DEBUG Identified arn:aws:ec2:us-east-1:account:instance/i-1234567890
2021-05-31 16:47:03 w remediate-imdsv1[12107] DEBUG Running against region us-east-2
2021-05-31 16:47:04 w remediate-imdsv1[12107] DEBUG Running against region us-west-1
2021-05-31 16:47:05 w remediate-imdsv1[12107] DEBUG Running against region us-west-2
2021-05-31 16:47:09 w remediate-imdsv1[12107] INFO Done

Infrastructure as Code (IaC)

Where IaC is used to deploy AWS resources, the templates should be updated to ensure all resources enforce IMDSv2.

CloudFormation

Several CloudFormation entities allow disabling IMDSv1, such as:

Unfortunately, support for all affected resources is incomplete at the time of writing.

Terraform

Terraform allows configuring metadata options for EC2 instance, e.g.:

provider "aws" {
  region = "us-east-1"
}

resource "aws_instance" "test_instance" {
  ami = "ami-..."
  ...
  metadata_options {
    http_tokens = "required" # enforces the use of IMDSv2
    ...
  }
}

Note that this configuration is also available in EC2 auto scaling launch templates and launch configurations .

Guardrails

Detecting and remediating are the first steps to rolling out security improvements. The next maturity stage is to prevent these issues from happening in the first place. Sometimes, that takes the shape of advanced auditing and monitoring. Sometimes you can prevent unsafe configurations altogether. Since IMDSv1 has no upsides except legacy compatibility, disabling it entirely might be a viable option. A counterexample might be public S3 buckets: they definitely need to be audited, but are often intentional and warranted, so you can’t simply disable the feature.

IAM Policies & Organizations Service Control Policies (SCPs)

IAM policies can be used to both disallow launching new instances with v1 enabled, as well as from making calls against the v1 endpoint from roles assigned to EC2 instances. Refer to the AWS documentation for additional details. An SCP is kind of like an IAM policy, but attached via AWS Organizations, limiting what an entire AWS account can do.

The following policy specifies that the RunInstances API cannot be called unless the instance is also opted in to require the use of IMDSv2:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RequireImdsV2",
      "Effect": "Deny",
      "Action": "ec2:RunInstances",
      "Resource": "arn:aws:ec2:*:*:instance/*",
      "Condition": {
        "StringNotEquals": {
          "ec2:MetadataHttpTokens": "required"
        }
      }
    }
  ]
}

The following policy permits only users with the ec2-imds-admins role to make changes to EC2 instances’ metadata options:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowOnlyImdsAdminsToModifySettings",
      "Effect": "Deny",
      "Action": "ec2:ModifyInstanceMetadataOptions",
      "Resource": "*",
      "Condition": {
        "StringNotLike": {
          "aws:PrincipalARN": "arn:aws:iam::*:role/ec2-imds-admins"
        }
      }
    }
  ]
}

The above statement can be used to control the use of the ModifyInstanceMetadataOptions API and ensure IAM users can’t modify running instances to re-enable IMDSv1. This is useful as there are currently no fine-grained access controls (conditions) for the ModifyInstanceMetadataOptions API.

The following policy specifies that, if this policy is applied to a role and the role is assumed by the EC2 service, requests made to IMDS must be made to the v2:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RequireAllEc2RolesToUseV2",
      "Effect": "Deny",
      "Action": "*",
      "Resource": "*",
      "Condition": {
        "NumericLessThan": {
          "ec2:RoleDelivery": "2.0"
        }
      }
    }
  ]
}

The above statement/policy can be applied generally because, if the request is not signed by EC2 role credentials, it has no effect.

This policy can be applied to an AWS Organization as an SCP, to ensure that in no cases can successful requests be made against IMDSv1. Where IMDSv1 is not required within an Organization or OU, this is the silver bullet.

Static Application Security Testing (SAST)

A number of IaC security scanners implement rules to detect IMDS misconfigurations. For example, the following Semgrep rule, which is part of the terraform collection , can be used to identify IMDSv1 (sandbox ):

---
rules:
  - id: ec2-imdsv1-optional
    languages:
      - generic
    message: |
      AWS EC2 Instance allowing use of IMDSv1      
    paths:
      include:
        - "*.tf"
    patterns:
      - pattern: http_tokens = "optional"
      - pattern-inside: "{ ... metadata_options { ... } ... }"
    severity: ERROR

Tools such as Semgrep can easily be integrated in CI/CD pipelines, for example via GitHub Actions .

Additional rules:

Automated Detection

A number of mechanisms can be leveraged to detect the use of IMDSv1, as well as the presence of misconfigured resources. For example:

Automated Remediation

A number of mechanisms can be leveraged to automatically remediate misconfigured resources. For example:

Additional considerations

Disabling IMDS for Instances

An additional HttpEndpoint configuration option exists, which can allow completely disabling IMDS for given instances. For example, the following EC2 modify-instance-metadata-options CLI call will disable IMDS altogether:

aws ec2 modify-instance-metadata-options \
    --http-endpoint disabled \
    --instance-id ${instance_id}

Use this option where instances have no roles attached.

Leveraging IP Deny Lists

Local firewall rules can be configured to disable access from some or all processes to IMDS. For example, the following prevents access to IMDS by all processes, except for those running in the user account trustworthy-user:

sudo iptables \
    --append OUTPUT \
    --proto tcp \
    --destination 169.254.169.254 \
    --match owner ! \
    --uid-owner trustworthy-user \
    --jump REJECT

Refer to the following for additional information:

Preventing EKS Pods from Accessing EC2 IMDS

Working in conjunction with the HttpTokens configuration option, the HttpPutResponseHopLimit option allows enforcing a response hop limit for IMDS requests. The goal of this is to help protect against open layer 3 firewalls and NATs.

By default, EKS pods have the ability to access IMDS endpoint of their underlying EC2 cluster instance. Consequently, an attacker that is able to run commands in a pod may be able to query the metadata service and access temporary credentials. Pods running on EC2 worker nodes can inherit the rights of the instance profile assigned to the worker node if they are able to access IMDS endpoint.

Requiring IMDSv2 and configuring the response hop limit on the EC2 instance to 1 will limit access to IMDS from the EC2 instance itself and deny access to IMDS endpoint from within pods.

In cases where pods require access to other AWS resources, the preferred approach is to use IAM Roles for Service Accounts (IRSA) , and grant privileges to the associated roles using standard IAM policies and the principle of least privilege. When configured, the Kubernetes API will mount a signed OIDC token, issued by the cluster OIDC endpoint, and mount it as a volume in any pods associated with the Kubernetes ServiceAccount associated with a given IAM role. Pods may then call sts:AssumeRoleWithWebIdentity and receive a temporary AWS role credential that grants the necessary access to interact with their associated AWS resources.

It is also possible to block a pod access to IMDS by manipulating iptables on the node, however, this can be a more cumbersome solution as node replacements which occur as part of an upgrade may not persist state, requiring a process to ensure the same firewall rules are applied across the required nodes.

Finally, if an application is using an older version of the AWS SDK that doesn’t support IAM Roles for Service Accounts (IRSA), and is running on a pod that will intentionally inherit the role of its EC2 instance node, it is possible to selectively allow access to EC2 metadata through the use of Kubernetes network policies. Note that this is also not a preferred solution as any change in privileges assigned to the node role will then propagate to the pods granted access to IMDS.

Refer to the following for additional information:

Preventing ECS Tasks from Accessing EC2 IMDS

Similar to EKS pods, ECS tasks have the ability to access IMDS of their underlying EC2 instances. While it’s possible to prevent containers from accessing the instance metadata using the bridge and awsvpc networking modes, it’s not possible to prevent access with the host networking mode, because the Amazon ECS agent runs on the host networking namespace and requires access to it.

Conclusion

In summary:

State diagram of IMDSv1 remediation flow

While IMDSv1 may appear like an easy problem to solve (just force everyone to use v2!), the reality is that for cloud providers managing technical debt is an impossible challenge to solve in the short term. Consequently, it’s the users’ shared responsibility to ensure that adequate security features are enabled, tailored to the deployed resources.

Further reading