We removed a pull_request_target-based preview workflow from our documentation repository after a responsible disclosure from security researcher Aviv Donenfeld. This was the last remaining instance of this pattern in our GitHub organization. The issue was limited to pull request preview environments for this repository, there is no indication it was ever exploited, and the overall impact was minimal.

One of our long-standing best practices is requiring maintainer approval before running workflows triggered by pull requests. That eliminates an entire class of CI/CD attack vectors involving untrusted code execution.

It is easy to assume that protection applies universally, including to workflows triggered by pull_request_target. That assumption breaks down when those workflows execute code, scripts, or artifacts influenced by an untrusted pull request.

Workflows triggered by pull_request_target run in the context of the base repository. That is not inherently unsafe if all executed code comes strictly from a trusted and protected branch. The problem is when such workflows execute code influenced by an untrusted pull request, or publish pull request content into a trusted location or environment. In that case, they can bypass the assumptions many teams make about pull request approval gates and create an avoidable CI/CD exposure.

What we changed​

Following Aviv's report, we removed the affected pattern from this repository's pull request preview workflow.

This was the last remaining instance of this pattern in our GitHub organization. We had already phased it out elsewhere some time ago, and it has now been fully removed here as well.

Scope and impact​

The issue was limited to preview environments for this documentation repository.

• There is no indication the issue was ever exploited
• We do not believe this had broader production impact
• The overall severity was low, but the pattern itself was still worth eliminating

Why this matters​

This is one of those edge cases that is easy to miss because the workflow may still look "approved" or "maintainer-controlled" at first glance. The security boundary is different for pull_request_target, and that difference matters when the workflow executes anything influenced by untrusted pull request content.

The practical takeaway is simple: pull_request_target can be a convenient choice for labeling pull requests. Beyond that, it should be used only when all executed code is strictly from a trusted and protected branch. Do not use it to execute code influenced by an untrusted pull request, or to publish pull request content into a trusted location or environment.

Guidance for customers and the community​

Because this repository is our documentation site, we are also using this as an opportunity to point customers and the community to the upstream guidance we recommend reviewing:

• GitHub Actions: pull_request_target event
• GitHub Actions security hardening
• GitHub Security Lab: Preventing pwn requests

We appreciate researchers like Aviv who take the time to surface issues like this clearly and responsibly. In this case, the report was thorough, accurate, and directly actionable.

We've replaced the monolithic account and account-settings components with six modular, single-responsibility components that give you a more granular approach to managing your AWS Organization.

What's Changing​

The old account component handled everything: the AWS Organization, organizational units, individual accounts, and service control policies — all in one. Similarly, account-settings bundled account configuration with budget management. We've broken these apart into focused components that each do one thing well.

Why Modular Components​

The New Components​

Part of Reference Architecture V2​

These new components are part of the broader Reference Architecture V2 changes, which also include the deprecation of account-map in favor of static configuration and Atmos Auth profiles. Together, these changes reduce Terraform dependencies, simplify cold starts, and better support brownfield deployments.

Adopting with Legacy Infrastructure​

provider "aws" {
  region = var.region
}

Exclude the upstream providers.tf and use the legacy mixin instead:

apiVersion: atmos/v1
kind: ComponentVendorConfig
spec:
  source:
    uri: github.com/cloudposse-terraform-components/aws-<component>.git//src?ref={{ .Version }}
    version: vX.x.x  # The new version with next-gen providers
    included_paths:
      - "**/**"
    excluded_paths:
      - "providers.tf"  # Exclude the next-gen providers.tf
  mixins:
    # Vendor in the legacy providers.tf that works with account-map
    - uri: https://raw.githubusercontent.com/cloudposse-terraform-components/mixins/{{ .Version }}/src/mixins/provider-with-account-map.tf
      version: v0.3.2
      filename: providers.tf

This gives you the new modular component logic while keeping your existing provider configuration intact.

Questions?​

If you have questions about adopting the new account components:

Over the past couple of months, we've shipped two changes that fundamentally simplify how components authenticate with AWS: Atmos Auth and the deprecation of account-map. This post is a migration guide for updating your component providers to match the version of infrastructure you're running — whether you're moving to next-gen or staying on legacy.

Which Guide Do You Need?​

Not sure which you are? Read the How to Tell Which Generation You're On section first.

Why We Made These Changes​

The driving force behind these changes is testability. The legacy account-map component was a globally required dependency — every component's providers.tf referenced it to resolve IAM roles. That tight coupling meant you couldn't test a single component in isolation without first deploying account-map and its entire dependency chain. By removing account-map from the default providers.tf, components become self-contained and significantly simpler to test.

Atmos Auth reinforces this by moving authentication out of the Terraform layer entirely. Instead of chained role assumptions wired through provider configuration, Atmos Auth resolves credentials before Terraform ever runs. The result is a provider block with nothing but region = var.region — no dynamic lookups, no remote state, no implicit dependencies on other components.

The trade-off is that this is a breaking change. Components that ship with the new providers.tf are not backwards compatible with environments still relying on account-map and team roles. That's exactly why we provide provider override mechanisms (described below) — they let you adopt the new authentication model at your own pace, on any component version, without waiting for upstream releases.

What Changed​

Two major improvements landed recently:

Together, these remove the need for account-map, aws-teams, and aws-team-roles. If you missed the announcement, see Reference Architecture v2: Deprecating Account-Map.

What "Generation" Means​

The generation is determined entirely by how providers.tf authenticates — not by component version numbers, module logic, or Terraform state format. A component's business logic (resources, variables, outputs) is unaffected by this change. The only thing that changes is the provider configuration and how credentials are resolved.

How to Tell Which Generation You're On​

The quickest way to tell is to look at your component's providers.tf. The majority of components use the module.iam_roles pattern described below. However, some cold start components (like account, account-map, account-settings, tfstate-backend) have always used a simple provider "aws" { region = var.region } — that's by design, because they run under your super admin profile before IAM roles infrastructure exists. Don't confuse a cold start component's simple provider with the next-gen pattern.

Legacy (Account-Map + Team Roles)​

If your providers.tf looks like this, you're on the older generation:

provider "aws" {
  region = var.region

  # Profile is deprecated in favor of terraform_role_arn. When profiles are not in use, terraform_profile_name is null.
  profile = module.iam_roles.terraform_profile_name

  dynamic "assume_role" {
    # module.iam_roles.terraform_role_arn may be null, in which case do not assume a role.
    for_each = compact([module.iam_roles.terraform_role_arn])
    content {
      role_arn = assume_role.value
    }
  }
}

module "iam_roles" {
  source  = "../account-map/modules/iam-roles"
  context = module.this.context
}

This pattern depends on account-map, aws-teams, and aws-team-roles being deployed. The iam_roles module reaches into the account-map remote state to figure out which role to assume.

Next-Gen (Atmos Auth)​

With Atmos Auth, authentication happens before Terraform runs — no dynamic role assumptions, no remote state lookups, no account-map dependency. The next-gen providers.tf includes the account_map variable (a static map of account names to IDs) and a dummy iam_roles module for backward compatibility:

variable "account_map_enabled" {
  type        = bool
  description = "Enable the account map component"
  default     = false
}

variable "account_map" {
  type = object({
    full_account_map              = map(string)
    audit_account_account_name    = optional(string, "")
    root_account_account_name     = optional(string, "")
    identity_account_account_name = optional(string, "")
    aws_partition                 = optional(string, "aws")
    iam_role_arn_templates        = optional(map(string), {})
  })
  description = "Map of account names to account IDs."
  default = {
    full_account_map              = {}
    audit_account_account_name    = ""
    root_account_account_name     = ""
    identity_account_account_name = ""
    aws_partition                 = "aws"
    iam_role_arn_templates        = {}
  }
}

provider "aws" {
  region = var.region
}

# Dummy module to satisfy legacy references to module.iam_roles
module "iam_roles" {
  source  = "cloudposse/label/null"
  context = module.this.context
}

Why the Dummy iam_roles Module?​

Many components reference module.iam_roles in their code — for example, to pass module.iam_roles.terraform_role_arn to sub-providers. The dummy module (sourced from cloudposse/label/null) satisfies Terraform's module reference validation so these components don't error during terraform init. It outputs empty/null values, which means any dynamic "assume_role" blocks that iterate over compact([module.iam_roles.terraform_role_arn]) simply produce zero iterations — no role is assumed, and Atmos Auth's pre-configured credentials are used instead.

Atmos Auth Prerequisites​

Before using Atmos Auth, ensure you have:

How Provider Overrides Work​

Both migration guides below rely on overriding a component's providers.tf so it matches your infrastructure. There are two mechanisms for this:

Recommended: atmos generate​

The terraform.generate block in your stack configuration tells Atmos to write files into the component directory before Terraform runs. Define it once at any stack level (org, tenant, stage) and every component that inherits from that config gets the generated providers.tf automatically. This is the recommended approach because it's centrally managed, inheritable, and doesn't require per-component changes.

Alternative: Vendor Mixins​

We also publish Atmos vendor mixins that can be added to individual component.yaml files. This is useful when you need per-component control or can't use atmos generate for a specific component. The two relevant mixins are:

provider-without-account-map.tf — The next-gen provider. Defines the account_map variables, configures provider "aws" with just region = var.region, and includes a dummy iam_roles module for backward compatibility.

provider-with-account-map.tf — The legacy provider. Restores the module.iam_roles pattern for components that need to run on infrastructure still using account-map.

Account Verification​

Regardless of which override method you use, we recommend adding the account-verification.mixin.tf mixin. This uses an aws_caller_identity data source to check the account ID of the credentials Terraform is running with, then compares it against the expected account from the account_map variable. The check runs during terraform plan — if there's a mismatch, Terraform fails before making any changes. This catches misconfigured Atmos Auth profiles, stale SSO sessions, or wrong environment targets.

Example failure output:

Error: Account verification failed
  Expected account "567890123456" (plat-dev) but authenticated to "789012345678" (plat-prod).
  Check your Atmos Auth profile configuration.

Migration Guide 1: Upgrading Your Infrastructure to Next-Gen​

Scenario: You're running legacy infrastructure with account-map, aws-teams, and aws-team-roles. You want to adopt Atmos Auth and remove the account-map dependency.

This is the full infrastructure migration — you're changing how your platform authenticates.

1. Set Up Atmos Auth​

Configure Atmos Auth profiles in your atmos.yaml. This tells Atmos how to authenticate to each account before Terraform runs:

# Authenticate with your profile
atmos auth login

See Atmos Auth for configuration details and the prerequisites section above.

2. Add the Static Account Map​

Define account IDs in your stack defaults so components can look up accounts without remote state:

# stacks/orgs/acme/_defaults.yaml
vars:
  account_map_enabled: false
  account_map:
    full_account_map:
      core-root: "123456789012"
      core-artifacts: "234567890123"
      core-audit: "345678901234"
      core-auto: "456789012345"
      plat-dev: "567890123456"
      plat-staging: "678901234567"
      plat-prod: "789012345678"
    root_account_account_name: core-root
    audit_account_account_name: core-audit
    iam_role_arn_templates:
      core-root: "arn:aws:iam::123456789012:role/acme-core-gbl-root-%s"
      core-artifacts: "arn:aws:iam::234567890123:role/acme-core-gbl-artifacts-%s"
      core-audit: "arn:aws:iam::345678901234:role/acme-core-gbl-audit-%s"
      core-auto: "arn:aws:iam::456789012345:role/acme-core-gbl-auto-%s"
      plat-dev: "arn:aws:iam::567890123456:role/acme-plat-gbl-dev-%s"
      plat-staging: "arn:aws:iam::678901234567:role/acme-plat-gbl-staging-%s"
      plat-prod: "arn:aws:iam::789012345678:role/acme-plat-gbl-prod-%s"

The iam_role_arn_templates map provides ARN templates for each account. The %s placeholder is replaced with the role name (e.g., terraform, planner) at runtime by components that need to assume cross-account roles.

3. Override providers.tf for Your Components​

Add the next-gen provider configuration to your stack defaults using atmos generate. This generates providers.tf for all components that inherit from this config:

# stacks/orgs/acme/_defaults.yaml (or any inherited stack config)
terraform:
  generate:
    "providers.tf": |
      variable "account_map_enabled" {
        type        = bool
        description = "Enable the account map component"
        default     = false
      }

      variable "account_map" {
        type = object({
          full_account_map              = map(string)
          audit_account_account_name    = optional(string, "")
          root_account_account_name     = optional(string, "")
          identity_account_account_name = optional(string, "")
          aws_partition                 = optional(string, "aws")
          iam_role_arn_templates        = optional(map(string), {})
        })
        description = "Static account map for components when account_map_enabled is false."
        default = {
          full_account_map              = {}
          audit_account_account_name    = ""
          root_account_account_name     = ""
          identity_account_account_name = ""
          aws_partition                 = "aws"
          iam_role_arn_templates        = {}
        }
      }

      provider "aws" {
        region = var.region
      }

      # Stub module that satisfies references to module.iam_roles in
      # upstream components. With Atmos Auth this is no longer needed,
      # so we replace it with a no-op label module.
      module "iam_roles" {
        source  = "cloudposse/label/null"
        context = module.this.context
      }

    # TEMPORARY: Override file to declare stale providers so OpenTofu can
    # load existing state that still references module.iam_roles from
    # account-map/modules/iam-roles. Override files merge with existing
    # required_providers blocks instead of conflicting.
    # Remove this after all components have been migrated and state is clean.
    "versions_override.tf": |
      terraform {
        required_providers {
          awsutils = {
            source  = "cloudposse/awsutils"
            version = ">= 0.1.0"
          }
          utils = {
            source  = "cloudposse/utils"
            version = ">= 0.1.0"
          }
        }
      }

Components that vendor a providers.tf from upstream need to exclude it so it doesn't conflict with the generated file:

# components/terraform/<component-name>/component.yaml
spec:
  source:
    excluded_paths:
      - "providers.tf"

Alternative: Use Vendor Mixins​

If you need per-component control or can't use atmos generate, you can vendor in the provider override directly in each component.yaml:

# components/terraform/<component-name>/component.yaml
apiVersion: atmos/v1
kind: ComponentVendorConfig
spec:
  source:
    uri: github.com/cloudposse-terraform-components/aws-<component>.git//src?ref={{ .Version }}
    version: v1.x.x
    included_paths:
      - "**/**"
    excluded_paths:
      - "providers.tf"
  mixins:
    - uri: https://raw.githubusercontent.com/cloudposse-terraform-components/mixins/{{ .Version }}/src/mixins/provider-without-account-map.tf
      version: v0.3.2
      filename: providers.tf
    - uri: https://raw.githubusercontent.com/cloudposse-terraform-components/mixins/{{ .Version }}/src/mixins/account-verification.mixin.tf
      version: v0.3.2
      filename: account-verification.mixin.tf

Then re-vendor:

atmos vendor pull -c <component-name>

4. Verify​

Run a plan against a non-production environment:

atmos terraform plan <component-name> -s plat-ue1-dev

Confirm the following:

5. Migrate Incrementally​

You don't have to do every component at once. Migrate one at a time, verify with a plan, and move on. Components using the next-gen providers.tf and components still on the legacy providers.tf can coexist in the same infrastructure — they just authenticate differently.

For the full migration path including IAM Identity Center setup and removing legacy components, see Migrate from Account-Map.

Migration Guide 2: Using New Component Versions on Legacy Infrastructure​

Scenario: You're still running account-map and team roles. You haven't set up Atmos Auth yet. But you want to upgrade to a newer version of a Cloud Posse component, and its providers.tf has already been updated to assume Atmos Auth.

This is the opposite problem — the component has moved to next-gen, but your infrastructure hasn't. The fix is the same tool: vendor in a providers.tf override via component.yaml.

How to Tell If a Component Has Moved to Next-Gen​

When you vendor a new version of a component, check its providers.tf. If you see this:

provider "aws" {
  region = var.region
}

Instead of the legacy module.iam_roles pattern, the component has been updated to assume Atmos Auth. It won't work out of the box with your account-map-based infrastructure.

The Fix: Vendor in a Legacy-Compatible providers.tf​

Override the component's providers.tf in your component.yaml to restore the legacy provider pattern:

# components/terraform/<component-name>/component.yaml
apiVersion: atmos/v1
kind: ComponentVendorConfig
spec:
  source:
    uri: github.com/cloudposse-terraform-components/aws-<component>.git//src?ref={{ .Version }}
    version: v2.x.x  # The new version with next-gen providers
    included_paths:
      - "**/**"
    excluded_paths:
      - "providers.tf"  # Exclude the next-gen providers.tf
  mixins:
    # Vendor in the legacy providers.tf that works with account-map
    - uri: https://raw.githubusercontent.com/cloudposse-terraform-components/mixins/{{ .Version }}/src/mixins/provider-with-account-map.tf
      version: v0.3.2
      filename: providers.tf

Then re-vendor:

atmos vendor pull -c <component-name>

This gives you the new component code with its bug fixes and features, but keeps the providers.tf compatible with your existing account-map infrastructure.

When to Use This Approach​

This is a bridge strategy — it lets you upgrade components now and migrate your infrastructure later, on your own timeline.

What Breaks and Common Errors​

Next-Gen Providers on Legacy Infrastructure​

If you vendor a component with the next-gen providers.tf but your infrastructure still uses account-map and team roles, Terraform will authenticate with whatever credentials are in your environment (or none at all) instead of assuming the correct role. Common symptoms:

Error: error configuring Terraform AWS Provider: no valid credential sources found

Error: AccessDenied: User: arn:aws:iam::123456789012:user/deploy is not authorized to perform: ...

Fix: Use Migration Guide 2 to vendor in the legacy-compatible providers.tf.

Legacy Providers on Next-Gen Infrastructure​

If you still have the legacy providers.tf with module.iam_roles sourced from ../account-map/modules/iam-roles, but you've already removed the account-map component:

Error: Module not found: module.iam_roles
  The module at "../account-map/modules/iam-roles" could not be found.

Fix: Use Migration Guide 1 to vendor in the next-gen providers.tf.

Account Verification Mismatch​

If Atmos Auth is configured but pointing to the wrong account:

Error: Account verification failed
  Expected account "567890123456" (plat-dev) but authenticated to "789012345678" (plat-prod).

Fix: Check your Atmos Auth profile mapping in atmos.yaml and run atmos auth login to refresh credentials.

Terraform State Impact​

Switching providers.tf does not require a state migration. The provider configuration change only affects how Terraform authenticates — it doesn't change resource addresses, module paths, or state structure. When you run terraform plan after the migration, you should see no drift related to the provider change itself.

The module.iam_roles is replaced by a dummy module, but since iam_roles only produces outputs consumed within providers.tf (not resources in state), there are no state entries to migrate or remove.

If you do see unexpected drift, it's likely caused by a different component version (new resource defaults, renamed attributes) rather than the provider change. Roll back the component version to isolate whether the drift comes from the provider switch or the component upgrade.

CI/CD Implications​

Atmos Auth changes how your CI/CD pipelines authenticate. The specifics depend on your runner.

GitHub Actions​

GitHub Actions workflows use OIDC to assume IAM roles directly. Deploy the iam-role component with GitHub OIDC trust policies in each target account. Your workflow authenticates via the standard aws-actions/configure-aws-credentials action before invoking atmos terraform.

Spacelift​

Spacelift manages its own AWS credentials via cloud integrations. If you're using Spacelift, it already authenticates before Terraform runs — the next-gen providers.tf (with just region = var.region) aligns naturally with this model. Ensure your Spacelift stacks have the correct AWS integration attached.

Atlantis​

Atlantis authenticates via IAM roles configured on the server or via OIDC. Configure your Atlantis server's assumed role to target each account, then use the next-gen providers.tf as-is.

General Pattern​

Regardless of runner, the pattern is the same: authenticate before Terraform runs, not inside providers.tf. If your CI system already handles AWS credential setup before executing Terraform commands, it's compatible with next-gen providers.

Summary​

The net result: fewer components to manage, simpler authentication, no deploy ordering dependencies, and a provider configuration you can actually read at a glance.

Learn More​

We're releasing Version 2 of the Cloud Posse Reference Architecture, which removes the account-map component and related identity components in favor of a simpler, more flexible approach using Atmos Auth profiles and static configuration.

What's Changing​

Version 2 of the Reference Architecture removes several components that were central to Version 1:

Additionally, the core-identity account is no longer used. Identity management is now centralized in core-root.

Why We Made This Change​

Reducing Terraform Dependencies​

The account-map component was a critical dependency that had to be deployed before most other components. It required careful ordering during cold starts and created circular dependency challenges. By replacing it with a static YAML configuration, we eliminate this complexity entirely.

This change also better supports brownfield deployments and alternate architectures where the full reference architecture isn't being adopted.

Simplifying Authentication​

Version 1 relied on a complex web of IAM roles created by aws-teams and aws-team-roles, which users accessed via Leapp. Version 2 uses:

This also removes our dependency on Leapp, which is deprecated.

Better Separation of Concerns​

With static account mappings defined in YAML, configuration is:

Version 2 Architecture​

For Human Users​

Human users authenticate via AWS SSO Permission Sets, accessed through Atmos Auth profiles:

# Select your profile based on your role
export ATMOS_PROFILE=devops

# Authenticate to AWS
atmos auth login

# Run Terraform commands
atmos terraform plan vpc -s plat-ue1-dev

Profiles map to different Permission Sets based on user role and target environment:

For Machine Users (CI/CD)​

GitHub Actions and other CI/CD systems use the iam-role component with GitHub OIDC:

Static Account Mapping​

Instead of the account-map component, account IDs are defined statically in your stack configuration:

# stacks/orgs/acme/_defaults.yaml
terraform:
  vars:
    account_map:
      root_account_account_name: core-root
      audit_account_account_name: core-audit
      full_account_map:
        core-root: "123456789012"
        core-artifacts: "234567890123"
        core-audit: "345678901234"
        core-auto: "456789012345"
        plat-dev: "567890123456"
        plat-staging: "678901234567"
        plat-prod: "789012345678"

Determining Your Version​

Not sure which version you're using? Check for these components in your infrastructure repository:

Version 1 indicators:

Version 2 indicators:

For detailed guidance, see Version Identification.

Migration Path​

Migrating from Version 1 to Version 2 is a significant undertaking. The high-level steps are:

Documentation​

This documentation site now supports versioned documentation:

Use the version selector in the navigation bar to switch between versions.

For a comprehensive history of how our identity architecture has evolved over time, see AWS Access Control Evolution.

Questions?​

If you have questions about this change or need migration assistance:

We're excited to announce comprehensive documentation for our suite of security and compliance Terraform components. These components enable you to deploy AWS security services across your entire AWS Organization using the delegated administrator pattern, providing centralized security monitoring and compliance assessment.

Hello SweetOps!

Security and compliance are critical for any organization running workloads on AWS. Whether you're pursuing SOC2, HIPAA, PCI DSS, FedRAMP, or CIS benchmarks, you need comprehensive visibility into threats, vulnerabilities, and configuration drift across all your accounts.

We've updated and documented our security and compliance components to make deploying these services straightforward and maintainable at scale.

What's Included​

Our security and compliance framework includes 9 Terraform components:

Key Architecture Decisions​

Our approach uses the delegated administrator pattern, centralizing security management while maintaining proper separation of concerns:

Deployment Models​

Different AWS services require different deployment approaches. We've documented each pattern:

3-Step Delegated Administrator​

Used by GuardDuty, Security Hub, and Macie:

1. Deploy to security account (creates the service)
2. Deploy to root account (delegates administration)
3. Deploy org settings to security account (configures organization-wide settings)

2-Step Delegated Administrator​

Used by Inspector and Access Analyzer:

1. Deploy to root account (delegates administration)
2. Deploy org settings to security account

Per-Account Deployment​

Used by Config and CloudTrail, with central aggregation in security/audit accounts.

Per-Resource Deployment​

Used by Shield Advanced for protecting specific resources like ALBs, CloudFront distributions, and Route53 hosted zones.

Compliance Framework Support​

These components support multiple compliance frameworks out of the box:

• CIS AWS Foundations Benchmark (v1.4, v1.5)
• AWS Foundational Security Best Practices
• PCI DSS (Payment Card Industry)
• HIPAA (Healthcare)
• SOC 2 (Service Organization Control)
• NIST 800-53 (Federal)
• FedRAMP (Federal Risk and Authorization)
• CMMC (Cybersecurity Maturity Model Certification)

Getting Started​

We've created comprehensive documentation to help you deploy these components:

1. Security and Compliance Overview - Architecture and component descriptions
2. Setup Guide - Step-by-step deployment instructions
3. FAQ - Common issues and troubleshooting

Each component also has its own detailed documentation page with stack configurations, deployment commands, and key variables.

Component Repositories​

All components are available in the cloudposse-terraform-components GitHub organization:

• aws-config
• aws-cloudtrail
• aws-guardduty
• aws-security-hub
• aws-inspector2
• aws-macie
• aws-access-analyzer
• aws-shield
• aws-audit-manager

What's Next​

We're continuing to improve our security and compliance components:

• Additional conformance pack templates for common compliance frameworks
• Enhanced integration between services
• More automated remediation patterns via EventBridge
• Expanded documentation for GovCloud deployments

We'd love to hear your feedback on these components. Let us know what compliance frameworks you're targeting and how we can make these components work better for your organization!

We've implemented the llms.txt standard to make our documentation more accessible to AI assistants, ensuring better responses when you ask ChatGPT, Claude, or other LLMs about Cloud Posse tools and best practices.

Hello SweetOps!

As AI assistants become increasingly integrated into developer workflows, we're excited to announce support for the llms.txt standard across our documentation site.

What is llms.txt?​

Think of llms.txt as the AI equivalent of robots.txt for search engines. It's an emerging standard that provides LLMs with structured, curated documentation in a format optimized for their understanding. Rather than crawling entire websites and hitting context window limits, AI assistants can now access our documentation in two curated formats:

• /llms.txt - A compact list of important documentation links
• /llms-full.txt - Full documentation content in markdown format

Why This Matters​

When you ask an AI assistant about Atmos, Terraform components, or Cloud Posse best practices, you'll get more accurate and up-to-date responses. The assistant can reference our curated documentation directly instead of relying on training data that may be outdated or incomplete.

Benefits for the Community​

• Better AI Assistance: More accurate responses when asking AI tools about our projects
• Efficient Context Usage: LLMs can access precisely what they need without crawling
• Up-to-Date Information: Always references current documentation, not stale training data
• Developer Velocity: Faster answers means less time searching, more time building

How It Works​

We're using the docusaurus-plugin-llms to automatically generate these files from our Docusaurus site. The plugin:

1. Prioritizes core documentation sections (Atmos, components, tutorials)
2. Includes blog content for recent updates and announcements
3. Generates both compact and full-text versions
4. Automatically updates with each documentation deployment

Try It Out​

Next time you're working with an AI assistant, try asking questions about:

• "How do I configure Atmos stacks?"
• "What's the latest on Cloud Posse component deprecation?"
• "Show me examples of Terraform component patterns"

The assistant will have direct access to our structured documentation, leading to better, more accurate responses.

Join the Movement​

The llms.txt standard is gaining adoption across the developer community. If you maintain documentation, consider implementing it for your projects. It's a simple way to make your content more accessible to the AI tools your users are already using.

We've documented our formal process for deprecating and archiving components to ensure transparency and give our community adequate notice when repositories are being sunset.

Hello SweetOps!

As part of our commitment to maintaining 300+ open source projects across Terraform modules, components, and other tooling, we occasionally need to deprecate repositories that are no longer actively maintained or have been superseded by better alternatives.

What to Expect​

We've added comprehensive documentation outlining our Deprecation and Archival Process to ensure this transition is as smooth as possible for everyone in our community.

When we deprecate a repository, here's what you can expect:

1. GitHub Issue Created: A pinned issue with detailed explanation, timeline, and migration guidance
2. README Warnings Added: Prominent deprecation notices at the top of documentation
3. Blog Post Published: Announcement in our changelog/blog about the deprecation
4. Pull Request Submitted: All changes announced via PR for community visibility
5. Grace Period: Typically 90+ days for the community to migrate and ask questions
6. Repository Archived: After the grace period, repos are archived (not deleted) and remain publicly accessible
7. Blog Post Updated: Announcement updated to reflect the archival completion

Why This Matters​

This structured approach ensures that:

• You have advance notice before any repository is archived
• Migration paths and alternatives are clearly documented
• Historical access to code is preserved
• The community can provide feedback during the deprecation period

Our Commitment​

As stated in our GitHub documentation, we commit to always provide free and public access to our Open Source repositories. Even when archived, repositories remain accessible for historical reference and continued use.

When simplicity meets automation, sometimes it's the hidden complexity that bites back.

For a while, running Karpenter on AWS Fargate sounded like a perfect solution. No nodes to manage, automatic scaling, and no EC2 lifecycle headaches. The AWS EKS Best Practices Guide and Karpenter's official documentation both present Fargate as a viable option for running the Karpenter controller.

But in practice, that setup started to cause problems for certain EKS add-ons. Over time, those lessons led us — and our customers — to recommend using a small managed node group (MNG) instead of relying solely on Fargate.

This recommendation diverges from some official AWS guidance, and we acknowledge that. Here's why we made this decision.

Why Fargate Was Attractive (and Still Is, Sometimes)​

The appeal of Fargate for Karpenter is understandable:

• No need to bootstrap a managed node group before deploying Karpenter
• Simpler initial setup for teams not using Infrastructure-as-Code frameworks
• Karpenter's early versions had limited integration with managed node pools
• It showcased Karpenter's capabilities in the most dramatic way possible

For teams deploying clusters manually or with basic tooling, Fargate eliminates several complex setup steps. But when you're using sophisticated Infrastructure-as-Code like Cloud Posse's Terraform components, that initial complexity is already handled—and the operational benefits of a managed node group become far more valuable.

The Problem with "No Nodes" (and the Terraform Catch-22)​

EKS cluster creation with Terraform requires certain managed add-ons — like CoreDNS or the EBS CSI driver — to become active before Terraform considers the cluster complete.

But Fargate pods don't exist until there's a workload that needs them. That means when Terraform tries to deploy add-ons, there are no compute nodes for the add-ons to run on. Terraform waits… and waits… until the cluster creation fails.

Terraform enforces a strict dependency model: it won't complete a resource until it's ready. Without a static node group, Terraform can't successfully create the cluster (because the add-ons can't start). And without those add-ons running, Karpenter can't launch its first node (because Karpenter itself is waiting on the cluster to stabilize).

This circular dependency means your beautiful "fully automated" Fargate-only cluster gets stuck in the most ironic place: bootstrap deadlock.

You can manually retry or patch things later, but that defeats the purpose of automation. We build for repeatability — not babysitting.

The Hidden Cost of "Serverless Nodes"​

Even after getting past cluster creation, there are subtle but serious issues with high availability.

By AWS and Cloud Posse best practices, production-grade clusters should span three availability zones, with cluster-critical services distributed across them.

However, during initial scheduling with managed node groups, Karpenter might spin up just one node large enough to fit all your add-on pods — even if they request three replicas with anti-affinity rules. Kubernetes will happily co-locate them all on that single node.

Once they're running, those pods don't move automatically, even as the cluster grows. The result?

A deceptively healthy cluster with all your CoreDNS replicas living on the same node in one AZ — a single point of failure disguised as a distributed system.

While topologySpreadConstraints can help encourage multi-AZ distribution, they don't guarantee it during the critical cluster bootstrap phase when Karpenter is creating its first nodes.

The Solution: A Minimal Managed Node Pool​

Our solution is simple:

Deploy a tiny managed node group — one node per availability zone — as part of your base cluster.

• This provides a home for cluster-critical add-ons during creation
• It ensures that CoreDNS, EBS CSI, and other vital components are naturally distributed across AZs
• It gives Karpenter a stable platform to run on
• And it eliminates the bootstrap deadlock problem entirely

You can even disable autoscaling for this node pool. One node per AZ is enough.

Think of it as your cluster's heartbeat — steady, predictable, and inexpensive.

Additional Fargate Constraints​

Beyond the HA challenges, Fargate has architectural constraints that can affect cluster add-ons:

• Each Fargate pod runs on its own isolated compute resource (one pod per node)
• No support for EBS-backed dynamic PVCs; only EFS CSI volumes are supported
• Fixed CPU and memory configurations with coarse granularity
• 256 MB memory overhead for Kubernetes components

While these constraints don't necessarily prevent Fargate from working, they add complexity when running cluster-critical infrastructure that needs precise resource allocation and high availability guarantees.

Cost and Flexibility​

Fargate offers convenience, but at a premium. A pod requesting 2 vCPUs and 4 GiB of memory costs about $0.098/hour, compared to $0.076/hour for an equivalent EC2 c6a.large instance.

And because Fargate bills in coarse increments, you often overpay for partial capacity.

By contrast, the hybrid approach unlocks significant advantages:

• Static MNG with On-Demand instances provides a stable foundation for cluster add-ons
• Use cost-effective Graviton instances (c7g.medium) to reduce baseline costs
• Karpenter provisions Spot instances exclusively for application workloads (not add-ons)
• Achieve cost savings on application pods while maintaining reliability for cluster infrastructure

The result: stable cluster services on On-Demand, cost-optimized applications on Spot.

The Evolution of Karpenter's Recommendations​

Interestingly, the Karpenter team's own guidance has evolved over time. Karpenter's current getting started guide now defaults to using EKS Managed Node Groups in its example configurations, with Fargate presented as an alternative that requires uncommenting configuration sections.

While we can't pinpoint exactly when this shift occurred, it suggests the Karpenter team recognized that managed node groups provide a more reliable foundation for most production use cases.

Lessons Learned​

At Cloud Posse, we love automation — but we love reliability through simplicity even more.

Running Karpenter on Fargate works for proof-of-concepts or ephemeral clusters.

But for production systems where uptime and high availability matter, a hybrid model is the clear winner:

• Static MNG with On-Demand instances for cluster-critical add-ons (CoreDNS, Karpenter, etc.)
• Karpenter provisioning Spot instances for dynamic application workloads
• Fargate only when you truly need pod-level isolation

It's not about Fargate being bad — it's about knowing where it fits in your architecture.

When Fargate-Only Might Still Work​

To be fair, there are scenarios where running Karpenter on Fargate might make sense:

• Long-lived development environments where the $120/month MNG baseline cost matters more than availability
• Clusters deployed manually (not via Terraform) where bootstrap automation isn't critical
• Proof-of-concept deployments demonstrating Karpenter's capabilities
• Organizations that have accepted the operational trade-offs and built workarounds

However, be aware that development clusters that are frequently rebuilt will hit the Terraform bootstrap deadlock problem more often—making automation failures a regular occurrence rather than a one-time setup issue.

Your Mileage May Vary​

It's worth noting that experienced practitioners in the SweetOps community have successfully run Karpenter on Fargate for years across multiple production clusters. Their setups work, and they've built processes around the constraints.

This proves our recommendation isn't absolute—some teams make Fargate work through careful configuration and accepted trade-offs. However, these same practitioners acknowledged they'd likely choose MNG if starting fresh today with modern tooling.

"Karpenter doesn't use voting. Leader election uses Kubernetes leases. There's no strict technical requirement to have three pods — unless you actually care about staying up."

— Ihor Urazov, SweetOps Slack

That's the key insight. The technical requirements are flexible—it's your operational requirements that determine the right choice.

If staying up matters, if automation matters, if avoiding manual intervention matters, then give your cluster something solid to stand on. A small, stable managed node pool does exactly that.

What About EKS Auto Mode?​

It's worth mentioning that AWS introduced EKS Auto Mode in December 2024, which takes a fundamentally different approach to solving these problems.

EKS Auto Mode runs Karpenter and other critical cluster components (like the EBS CSI driver and Load Balancer Controller) off-cluster as AWS-managed services. This elegantly sidesteps the bootstrap deadlock problem entirely—there's no chicken-and-egg dependency because the control plane components don't need to run inside your cluster.

The cluster starts with zero nodes and automatically provisions compute capacity as workloads are scheduled. While this solves the technical bootstrap challenge we've discussed, it comes with trade-offs:

• Additional 12-15% cost premium on top of EC2 instance costs
• Lock-in to AWS VPC CNI (can't use alternatives like Cilium or Calico)
• Less control over cluster infrastructure configuration
• Available only for Kubernetes 1.29+ and not in all AWS regions

For organizations willing to accept these constraints in exchange for fully managed operations, EKS Auto Mode may address many of the concerns raised in this post. However, for teams requiring fine-grained control, cost optimization, or running on older Kubernetes versions, the MNG + Karpenter approach remains highly relevant.

We're excited to announce our new Platform Advisory service—now available to Cloud Posse customers. Private access to Cloud Posse engineers.

Many of our larger customers—especially in fintech, health tech—who operate in regulated industries have asked for a way to get private, real-time access to senior Cloud Posse engineers for their most critical projects.

These teams often run into scenarios where:

• Delays, mistakes, or failed migrations would cost big
• They need to de-risk complex platform changes
• They want trusted guidance on Cloud Posse architecture and components—from the engineers who built it

Platform Advisory was designed specifically to address these needs.

What is Platform Advisory?​

Platform Advisory gives your team:

• Private Slack Connect → direct access to Cloud Posse’s staff-to-principal-level engineers
• On-demand Zoom sessions → architecture reviews, migration planning, compliance discussions, and more
• Same-day response (4-hour SLA) → for high-impact requests
• 10 hrs/month of Flexible Support included → for bug fixes, upgrades, Atmos enhancements, new components, and integration work

It’s designed for teams running on Cloud Posse’s reference architecture and open source components who need priority access to expert guidance—especially when getting it wrong isn’t an option.

Why we built this​

As more of our customers adopt Cloud Posse architecture for mission-critical platforms, they’ve asked for a way to engage more deeply—especially for projects where de-risking migrations and accelerating delivery matters.

When the team is facing a complex migration, rolling out new environments, or making high-impact platform changes—waiting days for answers isn’t good enough.

Platform Advisory gives them priority access to engineers who:

• Know Cloud Posse architecture inside and out
• Understand their environments and goals
• Can chart the best path forward quickly and safely

How it fits with our other support options​

Where to learn more​

You can explore all the details on the Platform Advisory support page.

If you’re unsure whether Platform Advisory is the right fit for your team, reach out to us—we’re happy to help.

Remember: When you invest in Cloud Posse, you’re not just helping your team—you’re strengthening the ecosystem your business depends on.

We’re excited to make Platform Advisory available—and we look forward to helping more teams succeed on Cloud Posse architecture.

We're excited to announce the completion of the second phase of our Component Testing project, which has added automated testing for 27 components. This milestone follows our successful migration of 160+ Terraform Components from a monorepo to individual repositories, making them more maintainable and testable.

Hello SweetOps!

A few months ago, we embarked on a MASSIVE project to enable Component Testing.

The goal is to improve the stability of our components, detect and fix integration errors, and pave the way for confident delivery of new features. In the first phase, we split the cloudposse/terraform-aws-components monorepo consisting of 160+ Terraform Components into individual repositories in the cloudposse-terraform-components GitHub organization. We updated the cloudposse/github-action-atmos-component-updater GitHub action to rewrite URLs in component manifests automatically, allowing you to smoothly migrate to new repositories.

Current Status​

Now, we are happy to announce that we have completed the second phase of this project, introducing automated tests for the first 27 components. Hopefully, you are already using components from the new organization!

The complete list of covered components can be found here.

We've developed a Go-based testing framework built on top of Terratest, optimized specifically for testing Atmos components.
Additionally, we created a step-by-step guide to help you write effective component tests.
You can track the project's progress on this board.

We invite everyone to contribute to this project.

Please like the "Add component tests" issue in the corresponding component repository for which you are interested in prioritizing test coverage. If you want to contribute more, we have the opportunity.

How can you help?​

We really need help writing tests.

You can take any "Add component tests" issue with the "Good First Question" label and contribute to the test following our documentation.

We will prioritize reviewing your PRs in the #pr-reviews channel and help ensure they get merged smoothly.. Feel free to DM to @Erik Osterman or @Igor Rodionov in Slack with any questions or feedback.

P.S.: Huge thanks to @RoseSecurity for the first community-driven component test contribution here.

The GitHub repository for Cloud Posse's Terraform components has migrated to a dedicated GitHub organization. All documentation remains here, but all future updates, contributions, and issue tracking for the source code should now be directed to the respective repositories in the new organization.

We're excited to announce that starting on November 12, 2024, we will begin migrating each component in the cloudposse/terraform-aws-components repository to individual repositories under a new GitHub organization. This change aims to improve the stability, maintainability, and usability of our components.

Why This Migration?​

Our goal is to make each component easier to use, contribute to, and maintain. This migration will allow us to:

What to Expect Starting November 12, 2024​

Migration Timeline​

The migration will begin on November 12 and is anticipated to finish by the end of the following week.

Code Freeze​

Starting on November 12, this repository will be set to read-only mode, marking the beginning of a code freeze. No new pull requests or issues will be accepted here after that date.

New Contribution Workflow​

After the migration, all contributions should be directed to the new individual component repositories.

Updated Documentation​

To support this transition, we are updating our documentation and cloudposse-component updater.

Future Archiving​

In approximately six months, we plan to archive this repository and transfer it to the cloudposse-archives organization.

Frequently Asked Questions​

Does this affect Terraform modules?​

No, only the terraform-aws-components repository is affected. Our Terraform modules will remain where they are.

We are committed to making this transition as seamless as possible. If you have any questions or concerns, please feel free to post them in this issue. Your feedback is important to us, and we appreciate your support as we embark on this new chapter!