================================================================================ ATMOS STORE HOOKS DEBUG REPORT ================================================================================ Atmos version: 1.218.0 OS: macOS darwin/arm64 Auth: atmos auth with AWS IAM Identity Center (SSO) Backend: S3 GitHub Issue: https://github.com/cloudposse/atmos/issues/2428 Related: https://github.com/cloudposse/atmos/issues/2357 ================================================================================ SUMMARY ================================================================================ Store hooks (after-terraform-apply) are not working. We've isolated 4 separate issues through testing. All tests used the same component (vpc-flow-logs-bucket) which already has deployed infrastructure (no changes on apply). Issue 1: Identity-based stores silently don't fire (with CI enabled) Issue 2: Generic stores silently don't fire (with CI enabled) Issue 3: Generic stores fire but fail on credentials (with CI disabled) Issue 4: Identity-based stores fire but fail with "auth resolver not set" (when identities are only in profiles, not atmos.yaml) Only workaround: ci.enabled=false + generic store (no identity) + atmos auth exec --identity -- atmos terraform apply ... ================================================================================ CONFIGURATION ================================================================================ --- .atmos.d/defaults.yaml (relevant sections) --- auth: realm: infra profiles: default: "admin" ci: enabled: true stores: ssm: type: aws-ssm-parameter-store options: region: us-east-1 prefix: /atmos/ --- profiles/admin/auth.yaml --- auth: providers: sso: kind: aws/iam-identity-center region: us-east-1 start_url: https://tembo.awsapps.com/start/ identities: tembo-dev: kind: aws/permission-set via: provider: sso principal: name: TerraformApplyAccess account: name: tembo-dev tembo-staging: kind: aws/permission-set via: provider: sso principal: name: TerraformApplyAccess account: name: tembo-staging tembo-prod: kind: aws/permission-set via: provider: sso principal: name: TerraformApplyAccess account: name: tembo-prod core-root: kind: aws/permission-set via: provider: sso principal: name: TerraformApplyAccess account: name: core-root --- stacks/orgs/_defaults.yaml --- vars: namespace: tembo tags: repository: "tembo/infra" terraform: hooks: store-outputs: events: ["after-terraform-apply"] command: store name: ssm backend_type: s3 backend: s3: bucket: "tembo-use1-root-tfstate" region: us-east-1 key: terraform.tfstate acl: bucket-owner-full-control encrypt: true use_lockfile: true --- stacks/catalog/vpc-flow-logs-bucket/defaults.yaml --- components: terraform: vpc-flow-logs-bucket: metadata: component: vpc-flow-logs-bucket hooks: store-outputs: outputs: vpc_flow_logs_bucket_arn: .vpc_flow_logs_bucket_arn vpc_flow_logs_bucket_id: .vpc_flow_logs_bucket_id vars: enabled: true name: vpc-flow-logs --- stacks/orgs/aws/tembo/dev/_defaults.yaml --- import: - ../_defaults.yaml vars: stage: dev auth: identities: tembo-dev: default: true ================================================================================ RESOLVED COMPONENT CONFIG (atmos describe component) ================================================================================ $ atmos describe component vpc-flow-logs-bucket -s tembo-use1-dev | grep -A 12 hooks hooks: store-outputs: command: store events: - after-terraform-apply name: ssm outputs: vpc_flow_logs_bucket_arn: .vpc_flow_logs_bucket_arn vpc_flow_logs_bucket_id: .vpc_flow_logs_bucket_id NOTE: The hooks resolve correctly in describe output for all configurations tested below. The issue is at runtime, not resolution. ================================================================================ ISSUE 1: Identity-based stores silently don't fire ================================================================================ Config change: store name = ssm/tembo-dev (with identity) stores: ssm/tembo-dev: type: aws-ssm-parameter-store identity: tembo-dev options: region: us-east-1 prefix: /atmos/ terraform.hooks.store-outputs.name: ssm/tembo-dev Command: atmos terraform apply vpc-flow-logs-bucket -s tembo-use1-dev -- -auto-approve Result: Hook NEVER fires. No "Fetching" output, no errors. Silent failure. Trace shows only CI hooks running (and being skipped): DEBU Running CI hooks event=after.terraform.apply force_ci=false DEBU CI provider not detected provider=generic DEBU CI platform not detected, skipping CI hooks error="CI provider not detected" DEBU Telemetry event enqueued No store hook activity whatsoever. This happens with BOTH ci.enabled=true and ci.enabled=false. ================================================================================ ISSUE 2: Generic stores silently don't fire (CI enabled) ================================================================================ Config change: store name = ssm (generic, no identity), ci.enabled=true terraform.hooks.store-outputs.name: ssm Command: atmos terraform apply vpc-flow-logs-bucket -s tembo-use1-dev -- -auto-approve Result: Same as Issue 1 — hook never fires. Silent failure. Identical trace output with only CI hooks running and being skipped. ================================================================================ ISSUE 3: Generic stores fire but fail on credentials (CI disabled) ================================================================================ Config change: store name = ssm (generic), ci.enabled=false Command: atmos terraform apply vpc-flow-logs-bucket -s tembo-use1-dev -- -auto-approve Result: Hook FIRES but fails reading terraform outputs from S3 backend: Fetching vpc_flow_logs_bucket_arn output from vpc-flow-logs-bucket in tembo-use1-dev ✗ Fetching vpc_flow_logs_bucket_arn output from vpc-flow-logs-bucket in tembo-use1-dev Error: failed to retrieve terraform outputs: hook "ssm" (event "after.terraform.apply") failed to get terraform output "vpc_flow_logs_bucket_arn" for component "vpc-flow-logs-bucket" in stack "tembo-use1-dev": failed to retrieve terraform outputs: exit status 1 Error: No valid credential sources found Please see https://opentofu.org/docs/language/settings/backends/s3 Error: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, exceeded maximum number of attempts, 3 The main terraform apply had credentials (it completed successfully), but the hook's subprocess for reading terraform output does not inherit them. ================================================================================ ISSUE 4: Identity-based stores + identities in atmos.yaml (not profiles) ================================================================================ Config change: Moved auth.providers and auth.identities from profiles/admin/auth.yaml into .atmos.d/defaults.yaml directly. Store name = ssm/tembo-dev, ci.enabled=false Command: atmos terraform apply vpc-flow-logs-bucket -s tembo-use1-dev -- -auto-approve Result: Hook fires (identity resolves!) but still fails on terraform output: Fetching vpc_flow_logs_bucket_arn output from vpc-flow-logs-bucket in tembo-use1-dev ✗ Fetching vpc_flow_logs_bucket_arn output from vpc-flow-logs-bucket in tembo-use1-dev Error: failed to retrieve terraform outputs: hook "ssm/tembo-dev" (event "after.terraform.apply") failed to get terraform output "vpc_flow_logs_bucket_arn" for component "vpc-flow-logs-bucket" in stack "tembo-use1-dev": failed to retrieve terraform outputs: exit status 1 Error: No valid credential sources found Key finding: When identities are only in profiles (not atmos.yaml), identity- based stores silently don't fire (Issue 1). When identities are in atmos.yaml, they fire but fail on terraform output credentials. Additional finding: With identities ONLY in profiles + atmos auth exec wrapper: $ atmos auth exec --identity tembo-dev -- atmos terraform apply ... (ci.enabled=false) ✓ Fetching vpc_flow_logs_bucket_arn output from vpc-flow-logs-bucket in tembo-use1-dev Error: store identity is configured but auth resolver is not set store requires identity "tembo-dev" but no auth resolver was injected Terraform output succeeds (creds from auth exec in env), but store write fails because auth resolver isn't injected into hook context. ================================================================================ WORKING WORKAROUND ================================================================================ The ONLY combination that works end-to-end: - ci.enabled: false - Generic store (name: ssm, no identity field) - Wrapped with atmos auth exec $ atmos auth exec --identity tembo-dev -- atmos terraform apply vpc-flow-logs-bucket -s tembo-use1-dev -- -auto-approve ✓ Fetching vpc_flow_logs_bucket_arn output from vpc-flow-logs-bucket in tembo-use1-dev ✓ Fetching vpc_flow_logs_bucket_id output from vpc-flow-logs-bucket in tembo-use1-dev ================================================================================ ROOT CAUSE ANALYSIS ================================================================================ 1. CI hooks block store hooks: When ci.enabled=true, the after-terraform-apply event is intercepted by the CI hooks system. When no CI provider is detected (local dev), CI hooks are skipped, but store hooks never get their turn. 2. Auth resolver not injected into hooks: The hook runs with SkipInit=true and authManager=nil (see pkg/terraform/output/executor.go). This means: - processYamlFunctions is set to false - Environment/credential setup is skipped - terraform output subprocess has no AWS credentials - Identity-based store writes fail ("auth resolver not set") 3. Profile identities not available to store system: When identities are defined in profiles (profiles/admin/auth.yaml) rather than directly in atmos.yaml (.atmos.d/defaults.yaml), identity-based stores silently fail to fire. Moving identities to atmos.yaml fixes the silent failure but doesn't fix the credential issue. Related code in pkg/terraform/output/executor.go: processYamlFunctions := true if opts != nil && opts.SkipInit && authManager == nil { processYamlFunctions = false }