[Feature Request] Better support usage of history when leveraging `terragrunt`

[rtizzy]

Explanation of typical terragrunt usage

When using terragrunt, you will often have a single set of modules that are shared across environments with the difference between each environment being the parameters that are fed into the modules. Typically the folder structure is a reflection of the underlying infrastructure and terraform state.

Lets consider a contrived example of a typical terragrunt folder layout.

• module_a and module_b can be assumed to be the same module call in each environment
• Our migration file will migrate from module_a to module_b

├── env1
│   ├── module_a
│   │   ├── .tfmigrate
│   │   │   └── 20250117133204_example_migration.hcl
│   │   └── terragrunt.hcl
│   └── module_b
│       └── terragrunt.hcl
└── env2
    ├── module_a
    │   ├── .tfmigrate
    │   │   └── 20250117133204_example_migration.hcl
    │   └── terragrunt.hcl
    └── module_b
        └── terragrunt.hcl

The issue

Lets assume we have the following config for tfmigrate

tfmigrate {
  # Instruct tfmigrate to look for migrations in the `.tfmigrate` folder of the current dir 
  migration_dir = "./tfmigrate"
  history {
    # Configure the history bucket 
    storage "s3" {
      bucket = "history-state-bucket"
      region = "us-east-1"
      key    = "tfmigrate/history.json"
    }
  }
}

If we run the migrations in env1/module_a the history.json for migrations will contain something like this.

{
    "version": 1,
    "records": {
        "20250117133204_example_migration.hcl": {
            "type": "multi_state",
            "name": "example_migration",
            "applied_at": "2025-01-17T13:20:23.085857+01:00"
        }
    }
}

If we go to the env2/module_a folder and attempt to leverage a history based migration (where we have not yet ran migrations), it will indicate that all migrations have been run.

╰─ TFMIGRATE_EXEC_PATH=terragrunt tfmigrate plan --config ../../../../../../.tfmigrate.hcl
2025/01/17 13:24:39 [INFO] [runner] no unapplied migrations

Possible solutions

Allow arbitrary definition of additional migration keys.

I think this is a solution that could be flexible and would resolve this issue.

Assuming it is possible read the underlying Terraform variables/parameters, allow users to define via .tfmigrate.hcl or command line flags additional keys that should be included as part of the history.json keys.

Example

This isn't the best example and the exact way it's handled can be improved.

We have a variable in Terraform that indicates the current environment, var.environment.

In env1 the value will be env1 and in env2 the value will be env2.

Our .tfmigrate.hcl might contain something like

tfmigrate {
    migration_dir = "./tfmigrate"
    history {
      additional_keys = [
        "environment"
      ]
      # Configure the history bucket 
      storage "s3" {
        bucket = "history-state-bucket"
        region = "us-east-1"
        key    = "tfmigrate/history.json"
      }
    }
  }

Which might result in something like this in the history file

{
    "version": 1,
    "env1": {
        "records": {
            "20250117133204_example_migration.hcl": {
                "type": "multi_state",
                "name": "example_migration",
                "applied_at": "2025-01-17T13:20:23.085857+01:00",
                "state_path": "$PATHTOSTATEOFTHEINITIALMODULE"
            }
        }
    }
}

If we run this same thing in env2 it would see that there is no key for env2, then run the migration as expected.

Include a fixed key

Include an additional fixed key in the migration for something like the folder_path or state_path

Example

{
    "version": 1,
    "records": {
        "20250117133204_example_migration.hcl": {
            "type": "multi_state",
            "name": "example_migration",
            "applied_at": "2025-01-17T13:20:23.085857+01:00",
            "state_path": "$PATHTOSTATEOFTHEINITIALMODULE"
        }
    }
}

[minamijoyo]

Hi @rtizzy, Thank you for your proposal! I don't fully understand the terragrunt use case, but let me explain my thoughts from the point of view of tfmigrate design.

Since terraform variables are resolved when the terraform command is executed, their value is unknown when tfmigrate reads the migration history. If we were to read the tfvars file directly, this would create an architectural inconsistency since tfmigrate can also move resources between directories.
For the include a fixed key proposal, if we set the fixed arbitrary attributes, they must be unique keys in the history file. In this case, it won't be easy to track history if the attributes are changed in the future. In refactoring scenarios where tfmigrate is required, the granularity of common parts is unstable and should be assumed to be split and merged.

Based on the above, why not simply split the history files? Splitting the configuration file and separating the history file paths is feasible as it is now.

There was a similar topic #91 about reusing the same migration file between workspaces. To address this issue, in #171, we made it possible for the migration file to reference environment variables. To use this, we need to split the history file for each workspace and, therefore, split the configuration file. If you want to reuse configuration files and make them DRY, extending this concept to include environment variable references in the tfmigrate configuration files may be helpful. This would allow us to switch history file paths at runtime.

For example:

tfmigrate {
    migration_dir = "./tfmigrate"
    history {
      # Configure the history bucket 
      storage "s3" {
        bucket = "history-state-bucket"
        region = "us-east-1"
        # export TFMIGRATE_ENVIRONMENT= env1 or env2
        key    = "tfmigrate/${env.TFMIGRATE_ENVIRONMENT}/history.json"
      }
    }
  }

[rtizzy]

@minamijoyo

Thanks for the explanation.

If it's possible to leverage environment variables, that may be a solution for this use case that allows leveraging a single config file (one of my desired outcomes).

Will look into this later and update.

[rtizzy]

@minamijoyo

I may be holding this wrong but attempting to use things in the way you suggest throws the following error on the latest version

tfmigrate plan --config=../../../../../../../.tfmigrate.hcl
failed to load config file: ../../../../../../../.tfmigrate.hcl:7,19-22: Variables not allowed; Variables may not be used here., and 1 other diagnostic(s)

tfmigrate --version
0.4.1

tfmigrate {
  migration_dir = "./migrations"
  history {
    storage "s3" {
      bucket = "bucketname"
      region = "regionname"
      key    = "${env.TFMIGRATE_S3_PREFIX}/history.json"
    }
  }
}

[minamijoyo]

Environment variables are supported in the migration file but not currently in the configuration file. My previous comment is about a hypothetical configuration file that would hopefully have such a feature. Since we know how to inject environment variables into the HCL file, it would be relatively easy to implement this feature in the configuration file.

[minamijoyo]

Injecting environment variables into .tfmigrate.hcl is now available in tfmigrate v0.4.2.