[Feature Request] Multiple migrations per file

[scoplin]

Our use case is one where we wish to refactor some existing infrastructure code from having multiple terragrunt files with independent stacks to a single stack that invokes a module that consolidates the prior infrastructure components. Our reason is to support greater promotability of the consolidated component across environments. During the course of this, we do not wish to necessarily change application architecture or the actual cloud resources, only the way we organize our terraform code and invoke it across multiple operational environments: qa, staging (multiple), production (multiple).

I see comments in the code such as this one that suggest that multi-state migration (multiple sources, single target) has been explicitly considered, but either not implemented yet or ruled out by design. I'm not sure which. I also see this comment from @minamijoyo which seems to suggest that this might be just a "not implemented yet" feature:

The current implementation is limited to defining only one migration per file, but how about using a file boundary as a transaction boundary to group multiple migrations as an atomic transaction?

Is it possible to get a feature to serve this use case implemented? Given our desire for atomic migrations involving 3+ terraform state files, it's a blocker for our current use case since we don't wish to break down our code refactor into a series of 2-state partial migrations because of the added operational complexity in rolling it out. We're currently considering simply mimicking the state migration logic and local state manipulations instead of using tfmigrate as-is.

[minamijoyo]

I understand the necessity of the concept of transactions, but it has not been implemented yet. Technically, I think it is possible, but it would require managing the intermediate states of an arbitrary number of directories in memory, which would necessitate a significant overhaul of the internal architecture.

[scoplin]

Hi @minamijoyo - we're willing to take a stab at doing those modifications and sending a PR. Any rough-cut ideas that you could suggest up-front?

[minamijoyo]

I'm thinking of how to implement it. My rough design sketch is as follows.

To enable new experiments safely without a significant overhaul of the internal architecture, how about forgetting the idea of implicitly using a file boundary as a transaction and defining it as a new migration type?

# Avoid completely rewriting the existing internal architecture and define it
# as a new migration type.
migration "txn_state" "merge_to_dir3" {
  # A step block is almost the same as the migration block,
  # but some ambiguous flags are removed.
  # For example, to_skip_plan causes a conflict when different values are
  # specified in multiple steps.
  step "multi_state" "mv_dir1_dir3" {
    from_dir = "dir1"
    to_dir   = "dir3"
    actions = [
      "mv aws_security_group.foo aws_security_group.foo1",
      "mv aws_security_group.bar aws_security_group.bar1",
    ]
  }

  # Pass state between steps.
  step "multi_state" "mv_dir2_dir3" {
    from_dir = "dir2"
    to_dir   = "dir3"
    actions = [
      "mv aws_security_group.foo aws_security_group.foo2",
      "mv aws_security_group.bar aws_security_group.bar2",
    ]
  }

  # The design should allow for the implementation of a single state if needed,
  # but its implementation is not mandatory for the initial release.
  step "state" "rename_dir3" {
    dir   = "dir3"
    actions = [
      "mv aws_security_group.foo aws_security_group.foo3",
      "mv aws_security_group.bar aws_security_group.bar3",
    ]
  }
}

Initially, I considered wrapping a single migration as a transaction and extending the multi_state type, but considering the need for rollback control during error handling, it seems that the current design would require significant changes. I feel it's somewhat risky. By implementing it as a completely new migration type, we can minimize the impact on existing code while enabling new experiments.

The advantage of this approach is that we can custom-build error handling and rollback control, which are expected to become complex in transaction management, and it also doesn’t conflict with the data structure of existing history files. I don’t see any drawbacks other than slightly verbose code. In the future, if the new implementation proves stable, it might be possible to replace the existing one.

The transaction needs to manage the following contexts:

• A map with workspace identifiers as keys and workspace contexts as values.
• A workspace identifier is a structure that has a directory name and a workspace name. (In Go, you can use any comparable structure as a type of key in a map.)
• The workspace context includes the current tfstate, rollback function, and Terraform CLI instance.

At the start of the transaction, initialize all workspaces to be operated on.
Since the same workspace is being manipulated, it is necessary to carry over the state between steps.
At the end, check the plan and then push all states to remote.
If an error occurs during the transaction, be sure to roll back the backend settings for all workspaces.

How about something like this?