GitOps friendly: Write terraform state mv/rm/import commands in HCL, plan and apply it.
Monorepo style support: Move resources to other tfstates to split and merge easily for refactoring.
Dry run migration: Simulate state operations with a temporary local tfstate and check to see if terraform plan has no changes after the migration without updating remote tfstate.
Migration history: Keep track of which migrations have been applied and apply all unapplied migrations in sequence.
You can apply terraform state operations in a declarative way.
In short, write the following migration file and save it as
It works as you expect, but it's just a text file, so you can commit it to git.
Why?
If you have been using Terraform in production for a long time, tfstate manipulations are unavoidable for various reasons. As you know, the terraform state command is your friend, but it's error-prone and not suitable for a GitOps workflow.
In team development, Terraform configurations are generally managed by git and states are shared via remote state storage which is outside of version control. It's a best practice for Terraform.
However, most Terraform refactorings require not only configuration changes, but also state operations such as state mv/rm/import. It's not desirable to change the remote state before merging configuration changes. Your colleagues may be working on something else and your CI/CD pipeline continuously plan and apply their changes automatically. At the same time, you probably want to check to see if terraform plan has no changes after the migration before merging configuration changes.
To fit into the GitOps workflow, the answer is obvious. We should commit all terraform state operations to git.
This brings us to a new paradigm, that is to say, Terraform state operation as Code!
Supported Terraform versions
The tfmigrate invokes
terraform
command under the hood. This is because we want to support multiple terraform versions in a stable way. Currently supported terraform versions are as follows:
Terraform v0.14.x
Terraform v0.13.x
Terraform v0.12.x
Getting Started
As you know, terraform state operations are dangerous if you don't understand what you are actually doing. If I were you, I wouldn't use a new tool in production from the start. So, we recommend you to play an example sandbox environment first, which is safe to run terraform state command without any credentials. The sandbox environment mocks the AWS API with
localstack
and doesn't actually create any resources. So you can safely run the
tfmigrate
and
terraform
commands, and easily understand how the tfmigrate works.
Build a sandbox environment with docker-compose and run bash:
$ git clone https://github.com/minamijoyo/tfmigrate
$ cd tfmigrate/
$ docker-compose build
$ docker-compose run --rm tfmigrate /bin/bash
In the sandbox environment, create and initialize a working directory from test fixtures:
has no changes after the migration without updating remote tfstate:
# tfmigrate plan tfmigrate_test.hcl
(snip.)
YYYY/MM/DD hh:mm:ss [INFO] [migrator] state migrator plan success!
# echo $?
0
The plan command computes a new state by applying state migration operations to a temporary state. It will fail if terraform plan detects any diffs with the new state. If you are wondering how the
tfmigrate
command actually works, you can see all
terraform
commands executed by the tfmigrate with log level
DEBUG
:
# TFMIGRATE_LOG=DEBUG tfmigrate plan tfmigrate_test.hcl
Available commands are:
apply Compute a new state and push it to remote state
plan Compute a new state
$ tfmigrate plan --help
Usage: tfmigrate plan [PATH]
Plan computes a new state by applying state migration operations to a temporary state.
It will fail if terraform plan detects any diffs with the new state.
Arguments:
PATH A path of migration file
Required in non-history mode. Optional in history-mode.
Apply computes a new state and pushes it to remote state.
It will fail if terraform plan detects any diffs with the new state.
Arguments
PATH A path of migration file
Required in non-history mode. Optional in history-mode.
Options:
--config A path to tfmigrate config file
Configurations
Environment variables
You can customize the behavior by setting environment variables.
TFMIGRATE_LOG
: A log level. Valid values are
TRACE
,
DEBUG
,
INFO
,
WARN
,
ERROR
. Default to
INFO
.
TFMIGRATE_EXEC_PATH
: A string how terraform command is executed. Default to
terraform
. It's intended to inject a wrapper command such as direnv. e.g.)
direnv exec . terraform
.
Some history storage implementations may read additional cloud provider-specific environment variables. For details, refer to a configuration file section for storage block described below.
Configuration file
You can customize the behavior by setting a configuration file.
The path of configuration file defaults to
(optional): Name of AWS profile in AWS shared credentials file or AWS shared configuration file to use for credentials and/or configuration. This can also be sourced from the
AWS_PROFILE
environment variable.
The following attributes are also available, but they are intended to use with
localstack
for testing.
endpoint
(optional): Custom endpoint for the AWS S3 API.
skip_credentials_validation
(optional): Skip credentials validation via the STS API.
You can write terraform state operations in HCL. The syntax of migration file is as follows:
A migration file must be written in the HCL2.
The extension of file must be
.hcl
(for HCL native syntax) or
.json
(for HCL JSON syntax).
Although the filename can be arbitrary string, note that in history mode unapplied migrations will be applied in alphabetical order by filename. It's possible to use a serial number for a filename (e.g.
123.hcl
), but we recommend you to use a timestamp as a prefix to avoid git conflicts (e.g.
The above example is written in HCL native syntax, but you can also write them in HCL JSON syntax.
This is useful when generating a migration file from other tools.
The first label is the migration type. There are two types of
migration
block,
state
and
multi_state
, and specify one of them.
The second label is the migration name, which is an arbitrary string.
The file must contain only one block, and multiple blocks are not allowed, because it's hard to re-run the file if partially failed.
migration block (state)
The
state
migration updates the state in a single directory. It has the following attributes.
dir
(optional): A working directory for executing terraform command. Default to
.
(current directory).
actions
(required): Actions is a list of state action. An action is a plain text for state operation. Valid formats are the following.
"mv "
"rm ...
"import "
force
(optional): Apply migrations even if plan show changes
Note that
dir
is relative path to the current working directory where
tfmigrate
command is invoked.
We could define strict block schema for action, but intentionally use a schema-less string to allow us to easily copy terraform state command to action.
Examples of migration block (state) are as follows.
We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.