Lessons Learned from Using Terraform to Manage Snowflake in Production

During our migration to Snowflake as our cloud data warehouse, we sought an automated and scalable approach to manage our Snowflake infrastructure & architecture. We chose Terraform due to its widespread adoption & official support from Snowflake. However, this decision introduced new challenges.

In this post, I will share some of the lessons I learned during the project:

Always maintain a separate Snowflake development account

When we began the migration, we used a single Snowflake account for the production environment. This was fine initially since we didn't have any live data yet. However, once we went live, making changes directly in the production account became riskier. No matter how many times you checked and confirmed the "terraform plan" output, there was always that anxiety: "What if I missed something that will be destroyed?". Therefore, introducing the development environment was an easy decision, and the best part is that it also came at almost no extra costs. In the development account, we have no live data or ELT processes running. It’s just used for infrastructure and architectural development tasks in Terraform.

Use Terragrunt

Terragrunt is a wrapper around Terraform that simplifies managing multi-environment Terraform setups. We have a single Terraform directory in our project where the Terraform .tf files are located. This directory is shared across environments. In addition, each environment has its own directory with .yml configuration files. These .yml files contain high-level settings, allowing users to easily configure databases, schemas, RBAC assignments, and more without needing to handle complex Terraform .tf files.

$ terragrunt plan --terragrunt-working-dir=environments/development

Don’t use import or mv Terraform commands

The import & mv commands are irreversible and you cannot version-control what was executed. You should always avoid these 2 commands. Instead, it’s recommended to use import & moved blocks within Terraform .tf files.

Keep the Terraform Snowflake provider version up-to-date

As Snowflake introduces new features, it’s important to keep updating the Snowflake provider in Terraform to have these new features accessible to your account and simultaneously avoid Terraform state drift, where the state in Terraform is different than your real-world architecture.

Use prevent_destroy wisely

Prevent_destroy is a Terraform lifecycle boolean argument. If you set it to true for a resource, it will avoid destroying this resource when you execute “terraform plan” or “terraform apply”. This is really important for resources that we cannot afford to destroy by mistake in Terraform, for example: Snowflake databases, schemas, and stages.

On-board the team

Probably the most important lesson! The team should be using Terraform to manage Snowflake architecture & infrastructure comfortably. It’s ok to have some members only doing small & direct .yml changes (Terraform is not a top priority topic for Data Engineers, especially at a junior level) but you must also fully onboard other team members to be able to resolve any issue that might appear from state locks to more complex Terraform errors and to further improve the project and avoid a “single point of failure”.

Conclusion

Terraforming Snowflake is not an easy task, but it's important to remember the benefits of using Terraform. With Terraform, we automate the RBAC process, maintain proper version control, and avoid the confusion that can occur when database admins run commands and lose track in the query history.