Search
Close this search box.
Intro to Data Engineer's Lunch Managed Instance Groups with Terraform

Data Engineer’s Lunch #91: Deploying Google’s Managed Instance Groups with Terraform

GCP Managed Instance Groups and Terraform

Managed Instance Groups give you some features of Kubernetes with less complexity. We’ll show you how to deploy them with Terraform!

There are two types of instance groups in GCP, managed and unmanaged groups.

It is recommended to use managed instance groups (MIG) to benefit from the following set of features: 

  • Maintains a certain number of instances: If any of the machines goes down, the MIG will launch a new one immediately 
  • Self Healing: it runs some health checks and detects if any of the machines is not fully functional
  • Auto Scaling: the machine will take care of adding nodes to the group if your application requires more instances
  • Load balancer: This is used to distribute the load on the different machines in your group
  • Regional MIGs: this feature provides high availability, if you go with a regional MIG, you are enabling then you are multi zones and have higher availability compared to single-zoned MIGs.
  • Rolling Updates: releasing versions of your application with 0 downtime

So the advantages are: 

  • High Availability
  • Scalability
  • Automatic updates 
  • It also supports stateful workloads.

Terraform is capable of deploying and configuring all of these resources using several resource blocks.

  • The Managed Instance Group Itself
  • Custom Images or Default Images stored in Google’s Image repository

The Process

The process used in this demo takes an instance template, which can include a container pulled from Google’s Container Registry.

With the Instance Template, you can spin up an instance using that template and do development on that instance such as adding dependencies, configuring resources, or any edits you need.

Then create an image (note this demo will not work if you create “Machine Image” rather than an image due to how Terraform uses the “source_image” argument.

Setting Up Your Environment for the Demo

In order to run this demo or utilize the Google Cloud Platform to run managed instance groups, there are several things you MUST do.

Enable the APIs

In GCP, you have to enable the following:

  • Compute Engine API
  • Storage Engine API (if you want to save your tfstate file remotely)
  • Google Compute Engine Instance Group Manager API

and you will likely benefit from deploying these resources as well (though they aren’t strictly necessary.

  • Google Container Registry API (deprecated but still useful)
  • Artifact Registry API

Note that most of these APIs require billing to be enabled

Roles

Terraform will use a service account to deploy the resources.

Make sure that the service account has the following permissions:

  • MIG Manager
  • Compute Admin (Role)
    • Compute Image User
    • Compute Instance Admin (v1)
  • Storage Admin (Role)

The Key Google Resources

Container Registry

There are a huge variety of pre-prepared Containers in Google’s container registry that can be invoked by a few lines from a Terraform module.

Compute Engine

Compute Engine is where you can view a few useful resources.

  • The Instance from which you create your image.
  • The Instances deployed as part of the Managed Instance Group.

Compute Image / Storage / Images

There are a huge variety of pre-prepared images in Google’s existing repository. These can be used in your development.

Any image that you generate from an existing instance will also be stored in the existing

IAM and Admin

Use it to edit your permissions and add the permissions mentioned above.

Cloud Storage

You’ll use Cloud Storage to store your tfstate files remotely.

The Key Terraform Resources

The google_compute_instance_template

This is the template that the manager resource below will be duplicating and its configurations.

You can make this configuration from an image or an instance template.

The compute_instance_group_manager

This resource defines the group itself. The number and configuration of the instances can be defined here, along with specifications for the auto-healing and auto-scaling properties.

Looking to effortlessly manage your Google Cloud resources with Terraform? Don’t settle for complicated, time-consuming cloud management. Trust our team to handle the heavy lifting and optimize your cloud resources for maximum efficiency. Contact us today to learn more about how our services can benefit your business.

Resources for Further Study

  • https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_instance_group_manager
  • https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_instance_template
  • https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_region_instance_group_manager
  • https://github.com/terraform-google-modules/terraform-google-vm/blob/master/examples/mig/full/main.tf
  • https://github.com/terraform-google-modules/terraform-docs-samples/blob/main/storage/flask_google_cloud_quickstart/main.tf
  • https://cloud.google.com/compute/docs/images/create-custom
  • https://cloud.google.com/compute/docs/images
  • https://cloud.google.com/compute/docs/instance-groups
  • https://cloud.google.com/storage/docs/access-control/using-iam-permissions

Anant’s Reference Video and Code

Watch the video associated with this blog here:

View the code for this blog here:
https://github.com/Anant/example-managed-instance-groups-w-terraform

Leave a Comment

Your email address will not be published. Required fields are marked *