Terraform Providers for Deploying Kafka

What is Kafka?

Apache Kafka is a distributed event streaming platform that is used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. Kafka is designed to handle real-time data feeds with low latency and high throughput. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies. Kafka is able to handle trillions of events per day, providing an ideal platform for large-scale message-processing applications.

How does Kafka work?

• Kafka maintains feeds of messages in categories called topics.
• Producers write data to topics and consumers read from topics.
• Since Kafka is a distributed system, topics are partitioned and replicated across multiple nodes.

What is Terraform?

Terraform is an open-source infrastructure as code software tool that provides a consistent CLI workflow to manage hundreds of cloud services. Terraform codifies APIs into declarative configuration files. This allows a blueprint of your data center to be versioned and treated as you would any other code.

Terraform Providers for Kafka

Optimizing your infrastructure for cloud-based services can be significantly streamlined by transforming your data center into a version-controlled system that treats infrastructure as code (IaC). Terraform is an excellent tool that enables this transformation, providing you the power to manage resources programmatically.

There’s an array of Terraform providers specifically designed for managing and deploying Apache Kafka clusters, the powerful distributed event streaming platform. Here, we walk you through some of the most sought-after ones:

Custom Terraform Providers for Kafka

1. Confluent Cloud Provider: Harnessing the power of open-source Apache Kafka, Confluent Cloud has emerged as a leading fully-managed streaming data service. It comes with its very own Terraform provider that lets you automate the management of Confluent Cloud resources, opening up a whole new way of managing your resources effectively and efficiently.

2. AWS MSK (Managed Streaming for Apache Kafka): Amazon Web Services (AWS) offers a fully managed Kafka service, equipped with a dedicated Terraform provider. By using the aws_msk_cluster resource, you can conveniently define and provision an entire Kafka cluster using Terraform, bridging the gap between infrastructure management and your application needs.

3. Azure HDInsight Kafka: Microsoft Azure also provides a ‘Kafka as a Service’ offering. Its Azure Resource Manager (AzureRM) provider for Terraform helps streamline the management of your Kafka clusters, making it easier to focus on building applications that meet your business needs.

4. Instaclustr Terraform Provider: As a provider of Apache Kafka as a managed service, Instaclustr has its own open-sourced Terraform provider. With this, you can effortlessly manage resources within your Instaclustr account, improving the way you deploy and control your Kafka clusters.

5. Aiven Provider: Aiven, known for its managed open-source database and messaging service, offers robust support for Apache Kafka. Their Terraform provider simplifies the management of Aiven project resources, making it an excellent choice for those who embrace infrastructure as code. With this provider, you can manage an array of Aiven services, including Kafka, directly via your Terraform scripts.

These providers offer significant relief from the complexities of setting up and managing Kafka clusters natively. They offer varied configuration options to fine-tune your Kafka cluster performance, easing the burden for developers who leverage Kafka in their applications. By leveraging these Terraform providers, you can detail the desired state of your Kafka clusters using a declarative language, allowing Terraform to carry out the necessary steps to reach that state. This process vastly simplifies Kafka cluster management tasks, particularly beneficial in large-scale organizations or for intricate applications.

Code Examples for Terraform and Kafka

By integrating a specific provider into your Terraform setup, you unlock the capability to define, configure, and manage your Kafka clusters as code. This process entails defining a provider-specific Kafka cluster resource in your Terraform scripts, thereby specifying the properties of your desired Kafka service. The power of Terraform combined with your chosen provider then handles the heavy lifting, automating the deployment and management of your Kafka clusters. In the following sections, we’ll guide you through a step-by-step description of implementing a provider and defining a resource for any of these Kafka services, making your transition to infrastructure as code a smooth journey.

The Provider Declaration

Declaring a provider in Terraform is an essential step that informs the tool about the infrastructure type you’re going to create and manage. Each provider offers a predefined set of resources that are relevant to a specific service or platform, such as AWS, Azure, Google Cloud, or Apache Kafka services, to name a few. By declaring a provider, you instruct Terraform on how to interact with the APIs of the chosen platform, facilitating the creation, management, and manipulation of those resources. This declaration sets the foundation for your infrastructure as code (IaC), allowing Terraform to understand the context of your infrastructure and execute the appropriate actions to achieve the desired state of your resources.

The Provider Declaration for Confluent Cloud

provider "confluentcloud" {
  api_key    = "your-api-key"
  api_secret = "your-api-secret"
}

The Provider Declaration for AWS MSK (Managed Streaming for Apache Kafka)

provider "aws" {
  region = "us-west-2"
  access_key = "your-access-key"
  secret_key = "your-secret-key"
}

The Provider Declaration for Azure HDInsight Kafka

provider "azurerm" {
  features {}
}

The Provider Declaration for Instaclustr

provider "instaclustr" {
  username = "your-username"
  api_key  = "your-api-key"
  url      = "https://api.instaclustr.com/provisioning"
}

The Provider Declaration for Aiven

provider "aiven" {
  api_token = "your-api-token"
}

Similarities and Differences between the Provider Declaration

Terraform providers, regardless of the platform or service they target, all follow a similar basic structure in their definitions. The commonalities and differences typically center around the required parameters that each provider needs in order to interact with its respective platform.

Similarities:

1. Provider Keyword: Each provider declaration begins with the provider keyword, followed by the name of the provider.
2. Configuration Attributes: All provider declarations will include a set of attributes to configure the provider. These attributes are usually in the form of API keys, tokens, or credentials required to authenticate with the service.
3. Terraform Block Structure: Terraform uses a declarative language with a specific block structure. Each block begins with a keyword like provider, followed by a name and then a block of configurations enclosed in {}. All provider declarations follow this pattern.

Differences:

1. Provider-Specific Attributes: Each provider will have a different set of attributes specific to that provider. For instance, the AWS provider will need AWS-specific credentials (like access key and secret key), while the Azure provider will require Azure-specific credentials.
2. Resource Types: The types of resources that can be created and managed will differ between providers. While the AWS provider may offer resources like aws_instance for managing EC2 instances, the Azure provider offers resources like azurerm_virtual_machine for managing VM instances.
3. API Interaction: How each provider interacts with its respective service’s API will also differ. For instance, the Aiven provider may interact with Aiven’s RESTful API, while the AWS provider interacts with the AWS API.
4. Region or Zone Specification: Some providers require you to specify the region or zone where your resources will be created, like AWS and Azure. Others, like Aiven, may not need this information because it’s specified when you create individual resources.

The Resource Declarations

Actually declaring the Kafka cluster resource comes after the

The Resource Declaration for Confluent Cloud

resource "confluentcloud_environment" "env" {
  name = "terraform-provider-confluentcloud"
}

resource "confluentcloud_kafka_cluster" "cluster" {
  name         = "terraform-provider-confluentcloud"
  environment  = confluentcloud_environment.env.id
  deployment   = "Basic"
  region       = "us-west-2"
  availability = "LOW"
  service_type = "kafka"
}

The Resource Declaration for AWS MSK (Managed Streaming for Apache Kafka)

resource "aws_msk_cluster" "example" {
  cluster_name = "example"
  kafka_version = "2.4.1"
  number_of_broker_nodes = 3

  broker_node_group_info {
    instance_type   = "kafka.m5.large"
    ebs_volume_size = 1000
    client_subnets  = ["subnet-abcde012", "subnet-bcde012a", "subnet-fghi345a"]
    security_groups = ["sg-abcde012"]
  }
}

The Resource Declaration for Azure HDInsight Kafka

resource "azurerm_hdinsight_kafka_cluster" "example" {
  name                = "example"
  resource_group_name = "example"
  location            = "West US 2"
  cluster_version     = "3.6"
  tier                = "Standard"

  component_version {
    kafka = "2.1"
  }

  gateway {
    enabled  = true
    username = "acctestusg"
    password = "TerrAform123!"
  }

  storage_account {
    storage_container_id = azurerm_storage_container.example.id
    storage_account_key  = azurerm_storage_account.example.primary_access_key
  }

  roles {
    head_node {
      vm_size  = "A6"
      username = "acctesthn"
      password = "TerrAform123!"
    }

    worker_node {
      vm_size               = "A6"
      username              = "acctestwn"
      password              = "TerrAform123!"
      target_instance_count = 3
    }

    zookeeper_node {
      vm_size  = "A6"
      username = "acctestzn"
      password = "TerrAform123!"
    }
  }
}

The Resource Declaration for Instaclustr

resource "instaclustr_cluster" "example" {
  cluster_name = "example"
  node_size = "t3.small"
  data_centre = "US_WEST_2"
  sla_tier = "NON_PRODUCTION"
  cluster_network = "192.168.0.0/18"
  private_network_cluster = false
  cluster_provider = {
    name = "AWS_VPC"
  }
  rack_allocation = {
    number_of_racks = 3
    nodes_per_rack = 1
  }

  bundle {
    bundle = "KAFKA"
    version = "2.4.0"
    options = {
      client_encryption = false
    }
  }
}

The Resource Declaration for Aiven

data "aiven_project" "tf" {
  project = "your-project"
}

resource "aiven_kafka" "tf" {
  project = data.aiven_project.tf.project
  cloud_name = "aws-us-east-1"
  plan = "business-4"
  service_name = "tf-test"
  kafka_version = "2.8"

  kafka_user_config {
    kafka_connect = false
    kafka_rest = false
    schema_registry = false
  }
}

These are basic configurations and don’t include everything you might need for a real deployment. Make sure you replace the placeholder values like "your-api-key" with your actual API keys, and check the documentation for each provider to see all the available options.

Please be aware that it is a bad practice to hard-code secrets (like api_key or password) directly into your Terraform scripts. You can use environment variables or Terraform variables to inject these values at runtime. And always keep your Terraform scripts in a secure, private repository if they contain sensitive data.

Conclusion

As we’ve navigated through the realm of Apache Kafka cluster management using Terraform, it’s evident that the process is streamlined and made much more efficient with the use of Terraform providers and resource declarations. Providers like AWS MSK, Azure HDInsight Kafka, Aiven, Confluent Cloud, or Instaclustr serve as the vital bridge between Terraform and their respective cloud-based Kafka services.

Resource declarations, on the other hand, lay down the blueprint of your Kafka cluster. They encapsulate all the configuration details necessary to build and manage your Kafka cluster on the specified platform. This clear, declarative style allows developers and DevOps teams to define the desired state of their infrastructure, and Terraform works its magic to realize this state.

Terraform, with its providers and resource declarations, truly embodies the concept of ‘Infrastructure as Code’, enabling efficient management of resources while promoting consistency and reliability. Whether you’re a small business owner, a large organization, or anything in between, managing your Kafka clusters with Terraform providers can take a load off your plate, allowing you to focus more on building powerful, scalable applications. So, embrace the power of IaC, and transform the way you manage your Kafka clusters with Terraform.

About Anant

As we’ve navigated the exciting world of Apache Kafka management with Terraform in this blog post, you might be eager to start implementing these best practices in your own infrastructure. If that’s the case, Anant is here to help. Our mission is to help businesses, like yours, modernize and maintain their data platforms.

We’re not just a Cassandra consulting firm; we empower our clients to succeed with the latest and most effective technology, providing them with comprehensive solutions for their biggest data challenges. Our expertise in the data engineering landscape is vast, extending well beyond Cassandra to include services such as Apache Kafka and other modern data tools. Reach out to us now to learn more!