Terraform - Setting Up Private Endpoints for Azure Storage Accounts

Hi folks, I hope everyone is doing well. We all use Azure Storage Accounts to keep our data, like files and blobs. And it’s super important to keep that data safe.

With Azure’s Private Endpoints, we can turn public services, which are open by default, into private ones inside our Azure network. Our Storage Accounts connect safely to a virtual network; only the right people or services can see the data.

In this article, I’ll show you how to set up these Private Endpoints using Terraform, a tool I like for managing cloud stuff. By the end, your Azure Storage will be private and secure. Let’s get to it!

Prerequisites #

• You need Terraform CLI on your local machine, if you’re new to using Terraform to deploy Microsoft Azure resources, then I recommend you check out this link.
• A text editor or IDE of your choice (Visual Studio Code with Terraform extension is my recommendation)

Declare Azure Provider in Terraform #

The provider.tf file in Terraform is used to specify and configure the providers used in your Terraform configuration. A provider is a service or platform where the resources will be managed. This could be a cloud provider like Microsoft Azure, AWS, Google Cloud, etc.

This file is important because it tells Terraform which provider’s API to use when creating, updating, and deleting resources. Without it, Terraform wouldn’t know where to manage your resources.

provider "azurerm" {
  features {}
}

Deploy Azure Resources Using Terraform #

In the case of setting up Azure Private Endpoints for Storage Accounts, the main.tf file contains these key components:

• locals: This block maps domain names for specific storage services.
• data “azurerm_resource_group” “existing”: This data block fetches details about the existing resource group where the private endpoints will be deployed.
• data “azurerm_virtual_network” “existing”: Retrieves the specifications of the virtual network where the private endpoints will connect.
• data “azurerm_subnet” “existing”: Collects information about the subnet where the private endpoints will reside.
• azurerm_private_dns_zone: Sets up private DNS zones, ensuring the storage account endpoints are resolved within the private network.
• azurerm_private_dns_zone_virtual_network_link: Establishes the connection between the private DNS zones and the virtual network.
• azurerm_private_endpoint: This is the heart of the module, creating private endpoints in the specified subnets and connecting them to the target storage resources. Additionally, it links the private endpoints to the appropriate DNS zones.
• azurerm_private_dns_a_record: Creates ‘A’ records in the private DNS zones, pointing to the private endpoints.

// Using locals to define a mapping of service domains for easy reference.
locals {
  service_domains = {
    blob  = "privatelink.blob.core.windows.net",
    table = "privatelink.table.core.windows.net",
    queue = "privatelink.queue.core.windows.net",
    file  = "privatelink.file.core.windows.net"
  }
}

// Fetch details of the existing resource group where private endpoints will be created.
data "azurerm_resource_group" "existing" {
  for_each = var.private_endpoints
  name     = each.value.resource_group_name
}

// Fetch details of the existing virtual network where private endpoints will connect.
data "azurerm_virtual_network" "existing" {
  for_each            = var.private_endpoints
  name                = each.value.virtual_network_name
  resource_group_name = each.value.resource_group_name
}

// Fetch details of the subnet where private endpoints will reside.
data "azurerm_subnet" "existing" {
  for_each             = var.private_endpoints
  name                 = each.value.subnet_name
  virtual_network_name = each.value.virtual_network_name
  resource_group_name  = each.value.resource_group_name
}

// Create private DNS zones based on the service domains.
resource "azurerm_private_dns_zone" "pdnszone" {
  for_each            = var.private_endpoints
  name                = local.service_domains[each.value.private_service_subresource_name]
  resource_group_name = each.value.resource_group_name
  tags                = each.value.tags
}

// Establish the link between private DNS zones and the virtual network.
resource "azurerm_private_dns_zone_virtual_network_link" "pdnszlink" {
  for_each              = var.private_endpoints
  name                  = "link-${each.key}"
  resource_group_name   = each.value.resource_group_name
  private_dns_zone_name = azurerm_private_dns_zone.pdnszone[each.key].name
  virtual_network_id    = data.azurerm_virtual_network.existing[each.key].id
  tags                  = each.value.tags
}

// Create private endpoints in the specified subnets and connect to the target resources.
resource "azurerm_private_endpoint" "pep" {
  for_each            = var.private_endpoints
  name                = "pep-${each.key}"
  location            = data.azurerm_resource_group.existing[each.key].location
  resource_group_name = each.value.resource_group_name
  subnet_id           = data.azurerm_subnet.existing[each.key].id

  private_service_connection {
    name                           = "connection-${each.key}"
    is_manual_connection           = false
    private_connection_resource_id = each.value.resource_id_to_link
    subresource_names              = [each.value.private_service_subresource_name]
  }

  // Link private endpoints to the respective DNS zones.
  private_dns_zone_group {
    name                 = "pdnszgroup-${each.key}"
    private_dns_zone_ids = [azurerm_private_dns_zone.pdnszone[each.key].id]
  }

  tags = each.value.tags
}

// Create A records in private DNS zones for the private endpoints.
resource "azurerm_private_dns_a_record" "pdnsrecord" {
  for_each            = var.private_endpoints
  name                = "pep-${each.key}"
  resource_group_name = each.value.resource_group_name
  zone_name           = azurerm_private_dns_zone.pdnszone[each.key].name
  ttl                 = 300
  records             = [azurerm_private_endpoint.pep[each.key].private_service_connection[0].private_ip_address]
  tags                = each.value.tags
}

Declaration of input variables #

In Terraform, the variables.tf file is essential to make our configurations. It allows us to define important parameters to be used in our main.tf. By using variables, we can create versatile configurations that can adapt to different scenarios.

In this example, the variables are defined in the variables.tf include:

• private_endpoints: This complex variable is designed to capture an array of attributes essential for each private endpoint’s creation:
  • resource_group_name: The name of the Azure resource group where the private endpoint will be created.
  • virtual_network_name: The name of the virtual network in Azure to which the private endpoint will be connected.
  • subnet_name: Specifies the particular subnet within the virtual network where the private endpoint will reside.
  • resource_id_to_link: The unique Azure resource ID of the storage service to which the private endpoint should connect.
  • private_service_subresource_name: This determines the specific Azure storage service (like blob, file, queue, or table) that the endpoint should connect to.
  • tags: A map of metadata tags to assign to the private endpoint.

This structured design ensures each private endpoint is correctly associated with its designated storage service and is set up appropriately.

// Define a variable for configuring private endpoints
variable "private_endpoints" {
  description = "Details for the Private Endpoints creation"
  type = map(object({
    resource_group_name              = string
    virtual_network_name             = string
    subnet_name                      = string
    resource_id_to_link              = string
    private_service_subresource_name = string
    tags                             = map(string)
  }))
  default = {}

  validation {
    condition = alltrue([
      for ep in values(var.private_endpoints) :
      contains(["blob", "file", "queue", "table"], ep.private_service_subresource_name)
    ])
    error_message = "Valid values for private_service_subresource_name are: blob, file, queue, table."
  }
}

Validation Block: This is our guardrail. It checks the private_service_subresource_name attribute to ensure the service name provided aligns with one of the predefined accepted values: “blob”, “file”, “queue”, or “table”. Such validations catch potential misconfigurations early on, ensuring the deployment process is seamless and error-free.

Declaration of output values #

The output.tf file in Terraform extracts and displays information about the resources created or managed by your Terraform configuration. These outputs are defined using the output keyword and can be used to return information that can be useful for the user, for other Terraform configurations, or for programmatically using the information in scripts or other tools.

In this example, the output.tf file returns the following information

• The IDs of the created private endpoints
• The private IP addresses linked to these endpoints
• The fully qualified domain names (FQDNs) for the associated DNS A records

After Terraform successfully applies your configuration, it will showcase your defined outputs. This provides instant access to key attributes of the resources you’ve set up.

// Output Block for Storage Account Private Endpoints
output "private_endpoints_output" {
  description = "Attributes of created private endpoints and DNS records."
  value = {
    // IDs of private endpoints.
    ids = { for key, pe in azurerm_private_endpoint.pep : key => pe.id },

    // IP addresses of private endpoints.
    ips = { for key, pe in azurerm_private_endpoint.pep : key => pe.private_service_connection[0].private_ip_address },

    // FQDNs of DNS A records.
    dns = { for key, dns in azurerm_private_dns_a_record.pdnsrecord : key => dns.fqdn }
  }
}

Executing the Terraform Deployment #

Now that you’ve declared the resources correctly, it’s time to take the following steps to deploy them in your Azure environment.

• Initialization: To begin, execute the terraform init command. This will initialize your working directory that holds the .tf files and download the provider specified in the provider.tf file, and configure the Terraform backend. I suggest looking at this link if you’re curious about the process.

• Planning: Next, execute the terraform plan. This command creates an execution plan and shows Terraform’s actions to achieve the desired state defined in your .tf files. This gives you a chance to review the changes before applying them.

• Apply: When you’re satisfied with the plan, execute the terraform apply command. This will implement the required modifications to attain the intended infrastructure state. Before making any changes, you will be asked to confirm your decision.

• Inspection: After applying the changes, you can use terraform show command to see the current state of your infrastructure.

• Destroy (optional): when a project is no longer needed or when resources have become outdated. You can use the terraform destroy command. This will remove all the resources that Terraform has created.

References and useful links #

• Official Azure Documentation

• Official Terraform Documentation

• Another Article from My Archive

Thank you for taking the time to read my post. I sincerely hope that you find it helpful.