Terraform: Create a Kubernetes Cluster

In this tutorial, you use Terraform to set up a Kubernetes cluster in your Oracle Cloud Infrastructure account.

Key tasks include how to:

  • Copy your existing scripts from the Terraform Get Started tutorials.
  • Edit existing scripts for reuse.
  • Write new scripts for a Kubernetes cluster.
A diagram of the components needed to create a Kubernetes cluster with Terraform. From a local Linux environment, the user connects to an Oracle Cloud Infrastructure account and creates a cluster. The cluster creation includes creating network resources. The network resources include one private regional subnet for worker nodes. These nodes are compute nodes. There is also a public regional subnet created to use for load balancers. Each regional subnet spans three availability domains: AD1, AD2, and AD3. The public subnet is connected to the internet with a two-way arrow. The private subnet has two one-directional, outgoing arrows, one to the internet and the other to Oracle services network.

For additional information, see:

1. Gather Required Information

Gather information for the compute instances in the node pool.

Get Node Shape
This tutorial uses VM.Standard2.1 for the compute instances in the node pool..
  1. Save the <node-shape>, VM.Standard2.1 in your notepad.
  2. Go to VM Standard Shapes to learn more about the shape.
Get Image ID
  1. In the Console's top navigation bar, find your region.
  2. Go to Oracle Provided Images.
  3. Click Oracle Linux 7.x.
  4. Click the latest Oracle Linux 7.x-<date>. Don't select any images labeled Gen2-GPU.
  5. Copy the Image OCID for your region.
  6. Save the <image-ocid> in your notepad.
    Note

    Ensure that you select a commercial OCID without gov in its OCID.

2. Copy Existing Scripts

Copy the Terraform: Set Up a Simple Infrastructure tutorial scripts. Then, remove the scripts and outputs related to the compute instance. In the next section, you declare a node pool with compute instances.

Copy the Scripts
  1. In your $HOME directory, create a directory called tf-cluster and change to that directory.
    mkdir tf-cluster
    cd tf-cluster
  2. Copy all the files ending in .tf from the tf-simple-infrastructure directory.
    cp ../tf-simple-infrastructure/*.tf .
  3. Confirm that you have the following files in your directory.
    ls
    availability-domains.tf
    compartment.tf
    compute.tf
    outputs.tf
    private-security-list.tf
    private-subnet.tf
    provider.tf
    public-security-list.tf
    public-subnet.tf
    vcn-module.tf
    Note

    Do not copy the terraform.tfstate and terraform.tfstate.backup files. They contain the state of the resources in the directory that they are located. You will get a new state file, after you run the scripts in your new directory.
Remove Irrelevant Scripts
  1. Remove the compute.tf file from the tf-cluster directory.
    rm compute.tf
  2. In the outputs.tf file, remove all the outputs for the compute instance.
    
    
    # Outputs for compute instance
    
    output "public-ip-for-compute-instance" {
      value = oci_core_instance.ubuntu_instance.public_ip
    }
    output "instance-name" {
      value = oci_core_instance.ubuntu_instance.display_name
    }
    
    output "instance-OCID" {
      value = oci_core_instance.ubuntu_instance.id
    }
    
    output "instance-region" {
      value = oci_core_instance.ubuntu_instance.region
    }
    
    output "instance-shape" {
      value = oci_core_instance.ubuntu_instance.shape
    }
    
    output "instance-state" {
      value = oci_core_instance.ubuntu_instance.state
    }
    
    output "instance-OCPUs" {
      value = oci_core_instance.ubuntu_instance.shape_config[0].ocpus
    }
    
    output "instance-memory-in-GBs" {
      value = oci_core_instance.ubuntu_instance.shape_config[0].memory_in_gbs
    }
    
    output "time-created" {
      value = oci_core_instance.ubuntu_instance.time_created
    }

3. Create Scripts

Create three scripts: one for a cluster, one for a node pool and one to print outputs.

Declare a Cluster
  1. Create a file called cluster.tf.
  2. Add the following code to cluster.tf.
    • Replace <your-cluster-name> with a name of your choice. Example: tf-cluster.
    # Source from https://registry.terraform.io/providers/oracle/oci/latest/docs/resources/containerengine_cluster
    
    resource "oci_containerengine_cluster" "oke-cluster" {
        # Required
        compartment_id = oci_identity_compartment.tf-compartment.id
        kubernetes_version = "v1.21.5"
        name = "<your-cluster-name>"
        vcn_id = module.vcn.vcn_id
    
        # Optional
        options {
            add_ons{
                is_kubernetes_dashboard_enabled = false
                is_tiller_enabled = false
            }
            kubernetes_network_config {
                pods_cidr = "10.244.0.0/16"
                services_cidr = "10.96.0.0/16"
            }
            service_lb_subnet_ids = [oci_core_subnet.vcn-public-subnet.id]
        }  
    }
  3. Save the cluster.tf file.
Explanation
  • Go to Oracle Cloud Infrastructure Provider.
  • In the left navigation Filter, enter container engine.
  • Under Container Engine, go to Resources and click oci_containerengine_cluster.
  • Find the Resource Type from the title of the page:
    • Type: oci_containerengine_cluster
  • In the Argument Reference section, find all arguments (inputs) labeled as (Required)
    • compartment_id
    • kubernetes_version
    • name
    • vcn_id
  • Construct a resource block:
    • Declare a resource block with the keyword: resource
    • Add a label for resource type: "oci_containerengine_cluster"
    • Add a label for a local name (your choice):
      • The label can contain letters, digits, underscores (_), and hyphens (-). The first character must not be a digit.
      • Example: "oke-cluster"
    • Inside the code block, provide a value for the required arguments. They don't have a default value:
      • compartment_id: Point to the compartment declared in compartment.tf: oci_identity_compartment.tf-compartment.id
      • kubernetes_version: This tutorial uses version v1.21.5. You can check the Quick Create option in the Console for the latest version.
      • name: Assign a name of your choice.
      • vcn_id: Point to the compartment declared in vcn-module.tf: module.vcn.vcn_id
    • Provide values for the following optional arguments to override their default values.
      • add_ons: Assign true or false for the following arguments:
        • is_kubernetes_dashboard_enabled
        • is_tiller_enabled
      • kubernetes_network_config: Assign a CIDR block as a string for the following arguments:
        • pods_cidr
        • services_cidr
        Note

        The CIDR block for the pods must not overlap with the worker node and load balancer subnet CIDR blocks.

        The CIDR block for the Kubernetes service must not overlap with the VCN CIDR block.

        The example code in this tutorial uses the same CIDR blocks as the Quick Create option in the Console.

        For more explanation, see CIDR Blocks and Kubernetes Engine.

      • service_lb_subnet_ids: Assign the public subnet you declared in public-subnet.tf
        Note

        The argument, service_lb_subnet_ids accepts a list of subnet ids:
        • Even if you have one subnet, use square brackets to denote a list.
        • Example: [oci_core_subnet.vcn-public-subnet.id]
Declare a Node Pool
  1. Create a file called node-pool.tf.
  2. Add the following code to node-pool.tf.
    • Replace the following fields with the information you gathered in section one:
      • <node-shape> with VM.Standard2.1
      • <image-ocid>
    • Replace the following field with the name you chose when you declared a cluster:
      • <your-cluster-name>
    # Source from https://registry.terraform.io/providers/oracle/oci/latest/docs/resources/containerengine_node_pool
    
    resource "oci_containerengine_node_pool" "oke-node-pool" {
        # Required
        cluster_id = oci_containerengine_cluster.oke-cluster.id
        compartment_id = oci_identity_compartment.tf-compartment.id
        kubernetes_version = "v1.21.5"
        name = "pool1"
        node_config_details{
            placement_configs{
                availability_domain = data.oci_identity_availability_domains.ads.availability_domains[0].name
                subnet_id = oci_core_subnet.vcn-private-subnet.id
            } 
            placement_configs{
                availability_domain = data.oci_identity_availability_domains.ads.availability_domains[1].name
                subnet_id = oci_core_subnet.vcn-private-subnet.id
            }
             placement_configs{
                availability_domain = data.oci_identity_availability_domains.ads.availability_domains[2].name
                subnet_id = oci_core_subnet.vcn-private-subnet.id
            }
            size = 3
        }
        node_shape = "<node-shape>"
    
        # Using image Oracle-Linux-7.x-<date>
        # Find image OCID for your region from https://docs.oracle.com/iaas/images/ 
        node_source_details {
             image_id = "<image-ocid>"
             source_type = "image"
        }
     
        # Optional
        initial_node_labels {
            key = "name"
            value = "<your-cluster-name>"
        }    
    }
  3. Save the node-pool.tf file.
Explanation
  • Go to Oracle Cloud Infrastructure Provider.
  • In the left navigation Filter, enter container engine.
  • Under Container Engine, go to Resources and click oci_containerengine_node_pool.
  • Find the Resource Type from the title of the page:
    • Type: oci_containerengine_node_pool
  • In the Argument Reference section, find all arguments (inputs) labeled as (Required)
    • cluster_id
    • compartment_id
    • kubernetes_version
    • name
    • node_config_details
      • placement_configs
        • availability_domain
        • subnet_id
    • node_shape
    • node_source_details
      • image_id
      • source_type
  • Construct a resource block:
    • Declare a resource block with the keyword: resource
    • Add a label for resource type:
      "oci_containerengine_node_pool"
    • Add a label for a local name (your choice):
      • The label can contain letters, digits, underscores (_), and hyphens (-). The first character must not be a digit. Example:
        "oke-node-pool"
    • Inside the code block, provide a value for the required arguments:
      • cluster_id: Point to the cluster declared in cluster.tf:
        oci_containerengine_cluster.oke-cluster.id
      • compartment_id Point to the compartment declared in compartment.tf:
        oci_identity_compartment.tf-compartment.id
      • kubernetes_version: This tutorial uses the same version as the Create Cluster wizard in the Console.
      • name: Assign a name of your choice. The Create Cluster wizard, uses the name pool1.
      • node_shape: Enter information you gathered in section one.
      • node_source_details:
        • image_id: Enter information you gathered in section one.
        • source_type: Set to "image".
    • Provide values for the following optional arguments to override their default values.
      • initial_node_labels: Assign key/value pairs for the nodes.
        • key: Assign a key of your choice. The Console's Quick Create option creates the key "name".
        • value: Assign a value for the key. The Console's Quick Create option assigns "<your-cluster-name>" to the "name" key.
Add Outputs

In this section, you declare outputs for the cluster and the node pool.

  1. Add the following code to outputs.tf.
    # Outputs for k8s cluster
    
    output "cluster-name" {
      value = oci_containerengine_cluster.oke-cluster.name
    }
    output "cluster-OCID" {
      value = oci_containerengine_cluster.oke-cluster.id
    }
    output "cluster-kubernetes-version" {
      value = oci_containerengine_cluster.oke-cluster.kubernetes_version
    }
    output "cluster-state" {
      value = oci_containerengine_cluster.oke-cluster.state
    }
    
    # Outputs for k8s node pool
    
    output "node-pool-name" {
      value = oci_containerengine_node_pool.oke-node-pool.name
    }
    output "node-pool-OCID" {
      value = oci_containerengine_node_pool.oke-node-pool.id
    }
    output "node-pool-kubernetes-version" {
      value = oci_containerengine_node_pool.oke-node-pool.kubernetes_version
    }
    output "node-size" {
      value = oci_containerengine_node_pool.oke-node-pool.node_config_details[0].size
    }
    output "node-shape" {
      value = oci_containerengine_node_pool.oke-node-pool.node_shape
    }
  2. Save the outputs.tf file.
Explanation
Outputs for List Items
  • Usually list attributes are plural: (end in s).
  • List attribute example for Node Pool:
    • node_config_details
  • To output all the attributes in a list, use the list attribute by itself, without any brackets.
  • Example:
    output "node-configuration-details"{
      value = oci_containerengine_node_pool.oke-node-pool.node_config_details
    }

    Sample output:

    node-configuration-details = [
      {
        "placement_configs" = [
          {
            "availability_domain" = "QnsC:US-ASHBURN-AD-1"
            "subnet_id" = "ocid1.subnet.xxx"
          },
          {
            "availability_domain" = "QnsC:US-ASHBURN-AD-2"
            "subnet_id" = "ocid1.subnet.xxx"
          },
          {
            "availability_domain" = "QnsC:US-ASHBURN-AD-3"
            "subnet_id" = "ocid1.subnet.xxx"
          },
        ]
        "size" = 3
      },
    ]
  • To output or call an item from a list:
    • Use the following format:

      <list-attribute-name>[index].<attribute-from-list>

    • Replace [index] with:
      • [0] for the first item.
      • [1] for the second item.
      • ...
      • [n] for the (n+1)th item.
  • Example:

    Value for the size attribute:

    output "node-size" {
      value = oci_containerengine_node_pool.oke-node-pool.node_config_details[0].size
    }

4. Run Scripts

Run your Terraform scripts to create a compartment, a virtual cloud network, a Kubernetes cluster, and a node pool.

Initialize
  1. Initialize a working directory in the tf-cluster directory.
    terraform init

    Example output:

    Initializing the backend...
    
    Initializing provider plugins...
    - Finding hashicorp/oci versions matching ">= 4.41.0"...
    - Installing hashicorp/oci v4.59.0...
    - Installed hashicorp/oci v4.59.0 (signed by HashiCorp)
    
    Terraform has been successfully initialized!
  2. Check the contents of the tf-cluster directory.
    ls -a

    You now have a folder called .terraform that includes the plugins for the oci provider.

    Note

    Troubleshooting:
    • After running terraform init
    • error message: Failed to query available provider packages:
      • If you are on a VPN, check your proxy settings.
Plan
  1. Create an execution plan:
    terraform plan
  2. Review the changes that Terraform plans to make to your account.

    Example output:

    Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
      + create
    
    Terraform will perform the following actions:
    
    Plan: 14 to add, 0 to change, 0 to destroy.
Apply
  1. Run your Terraform scripts:
    terraform apply
  2. When prompted for confirmation, enter yes, for your resources to be created.
  3. It might take 15 minutes or more for the cluster to be created. After Terraform creates the resources, review the output in the terminal.
    Apply complete! Resources: 14 added, 0 changed, 0 destroyed.
    
    Outputs:
    ...
    cluster-OCID = ocid1.cluster.xxx
    cluster-kubernetes-version = "v1.21.5"
    cluster-name = <your-cluster-name>
    cluster-state = ACTIVE
    ...
    node-pool-OCID = ocid1.nodepool.xxx
    node-pool-kubernetes-version = "v1.21.5"
    node-pool-name = "pool1"
    node-shape = "VM.Standard2.1"
    node-size = 3
    ...
Troubleshooting
  • 401 errors - (Service error:NotAuthenticated):
    • You have an incorrect value for one of the following:
      • tenancy OCID
      • user OCID
      • fingerprint
      • RSA private key (the path or the key)
  • no such host:
    • You have an incorrect value for the following:
      • region identifier

References:

What's Next

Congratulations! You have created a Kubernetes cluster using Terraform, in your Oracle Cloud Infrastructure account.

Now that you have a Kubernetes cluster, try the Kubernetes tutorials.

To explore more information about development with Oracle products, check out these sites: