OpenTofu

Management Guide OpenTofu / Terraform

Reference

Structure folders and files

terraform
| -- /.kube → our cluster Kubeconfig files
| -- /templates
| -- | -- apply.yml →  job that apply plans
| -- | -- delete-state.yml → contains delete-state job (currently not used)
| -- | -- destroy.yml → contains destroy job (currently not used)
| -- | -- fmt.yml → jop to check formation
| -- | -- plan.yml → job, that make planes
| -- | -- test.yml → job for tests (currently in pipe, does nothing)
| -- | -- validate.yml → job to validate all *.tf-files
| -- /modules → our templates
| -- | -- /gitrepo → template modul
| -- | -- | -- gitrepo.tf → template
| -- | -- | -- variables → variables
| -- | -- /namespace → template modul
| -- | -- | -- namespace.tf → template
| -- | -- | -- values → variables
| -- | -- /project → template modul
| -- | -- | -- project.tf → template
| -- | -- | -- values → variables
| -- | -- /roles-and_bindings
| -- | -- | -- | -- /global_role_binding → template modul
| -- | -- | -- | -- | -- global_role_binding.tf → template
| -- | -- | -- | -- | -- variables → variables
| -- | -- | -- | -- /project_role_binding → template modul
| -- | -- | -- | -- | -- project_role_binding.tf → template
| -- | -- | -- | -- | -- variables → variables
| -- | -- /templates → template for a new player
| -- | -- | -- player.CHANGEME.tf
| -- | -- /user → template modul for rancher2
| -- | -- | -- user.tf → template
| -- | -- | -- variables → variables
| -- repos_infrastructure.tf → contains module for the Fleet-System GitRepo Ressource
| -- repos_core_services.tf → contains module for the core-services
| -- main.tf → empty file, not needed
| -- providers.tf → contians terraform providers like kubernetes, and rancher2
| -- users-and-roles.tf → contains modules for user and roles

.kube files

There are two important files in .kube:

local.yaml runner.yaml (will likely not be needed in the future, but currently still needs to be updated) Both files need to regularly receive a new token, as they are only valid for 48 hours.

They are both used to access the cluster and environments.

Roles and access

Currently, we have two roles:

  • admin / The admins are managed in users-and-roles.tf.
  • user-base / The regular users each have their own file.

The admin role can access all projects and namespaces, whereas the user-base role can only view the projects assigned to them.

Deploy a new player onto the cluster

The player must provide the following information for the DevOps team:

  1. Name of the player
  2. Repo link of the player
  3. GitLab Runner, if desired

To deploy a player onto the cluster in Rancher, a new file needs to be created. The new file should be named player_CHANGEME.tf and placed in the The-Microservice-Dungeon/DevOps-Team/terraform folder. This file essentially contains two pieces of information: a profile for the player and the target Git repository. Additionally, there are several settings that can be configured to allow the player to access different namespaces or projects.

Here are the steps to follow:

  1. Create a new branch named player-CHANGEME.
  2. Open this branch and copy the file player_changeme.tf into the top-level directory of Terraform.
  3. Replace all occurrences of ‘CHANGEME’ in the file, including the filename, with the appropriate content.
  4. Review your changes, ensuring that all names (e.g., repo-link) and repository links are correct.
  5. Create a merge request to merge your branch into the main branch.

The code, which is explained in detail below, can be found as a complete file at the following location:

https://gitlab.com/the-microservice-dungeon/devops-team/terraform/-/tree/main/modules/templates?ref_type=heads

The file should be placed in the top directory of Terraform, similar to the existing players. Afterward, both the filename part ‘CHANGEME’ and every instance of the word ‘CHANGEME’ within the file must be replaced with the desired content. You’re done once you can no longer find the word ‘CHANGEME’ through a search.

OpenTofu generates a Kubernetes manifest of the type GitRepo. This is registered with the Fleet system. When changes are made to the resource, the Fleet system can detect the updates in the PlayerRepo and redeploy them accordingly. Additionally and a new profile is created, which can be used to log in to the cluster. With the new profile, you will see your own project. When logging in for the first time, a default password is required, which must be changed immediately afterwards!

https://the-microservice-dungeon.gitlab.io/docs/docs/getting-started/basics_player_on_prod/

Standard users need at least the following code

The following code snippets are contents of the file player_changeme.tf located in the terraform/modules/templates directory.

  • The word CHANGEME must be replaced with your player name.

new User

# User: player_CHANGEME / Rolle: user
module "user_player_CHANGEME" {
  source   = "./modules/user"
  name     = "player-CHANGEME"
  username = "player-CHANGEME"
  enabled  = true
}

user-role

module "global_role_binding_player_CHANGEME" {
  source         = "./modules/roles_and_bindings/global_role_binding"
  name           = "player-CHANGEME"
  global_role_id = "user-base"
  user_id        = module.user_player_CHANGEME.user_id
}

You can specify which access rights this player should have. For a normal player, this is currently limited to one role: user-base.

new project

module "project_player_CHANGEME" {
  source      = "./modules/project"
  name        = "player-CHANGEME"
  description = "Project of user player_CHANGEME"
}

new namespace

module "namespace_player_CHANGEME" {
  source     = "./modules/namespace"
  name       = "player-CHANGEME"
  project_id = module.project_player_CHANGEME.projectid
}

add user to the project

module "player_CHANGEME_user_project_binding" {
  source     = "./modules/roles_and_bindings/project_role_template_binding"
  name       = "player-CHANGEME-user-binding"
  project_id = module.project_player_CHANGEME.projectid
  user_id    = module.user_player_CHANGEME.user_id
}

add the gitrepo to the cluster

# Repo for player_CHANGEME on Prod Cluster msd
module "msd_player_CHANGEME" {
  source                = "./modules/gitrepo"
  repo_name             = "msd-player-CHANGEME"
  repo_source_link      = "CHANGEME"
  repo_paths            = ["./helm-chart"]
  repo_clientSecretName = "basic-auth-secret"
  repo_targets = [
    {
      clusterName = "msd"
    }
  ]
}
  • The repo_source_link must be set to your repository URL.

After adding the new file for the player and successfully running the pipelines, you can log in with the new player.

rancher.dungeon-space.de


Deleting a player

To delete a player, you only need to delete the corresponding file at the following folder The-Microservice-Dungeon/DevOps-Team/terraform.


Secret (temporary state)

Secret for fleet-system repo (Modul gitrepo)

kubectl -n NAMESPACE create secret generic NAME --type=kubernetes.io/basic-auth --from-literal=username=YOURUSERNAME --from-literal=password=YOURTOKEN

  • NAMESPACE: namespace where the secret will be installed / default: fleet-default
  • NAME: name of the secret and entered in repo_clientSecretName on Modul: gitrepo
  • YOURUSERNAME: name of your account (gitlab / github / etc.)
  • YOURTOKEN: Auth Token of your account (gitlab / github / etc. )

Code Structure

Root Directory

providers.tf

  1. required_providers specifies the necessary providers (external services) required for this Terraform configuration.
  2. backend “http” {} placeholder for configuring the backend where Terraform stores its state remotely (currently empty but meant for HTTP-based storage).
  • Multiple Providers: This configuration uses Rancher, Kubernetes, Helm, and GitLab providers, allowing management of Kubernetes clusters, Helm charts, and GitLab resources.
  • Security: The insecure = true flag in both the Kubernetes and Helm providers suggests that SSL verification is disabled, which could be used in development environments.
  • File Dependencies: Kubernetes provider configuration relies on two local files: .kube/local.yaml and .kube/runner.yaml, which must be present and updated regularly (as mentioned earlier, for tokens).
terraform {
  required_providers {
    rancher2 = {
      source  = "rancher/rancher2"
      version = "4.1.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "2.27.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "2.12.1"
    }
    gitlab = {
      source  = "gitlabhq/gitlab"
      version = "16.11.0"
    }

  }

  backend "http" {}
}

# Configure the Rancher2 provider to admin

provider "rancher2" {}

provider "kubernetes" {
  config_path    = ".kube/local.yaml"
  config_context = "local"
  insecure       = true
}

provider "helm" {
  kubernetes {
    config_path = ".kube/runner.yaml"
    insecure    = true
  }
}

provider "gitlab" {}

user-and-roles.tf

  • User Creation: Defines two users, “foo” (Administrator) and “bar” (Cluster User), using modules for user management.
  • Role Assignment: Binds each user to specific global roles, foo as “admin” and bar as “user-base.”
  • Modular Structure: Uses separate modules for creating users and assigning global roles, ensuring a modular and reusable setup.
# User: foo / Rolle: Administrator
module "user_foo" {
  source   = "./modules/user"
  name     = "foo"
  username = "foo"
  enabled  = true
}

module "global_role_binding_foo" {
  source         = "./modules/roles/global_role_binding"
  name           = "foo"
  global_role_id = "admin"
  user_id        = module.user_foo.user_id
}


# User: bar / Rolle: Cluster User
module "user_bar" {
  source   = "./modules/user"
  name     = "bar"
  username = "bar"
  enabled  = true
}

module "global_role_binding_bar" {
  source         = "./modules/roles/global_role_binding"
  name           = "bar"
  global_role_id = "user-base"
  user_id        = module.global_role_binding_bar.user_id
}

Modules


gitrepo

gitrepo.tf

  • Manifest Definition: Creates a Kubernetes resource with dynamically set values for apiVersion, kind, metadata, and spec, all controlled by variables (var.*).

  • Git Repository Configuration: In the spec, it configures the repository with:

    • repo (source link), branch, and paths for Git repo management.
    • insecureSkipTLSVerify to skip TLS verification if needed.
    • targets (likely clusters or environments) and clientSecretName for authentication.
  • Flexible Variables: The use of var.* allows dynamic customization of the resource, making it adaptable to different environments and repositories.

resource "kubernetes_manifest" "fleet_gitrepo" {
  manifest = {
    apiVersion = var.repo_api
    kind       = var.repo_kind
    metadata = {
      name      = var.repo_name
      namespace = var.repo_namespace
    }
    spec = {
      repo                  = var.repo_source_link
      branch                = var.repo_branch
      insecureSkipTLSVerify = var.repo_insecure
      paths                 = var.repo_paths
      targets               = var.repo_targets
      clientSecretName      = var.repo_clientSecretName
    }
  }
}

variables.tf

This Terraform configuration uses Basic Repository Configuration for Rancher Fleet, sets Git Settings like branch and paths, defines Cluster and Authentication with cluster targets and secrets, and specifies the Repository Link for managing Git resources in Kubernetes.

variable "repo_api" {
  description = "The api version"
  default     = "fleet.cattle.io/v1alpha1"
}

variable "repo_kind" {
  description = "kind of ressource"
  default     = "GitRepo"
}

variable "repo_name" {
  description = "The name of the repo in Rancher"
}

variable "repo_namespace" {
  description = "The namespace for the Git repository"
  default     = "fleet-default"
}

variable "repo_branch" {
  description = "The branch of the Git repository"
  default     = "main"
}

variable "repo_insecure" {
  description = "set insecure"
  default     = true
}

variable "repo_paths" {
  description = "list of the paths of the Git repository"
  type        = list(string)
  default     = [""]
}

variable "repo_targets" {
  description = "cluster targets of Git repository"
  type = list(object({
    clusterName = string
  }))
  default = [
    {
      clusterName = ""
    }
  ]
}

variable "repo_source_link" {
  description = "link of the Git repository"
}

variable "repo_clientSecretName" {
  description = "The name of the repo clientSecretName in Rancher"
  default     = ""
}

namespace

This Terraform configuration creates a namespace in a Rancher-managed Kubernetes cluster. A namespace acts as an isolation zone within a Kubernetes cluster, allowing resources to be organized and secured separately. This isolation helps manage resources, control access, and prevent conflicts between applications or teams sharing the same cluster.

namespace.tf

resource "rancher2_namespace" "the_ns" {
name       = var.name
project_id = var.project_id
}

values.tf

variable "name" {
  type = string
}

variable "project_id" {
  type = string
}

project

This configuration enables the creation of a project in a Rancher cluster, including essential details like the project name, description, and monitoring settings. It allows for organized management of resources within the cluster, providing a dedicated space for application development and deployment.

project.tf

resource "rancher2_project" "project" {
  cluster_id                = var.cluster_id
  name                      = var.name
  description               = var.description
  enable_project_monitoring = var.enable_project_monitoring
}

output "projectid" {
  value = rancher2_project.project.id
}

values.tf

variable "name" {
  type = string
}

variable "cluster_id" {
  type    = string
  default = "c-m-fcvfr6nf"
}

variable "description" {
  type = string
}

variable "enable_project_monitoring" {
  type    = bool
  default = true
}

user

user.tf

This file helps automate the creation and management of Rancher users.

# Create a new rancher2 User
resource "rancher2_user" "basic_user" {
  name     = var.name
  username = var.username
  password = var.password
  enabled  = var.enabled
}

output "user_id" {
  value = rancher2_user.basic_user.id
}

variables.tf

variable "name" {
  type    = string
  default = "changeme"
}

variable "username" {
  type    = string
  default = "changeme"
}

variable "password" {
  type    = string
  default = "changeme123456"
}

variable "enabled" {
  type    = bool
  default = true
}

Module / Roles

cluster role bindings

This configuration creates a global role binding for a specified user in Rancher, allowing the user to be assigned roles, thereby controlling access and permissions. The use of variables enhances flexibility, enabling easy customization of role bindings.

global_role_binding.tf

# Create a new rancher2 global_role_binding for User
resource "rancher2_global_role_binding" "changeme" {
  name           = var.name
  global_role_id = var.global_role_id
  user_id        = var.user_id
}

variables.tf

variable "name" {
  type = string
}

variable "global_role_id" {
  type    = string
  default = "user-base"
}

variable "user_id" {
  type = string
}

project role binding

This configuration allows the assignment of a role template to a user within a specific project in Rancher.

project_role_template_binding.tf

resource "rancher2_project_role_template_binding" "project_role_template_binding" {
  name             = var.name
  project_id       = var.project_id
  user_id          = var.user_id
  role_template_id = var.role_template_id
}

variables.tf

variable "name" {
  type = string
}

variable "role_template_id" {
  type    = string
  default = "project-member"
}

variable "user_id" {
  type = string
}

variable "project_id" {
  type = string
}

Last modified February 4, 2025: fix go & npm dependencies (8ff1fa0)