Cluster Setup

• Description
• Files and Dirs
• First Installation - Automated
• Installations by Hand
  • RKE2
  • Sealed-Secrets Certs
  • ArgoCD
  • Argo CD Values File
  • ArgoCD Application / First Repo
• Uninstalling
  • Uninstalling RKE2
  • Uninstall ArgoCD
• Miscellaneous and Issues

Description

This section describes the setup of the RKE2 Kubernetes cluster on the goedel-m01 server. It outlines the installation and basic configuration required to create a functional cluster environment. Additionally, it covers the installation and configuration of ArgoCD to automate application deployment.

Files and Dirs

Following list is alphabetical

First Installation - Automated

Execute the first.sh script located in /rke2_config. The script performs the following tasks:

1. Installs RKE2
2. Copies the rke2_config.yaml file to /etc/rancher/rke2/config.yaml
3. Starts the RKE2 cluster using systemctl and then enables it via systemctl
4. Executes /sealed-secrets/sealed-secrets.sh to generate the key pair required for decrypting secrets in future deployments
5. Executes /argocd/argocd.sh, which deploys Argo CD into the cluster using Helm and applies the first GitLab repository of our organization using kubectl

#!/bin/bash
set -e

echo ""
echo " ----> start install rke2-server"
echo ""


curl -sfL https://get.rke2.io/ | sh -

echo ""
echo " ----> copy rke2-config to /etc/rancher/rke2/config.yaml"
echo ""


cp rke2_config.yaml /etc/rancher/rke2/config.yaml

echo ""
echo " ----> start systemctl rke2-server and enable (this can take up to 5 minutes)"
echo ""


systemctl start rke2-server

systemctl enable rke2-server

echo ""
echo " ----> wait for rke2-server node is ready (timeout set up to 360s = 6 minutes)"
echo ""

kubectl get nodes
kubectl wait --for=condition=Ready --timeout=360s nodes --all
kubectl get nodes

echo ""
echo " ----> wait for cluster namespacd kube-system is ready (timeout set up to 360s = 6 minutes)"
echo ""

sleep 20s
kubectl rollout status deployment -n kube-system
#kubectl wait --for=condition=Ready --timeout=360s pods --all -n kube-system


echo ""
echo " ----> start deploy sealed-secrets certs"
echo ""

cd sealed-secret-certs

./sealed-secrets-cert.sh

cd ..

echo ""
echo " ----> start deploy argocd"
echo ""

cd argo

./argocd.sh

cd ..

echo ""
echo " ----> FINISH"
echo ""

Finally, execute the displayed command to retrieve the argocd-admin-password. You will need this password to log in as admin to the Argo CD UI at the provided web address.

Installations by Hand

RKE2

Components installed during the RKE2 installation:

• Canal (CNI)
• CoreDNS
• Kube-proxy
• RKE2 Server
• Nginx Ingress Controller

RKE2 is installed as described here: RKE2 Installation. Below are the steps in short form:

1. Install the RKE2 package (default installation as a server):

curl -sfL https://get.rke2.io | sh -

2. Place the config.yaml file in the /etc/rancher/rke2/ directory:

rke2_config.yaml

# This config file should be placed under /etc/rancher/rke2/config.yaml

# TLS (Additional addresses for the TLS certificate)
tls-san:
  - "dungeon-space.de"

# Cluster name, default is goedel-m01.gm.fh-koeln.de
node-name: "msd01-rke2"

# Directory where the Cluster will be installed
# data-dir, default is /var/lib/rancher/rke2
data-dir: "/store/rke2/msd01"

# List of labels for the node
#node-label:
#  role: "controlplane"

# Permissions for the kubeconfig file
#write-kubeconfig-mode: "0644"

# kube-sheduler-arg:
#  - "address=0.0.0.0"

Additional parameters for config.yaml: node-taint, kube-apiserver-arg, kube-controller-manager-arg, kubelet-arg, system-default-registry, private-registry, kube-scheduler-arg, disable

3. Start the RKE2 service:

sudo systemctl start rke2-server

4. Check the status of the RKE2 service:

sudo systemctl status rke2-server

5. Enable the RKE2 service to start on boot:

sudo systemctl enable rke2-server

6. Export the KUBECONFIG (to access the cluster):

export KUBECONFIG=/etc/rancher/rke2/rke2.yaml

7. Check the cluster status:

kubectl get nodes

Sealed-Secrets Certs

Execute the sealed-secrets-cert.sh script located in the /rke2_config/seald-secret-certs directory. The script performs the following tasks:

1. Checks if the sealed-secrets namespace exists, and creates it if necessary
2. Creates a TLS secret named sealed-secrets-key in the previously created namespace using kubectl. The certificate and private key are stored in the same directory

The file sealed-secret-tls.crt is the public key used to encrypt our secrets. This file should be made available to everyone in our organization.

The file sealed-secret-tls.key is the private key used to decrypt the secrets within the cluster. This key must not be shared or published.

sealed-secret-certs.sh

#!/bin/bash
set -e

# CHECK, if kubectl is installed
if ! command -v kubectl &>/dev/null; then
  echo "kubectl is not installed. Please install kubectl first."
  exit 1
fi

# CHECK, if namespace is present
echo "Check and create namespace sealed-secret, if not present"
if ! kubectl get namespace sealed-secrets &>/dev/null; then
  kubectl create namespace sealed-secrets
fi

echo "Create sealed-secrets tls cert in namespace sealed-secret"

kubectl -n sealed-secrets create secret tls sealed-secrets-key --cert=sealed-secret-tls.crt --key=sealed-secret-tls.key

ArgoCD

To install Argo CD in the cluster, execute the argocd.sh script located in the rke2_config/argo directory. The script performs the following tasks:

1. Deploys Argo CD into the cluster using a Helm chart. The argocd-values.yaml file in the same directory is used as the values source.
2. Applies the first Argo CD Application to the cluster using kubectl apply.

argocd.sh

#!/bin/bash
set -e

# CHECK, if kubectl is installed
if ! command -v kubectl &>/dev/null; then
  echo "kubectl is not installed. Please install kubectl first."
  exit 1
fi

echo "Add argo cd-helm chart and deploy it"
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm -n argocd install argocd argo/argo-cd --version 7.8.9 --create-namespace -f argocd-values.yaml

echo "wait for argocd apps are ready"

kubectl -n argocd rollout status statefulset

echo "apply gitlab-repo from msd"
kubectl apply -f argocd-first.yaml

echo "You can now access the ArgoCD dashboard at https://argocd.microservice-dungeon.de/."
echo "You can retrieve the initial admin password with the following command:"
echo "kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath="{.data.password}" | base64 -d && echo"

Argo CD Values File

Below is the argocd.yaml values file.

It is important that under configs.cm, both kustomize and helm are set to enabled. Additionally, --enable-helm must be included in the kustomize.buildOptions. Otherwise, Argo CD will not be able to render Kustomize and Helm-related configurations properly.

This file is also located in our GitLab repository at: https://gitlab.com/the-microservice-dungeon/devops-team/ci-cd/argocd/-/blob/main/application/bootstrap-manifests/argocd-application.yaml?ref_type=heads

openshift:
  enabled: false
crds:
  install: true
  keep: false
global:
  addPrometheusAnnotations: true
  domain: argocd.microservice-dungeon.de
configs:
  cm:
    create: true
    kustomize.enabled: true
    jsonnet.enabled: true
    helm.enabled: true
    admin.enabled: true
    kustomize.buildOptions: -enable-helm
server:
  name: server
  replicas: 1
  insecure: true
  ingress:
    enabled: true
    ingressClassName: nginx
    annotations:
      nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    tls: false

ArgoCD Application / First Repo

The following shows the argocd-first.yaml file. After applying it with kubectl apply, Argo CD is configured to track our GitLab repository, specifically the main branch and the folder application/bootstrap-manifests.

Argo CD will then automatically deploy all other Applications referenced from that location, effectively deploying the remainder of the cluster.

argocd-first.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: a-msd-full
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://gitlab.com/the-microservice-dungeon/devops-team/ci-cd/argocd.git
    path: application/bootstrap-manifests/
    targetRevision: main
  destination:
    server: https://kubernetes.default.svc
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Uninstalling

Uninstalling RKE2

To uninstall RKE2, follow the steps outlined here: RKE2 Uninstall. The uninstall script can be executed as follows:

/usr/bin/rke2-uninstall.sh

Uninstall ArgoCD

To uninstall ArgoCD from Cluster run:

helm -n argocd uninstall argocd

Miscellaneous and Issues

• CSI driver Longhorn not found
  • Occurred during the initial installation
  • Solution: Restart the node
• Argo CD does not deploy the repositories during initial setup
  • This may happen, but should theoretically be resolved after a few minutes through automatic sync
  • Solution: Trigger a sync either via the Web UI or manually in the cluster terminal
• Reinstalling RKE2 fails / RKE2 does not start after a reinstall
  • Old cluster data still exists on the system and is being reused, causing issues
    • Cluster data: Delete the folder where the cluster is stored: rm -rf /store/rke2/msd01
    • Longhorn: Some mounts were not properly unmounted by the RKE2 uninstaller: umount /var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/*/globalmount
    • Longhorn: Remove created storage: rm -rf /store/msd-cluster-store/