Helmfile Declarative CD Archi: Multi-Env

💡 Helmfile Declarative CD Archi: Multi-Env — Executive Summary

Technical architectural analysis, practical configuration manifests, and deployment instructions for Helmfile Declarative CD Archi: Multi-Env.

Information

References

gitops repository

Enterprise Cloudlab/cmp-helm

Cluster Version

Development environment information | Cluster Version

Architectures

Cluster Installation

Global Setting

bash

vi /etc/systemd/resolved.conf
>>>
[Resolve]
DNS=<management_node_ip> 8.8.8.8
<<<
systemctl restart systemd-resolved.service

in NFS Server

NFS Server Installation

bash

# 이 리포에서 내용 복사해 파일 생성 추천
vi setup-nfs-server.sh
chmod +x setup-nfs-server.sh
./setup-nfs-server.sh

in Business-cluster HAProxy Servers

HAProxy Installation & setting

bash

# Dev-LB, Stg-LB Nodes
sudo apt-get update && apt-get -y upgrade
apt install -y haproxy
sudo systemctl enable haproxy
sudo systemctl start haproxy
sudo systemctl status haproxy
haproxy -v

# config 생성
CP1_IP=""
CP2_IP=""
CP3_IP=""
WK1_IP=""
WK2_IP=""
WK3_IP=""
cat <<EOF > /etc/haproxy/haproxy.cfg
global
    log /dev/log local0
    log /dev/log local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
    log global
    mode http
    option httplog
    timeout connect 5000
    timeout client 50000
    timeout server 50000

frontend k8s_api
    bind *:6443
    mode tcp
    option tcplog
    default_backend k8s_api_backend

backend k8s_api_backend
    mode tcp
    balance roundrobin
    option tcp-check
    server master1 $CP1_IP:6443 check
    server master2 $CP2_IP:6443 check
    server master3 $CP3_IP:6443 check
    
frontend nginx_ingress
    bind *:443
    mode tcp
    option tcplog
    default_backend nginx_ingress_backend
backend nginx_ingress_backend
    mode tcp
    balance roundrobin
    option tcp-check
    server worker1 $WK1_IP:30443 check
    server worker2 $WK2_IP:30443 check
    server worker3 $WK3_IP:30443 check
frontend nginx_ingress_http
    bind *:80
    mode tcp
    option tcplog
    default_backend nginx_ingress_http_backend
backend nginx_ingress_http_backend
    mode tcp
    balance roundrobin
    option tcp-check
    server worker1 $WK1_IP:30080 check
    server worker2 $WK2_IP:30080 check
    server worker3 $WK3_IP:30080 check

frontend amqp
    bind *:5672
    mode tcp
    option tcplog
    default_backend amqp_backend
backend amqp_backend
    mode tcp
    balance roundrobin
    option tcp-check
    server worker1 $WK1_IP:30672 check
    server worker2 $WK2_IP:30672 check
    server worker3 $WK3_IP:30672 check
    
frontend mariadb
    bind *:3306
    mode tcp
    option tcplog
    default_backend mariadb_backend
backend mariadb_backend
    mode tcp
    balance roundrobin
    option tcp-check
    server worker1 $WK1_IP:30306 check
    server worker2 $WK2_IP:30306 check
    server worker3 $WK3_IP:30306 check

frontend redis
    bind *:6379
    mode tcp
    option tcplog
    default_backend redis_backend
backend redis_backend
    mode tcp
    balance roundrobin
    option tcp-check
    server worker1 $WK1_IP:30379 check
    server worker2 $WK2_IP:30379 check
    server worker3 $WK3_IP:30379 check

frontend keycloak_http
    bind *:8880
    mode tcp
    option tcplog
    default_backend keycloak_http_backend
backend keycloak_http_backend
    mode tcp
    balance roundrobin
    option tcp-check
    server worker1 $WK1_IP:31080 check
    server worker2 $WK2_IP:31080 check
    server worker3 $WK3_IP:31080 check
EOF

systemctl restart haproxy

in Management-cluster

~~Kube-VIP: Select VIP, check NIC (failed due to ARP blocking in Openstack Network) (35a00f54)~~

References and Guidelines (NOTE)

⚠️ (If the band is different from the business cluster or the NIC name may be different, proceed in that cluster. This is the same Openstack Network, so proceed in Management)

Select VIPs for dev and stg and remember them
VIP:
bash apt install -y nmap # Check host down IP ### Use the results below except x.x.x.0, x.x.x.1, x.x.x.255 (reservation address) needs_net="10.255.57.0/24" nmap -v -sn -n $needs_net | grep "host down"
NIC:
bash # Remember ens3 for the following results ifconfig | grep '10.255.57' -B1 | head ens3: flags=4163 mtu 1500 inet 10.255.57.142 netmask 255.255.255.0 broadcast 10.255.57.255
Reflect kubespray:

bash

GIT_ROOT=$(git rev-parse --show-toplevel 2>/dev/null)
  cd $GIT_ROOT/_iac/kubespray

      #kube-vip setting
      ### addons.yml 수정
      ##### kube_vip_address: <vip of env> 
      ##### kube_vip_interface: <nic of env> 
      vi ./inventory/dev-cmp/group_vars/k8s_cluster/addons.yml
      vi ./inventory/stg-cmp/group_vars/k8s_cluster/addons.yml
      ### k8s-cluster.yml 수정
      ##### kube_proxy_strict_arp: true
      vi ./inventory/dev-cmp/group_vars/k8s_cluster/k8s-cluster.yml
      vi ./inventory/stg-cmp/group_vars/k8s_cluster/k8s-cluster.yml
      ```

host setup

txt

# /etc/hosts
# mgnt
127.0.0.1 localhost re-mgmt-deploy mng-dpl
10.255.57.84  mng-nfs
# dev
10.255.57.182 dev-lbs
10.255.57.111 dev-cp1
10.255.57.103 dev-cp2
10.255.57.97  dev-cp3
10.255.57.169 dev-wk1
10.255.57.228 dev-wk2
10.255.57.122 dev-wk3
10.255.57.123 dev-dbp
10.255.57.147 dev-db1
10.255.57.136 dev-db2
#### 10.255.57.240 dev-vip
# stg
10.255.57.112 stg-lbs
10.255.57.142 stg-cp1
10.255.57.179 stg-cp2
10.255.57.221 stg-cp3
10.255.57.114 stg-wk1
10.255.57.206 stg-wk2
10.255.57.229 stg-wk3
10.255.57.134 stg-dbp
10.255.57.164 stg-db1
10.255.57.70  stg-db2
#### 10.255.57.241 stg-vip

installation Management Cluster

bash

# go-to
git clone https://[email protected]/Enterprise Cloudlab/cmp-helm.git
cd cmp-helm
git switch Enterprise Cloud-dev

GIT_ROOT=$(git rev-parse --show-toplevel 2>/dev/null)
cd $GIT_ROOT/_iac

# k3s 설치
./k3s/server-install-k3s.sh
source ~/.bashrc

# IaC & GitOps 관련 바이너리 세팅
./server.sh
sudo apt install python3-pip -y
sudo pip3 install -r kubespray/requirements.txt

installation Business Clusters

⚠️ Be sure to check /etc/hosts settings before proceeding

Currently, playbook is only tested based on an online environment.

bash

GIT_ROOT=$(git rev-parse --show-toplevel 2>/dev/null)
cd $GIT_ROOT/_iac/kubespray

ansible-playbook -i inventory/dev-cmp/inventory.ini cluster.yml -vv
ansible-playbook -i inventory/stg-cmp/inventory.ini cluster.yml -vv

trouble shooting

⚠️ The following error occurs due to a change in the Pull policy of the DockerHub registry.

txt

docker: Error response from daemon: toomanyrequests: You have reached your unauthenticated pull rate limit. https://www.docker.com/increase-rate-limit
nerdctl: FATA[0006] failed commit on ref "index-sha256:e21f8d0566ad4a2b7de56dae9e662147af6df51ed8a46692be3931513a1a56d7": commit failed: "index-sha256:e21f8d0566ad4a2b7de56dae9e662147af6df51ed8a46692be3931513a1a56d7" failed size validation: 341810 != 10229: failed precondition

solving

bash

# dockerhub 무료계정 로그인으로 임시 해소 가능 (무료계정은 pull 횟수제한있음)
  # 설치 실패한 모든 클러스터 node 에서: (이 시점에 `nerdctl`은 설치 완료 되어있음)
  nerdctl login docker.io

When Business Clusters Installed (in Management-cluster)

Import Kubeconfig

bash

GIT_ROOT=$(git rev-parse --show-toplevel 2>/dev/null)
cd $GIT_ROOT/_iac
./k8s_generate_kubeconfig_with_ssh_v2.sh dev dev-lbs [ sshport ] > $GIT_ROOT/_gitops/.tmp/.dev.kubeconfig
./k8s_generate_kubeconfig_with_ssh_v2.sh stg stg-lbs [ sshport ] > $GIT_ROOT/_gitops/.tmp/.stg.kubeconfig

# test
k get pods -A --kubeconfig $GIT_ROOT/_gitops/.tmp/.dev.kubeconfig
k get pods -A --kubeconfig $GIT_ROOT/_gitops/.tmp/.stg.kubeconfig

cat <<EOF >> ~/.bashrc
alias kd="kubectl --kubeconfig /workspace/cmp-helm/_gitops/.tmp/.dev.kubeconfig"
alias ks="kubectl --kubeconfig /workspace/cmp-helm/_gitops/.tmp/.stg.kubeconfig"
EOF
source ~/.bashrc

kd get pods -A
ks get pods -A

all worker Nodes

bash

# management node, business workers
sudoapt-get update
sudoapt-getinstall -y nfs-common

Application Installation

go to:

bash

GIT_ROOT=$(git rev-parse --show-toplevel 2>/dev/null)
cd $GIT_ROOT/_gitops

Required

Prerequisite work required to run Gitops via SSH in Helmfile
1. Copy server public key
bash cat ~/.ssh/id_rsa.pub # If not present, key-gen ssh-keygen
2. Access https://bitbucket.org/account/settings/ssh-keys/ and register SSH public key
Modify Variable

bash

vi values/clusters.hcl
  # 다음을 수정
  ### clusters.mgnt.hostIp = "Mgnt 마스터노드 IP"
  ### clusters.mgnt.nfs.ip = "Mgnt NFS IP" #현재 버전은 NFS 1개로 나누어 사용
  ### clusters.dev.hostIp = "Dev 마스터노드 IP"
  ### clusters.dev.nfs.ip = "Dev NFS IP" #현재 버전은 NFS 1개로 나누어 사용
  ### clusters.stg.hostIp = "Stg 마스터노드 IP"
  ### clusters.stg.nfs.ip = "Stg NFS IP" #현재 버전은 NFS 1개로 나누어 사용

in Business-clusters

Step0. Prepare Worker Nodes

bash

### All Business works
# DNS 사용
vi /etc/systemd/resolved.conf
>>>
[Resolve]
DNS=<management_node_ip>#8.8.8.8
<<<
systemctl restart systemd-resolved.service
# Harbor Certificate
sudomkdir -p /etc/containerd/certs.d/harbor.Enterprise Cloud.hanacloudia.com
cat<<EOF> /etc/containerd/certs.d/harbor.Enterprise Cloud.hanacloudia.com/ca.crt
##### <ca.crt 내용 삽입> #####
EOF
# or
cp /etc/kubernetes/ssl/certs/serverca /etc/containerd/certs.d/harbor.Enterprise Cloud.hanacloudia.com/ca.crt

in Management-cluster

Step1. Helmfile Sync (To set ArgoCD)

The following application(s) will be created.

Prepare hooks
Create All-Namespaces
Run on docker: local-registry
- Load & Push Images from backed-up Dir
Run kube-apiserver proxy: localhost:8080
Auto-Gen: Secret=$basicAuthSecret
nfs-subdir-external-provisioner
Required: package=nfs-common
cert-manager
Auto-Gen: ClusterIssuer=selfsigned-issuer
Auto-Gen: Certificates=hanacloudia.com-cacert
Auto-Gen: ClusterIssuer=rootca-issuer
Create File: ./.tmp/ca.crt (To call the Harbor API)
Send File: ./.tmp/ca.crt to every Nodes of Business Clusters
Create File: `./.tmp/ca-issuer.yaml for Business Clusters
ingress-nginx
adguard-home
Pre-Configure: DNS=harber, argo-cd
harbor
Call API: create project management-system
Call API: create project deploy-system
Call API: create project portal-system
Call API: create project dev-portal
Call API: create project stg-portal
Certificate & Authorize
Load & Push Images from backed-up Dir
argo-cd
Management Cluster Jobs
- Pre-Configure: argocd-helmfile-plugin
- Pre-Configure: Auto-Connect Private Repository [email protected]:Enterprise Cloudlab/cmp-helm.git with CRED
- Auto-Gen: PersistentVolumClaim=apache-htdocs-pvc
- Auto-Gen: Application=apache
- Auto-Gen: Application=jenkins
- Auto-Gen: Application=vault
Business Cluster Jobs
- Auto-Gen: Secret=$globalImagePullSecret to Every Namespaces
- Pre-Configure: Auto-Connect Clusters
- Auto-Gen: Application=nfs-subdir-external-provisioner
- Auto-Gen: Application=cert-manager
- Auto-Gen: Application=ingress-nginx
- Auto-Gen: Application=redis for CMP
- Auto-Gen: Application=mariadb for CMP
- Auto-Gen: Application=redis for CMP

How does it work? (helmfile Docs)

ℹ️ Read helmfile.yaml.gotmpl within the path.

Overwrite variables defined in .hcl and .yaml under ./values using the goTemplate syntax.

bash

# DNS 포트충돌 방지
sudo systemctl stop systemd-resolved
sudo systemctl disable systemd-resolved
sudo systemctl mask systemd-resolved
sudo rm -f /etc/resolv.conf
sudo bash -c 'cat > /etc/resolv.conf <<EOF
nameserver 127.0.0.1 #adguard-home
nameserver 8.8.8.8
nameserver 1.1.1.1
options edns0
EOF'
# 다른 서비스에서 수정금지
sudo chattr -i /etc/resolv.conf

# DNS사용 불가할 경우 사전 조치
### echo '127.0.0.1 harbor.Enterprise Cloud.hanacloudia.com argocd.Enterprise Cloud.hanacloudia.com' | sudo tee -a /etc/hosts

helmfile sync

Trouble shooting

bash

# 화면이 멈춘 경우 오류가 원인일 수 있다. 오류인 경우 ctrl+C로 escape
  # 그러나 다음의 경우 주의
  ---
  Release "harbor" does not exist. Installing it now.
  NAME: harbor
  LAST DEPLOYED: Sun Dec 14 20:44:41 2025
  NAMESPACE: management-system
  STATUS: deployed
  REVISION: 1
  TEST SUITE: None
  NOTES:
  Please wait for several minutes for Harbor deployment to complete.
  Then you should be able to visit the Harbor portal at https://harbor.Enterprise Cloud.hanacloudia.com
  For more details, please visit https://github.com/goharbor/harbor

  Listing releases matching ^harbor$
  harbor  management-system       1               2025-12-14 20:44:41.468450736 +0900 KST deployed        harbor-1.18.0   2.14.0
  ---
  # 다음 화면에서 멈춰있는 경우 오류가 아니라, backup image들을 Harbor에 push중일 확률이 높음.
  # harbor가 이미 설치완료되었으므로, UI에 접근하여 이미지 업로드중인지 확인.

Step2. Restore Image Registry

When the directory structure is roughly as follows:

txt

.
├── deploy-system
│         └── apache.tar
├── library
│         ├── broker-base.tar
│         ├── busybox.tar
│         ├── curlimage.tar
│         ├── ibmjava.tar
│         └── nginx.tar
├── portal-system
│         ├── keycloak.tar
│         ├── mariadb.tar
│         ├── rabbitmq.tar
│         ├── redis.tar
│         ├── vault-k8s.tar
│         └── vault.tar
...

Replace the $backup_dir and run as follows

bash

#helmfile.yaml.gotmpl에 통합됨

#./scripts/docker_load-and-push.sh /mnt/apache/html/backup-data/
### ./scripts/docker_load-and-push.sh $backup_dir

Step3. Install & Restore Vault

Install

bash

#helmfile.yaml.gotmpl에 통합됨
  ### kubectl apply -f argo-apps/mgnt/vault.yaml

Init Vault

bash

k exec -it -n portal-system vault-0 --container vault -- vault operator init > shared-key-dump.txt
  sed -r "s/\x1B\[([0-9]{1,3}(;[0-9]{1,2})?)?[mGK]//g" shared-key-dump.txt > ./.tmp/vault-shared-key.txt
  rm shared-key-dump.txt
  cat ./.tmp/vault-shared-key.txt

The following key-set appears. When restarting vault, you will need 3 Unseal Key, so save them well.

bash

Unseal Key 1: NmcbH0bxGABFSk0qNgKrXi9dxeiJsq8DrBDgEsVDegus
  Unseal Key 2: kY10hNpENinpiUFKJZwXiJOsCAoyjp7pWAt2tyTwTj5Z
  Unseal Key 3: xtAB/krc4fumYR6X+WaaC7fIAuqeThdF6O/ZirM526uR
  Unseal Key 4: iHHqqtjJYthnjFo2pxSPzxOLH2cAbOfFwmQsJFpxfrwO
  Unseal Key 5: HU9RLIOJVuMUvdzNAAFToZMWDb97UD1FFgji/3sxmY6k

  Initial Root Token: hvs.pWUTilJ5CxAi6S2daKcKPeha

[Auto-mode] Unseal And Create Secret

bash

# store root-token
  #cat ./.tmp/vault-shared-key.txt | sed -n '7p' | awk '{ print $4 }' | tee ./.tmp/vault-root-token.txt
  cat ./.tmp/vault-shared-key.txt | sed -n '7p' | awk '{ gsub(/\x1b\[[0-9;]*m/, "", $4); print $4 }' | tee ./.tmp/vault-root-token.txt

  # Auto
  ./scripts/vault_unseal.sh ./.tmp/vault-shared-key.txt
  k exec -it -n portal-system vault-0 --container vault -- vault login $(cat ./.tmp/vault-root-token.txt)
  k exec -it -n portal-system vault-0 --container vault -- vault secrets enable -path=secret kv-v2
  k exec -it -n portal-system vault-0 --container vault -- sh
  # --- start in vault container
  echo ''' 
  path "*" {
    capabilities = ["create", "read", "update", "delete", "list", "sudo"]
  }''' > /tmp/admin.hcl
  exit 
  # --- end
  ./scripts/vault_enable_userpass.sh admin Cloud000@ portal-system $(cat ./.tmp/vault-root-token.txt | tr '\n' '')

[Manual-mode] Unseal And Create Secret

bash

# Manual
  k exec -it -n portal-system vault-0 -- sh
  # --- start in vault container

  vault operator unseal NmcbH0bxGABFSk0qNgKrXi9dxeiJsq8DrBDgEsVDegus
  vault operator unseal kY10hNpENinpiUFKJZwXiJOsCAoyjp7pWAt2tyTwTj5Z
  vault operator unseal iHHqqtjJYthnjFo2pxSPzxOLH2cAbOfFwmQsJFpxfrwO

  # RootToken으로 로그인
  vault login hvs.pWUTilJ5CxAi6S2daKcKPeha
  # Secret 생성
  vault secrets enable -path=secret kv-v2

  # UserPass 활성화
  echo ''' 
  path "*" {
    capabilities = ["create", "read", "update", "delete", "list", "sudo"]
  }''' > /tmp/admin.hcl

  vault policy write ${POLICY_NAME} /tmp/${POLICY_NAME}.hcl

  if vault auth list | grep -q "userpass/"; then
    echo "[SKIP] userpass already enabled"
  else
    vault auth enable userpass
    # UI에서 기본 로그인으로 사용 옵션
    vault auth tune -listing-visibility=unauth userpass/
    echo "[OK] userpass enabled"
  fi

  vault write auth/userpass/users/admin \
  password="Cloud000@" \
  policies="admin"

  # --- end

push to upstream

bash

git add ./.tmp
  git commit -m '(re) init vault token'
  git push

Result Value

bash

safe target vault http://$(k get svc -n portal-system vault -o=jsonpath='{.spec}' | jq -r '"\(.clusterIP):\(.ports[] | select(.name=="http")).port")')
  safe auth # Initial Root Token 입력

  safe import < $backed_up_json

Step4. Etc.

bash

#helmfile.yaml.gotmpl에 통합됨
# Install Apache
### kubectl apply -f manifests/apache/pvc-htdocs.yaml
### kubectl apply -f argo-apps/mgnt/apache.yaml
# Install Jenkins
### kubectl apply -f argo-apps/mgnt/jenkins.yaml

Step5. Add Clusters to Argo CD

After context setting…

bash

#helmfile.yaml.gotmpl에 통합됨
### argocd cluster add dev_cluster
### argocd cluster add stg_cluster

References and Guidelines (NOTE)

⚠️ Be sure to edit .spec.destination.server in ./argo-apps/*/*.yaml files in advance. (Env: dev, stg)

Step6. Install & Restore Database

Install Database

bash

#helmfile.yaml.gotmpl에 통합됨

  ### cd argo-apps
  ### kubectl apply \
  ###  -f dev/mariadb.yaml -f dev/rabbitmq.yaml -f dev/redis.yaml \
  ###  -f stg/mariadb.yaml -f stg/rabbitmq.yaml -f stg/redis.yaml

Restore Database

bash

GIT_ROOT=$(git rev-parse --show-toplevel 2>/dev/null)
  cd $GIT_ROOT/_backups/sqls

  #dev
  kd exec -i -n portal-system mariadb-primary-0 --container mariadb -- mariadb -uroot -pCloud000@ <<EOF
    create database OKE_DEVOPS;
    create database ECP_USER;
    create database ECP_ADMIN;
    create database keycloak;
    use keycloak;
    CREATE USER IF NOT EXISTS 'keycloak'@'%' IDENTIFIED BY 'keycloak';
    GRANT ALL PRIVILEGES ON *.* TO 'keycloak'@'%' WITH GRANT OPTION;
    GRANT ALL PRIVILEGES ON *.* TO 'portal'@'%' WITH GRANT OPTION;
    FLUSH PRIVILEGES;
  EOF
  kd exec -i -n portal-system mariadb-primary-0 --container mariadb -- mariadb -uroot -pCloud000@ keycloak < 20251109.mariadb.keycloak.sql
  kd exec -i -n portal-system mariadb-primary-0 --container mariadb -- mariadb -uroot -pCloud000@ OKE_DEVOPS < 20251106.mariadb.OKE_DEVOPS.sql
  kd exec -i -n portal-system mariadb-primary-0 --container mariadb -- mariadb -uroot -pCloud000@ ECP_USER < 20251106.mariadb.ECP_USER.sql
  kd exec -i -n portal-system mariadb-primary-0 --container mariadb -- mariadb -uroot -pCloud000@ ECP_ADMIN < 20251106.mariadb.ECP_ADMIN.sql

  #stg
  ks exec -i -n portal-system mariadb-primary-0 --container mariadb -- mariadb -uroot -pCloud000@ <<EOF
    create database OKE_DEVOPS;
    create database ECP_USER;
    create database ECP_ADMIN;
    create database keycloak;
    use keycloak;
    CREATE USER IF NOT EXISTS 'keycloak'@'%' IDENTIFIED BY 'keycloak';
    GRANT ALL PRIVILEGES ON *.* TO 'keycloak'@'%' WITH GRANT OPTION;
    GRANT ALL PRIVILEGES ON *.* TO 'portal'@'%' WITH GRANT OPTION;
    FLUSH PRIVILEGES;
  EOF
  ks exec -i -n portal-system mariadb-primary-0 --container mariadb -- mariadb -uroot -pCloud000@ keycloak < 20251109.mariadb.keycloak.sql
  ks exec -i -n portal-system mariadb-primary-0 --container mariadb -- mariadb -uroot -pCloud000@ OKE_DEVOPS < 20251106.mariadb.OKE_DEVOPS.sql
  ks exec -i -n portal-system mariadb-primary-0 --container mariadb -- mariadb -uroot -pCloud000@ ECP_USER < 20251106.mariadb.ECP_USER.sql
  ks exec -i -n portal-system mariadb-primary-0 --container mariadb -- mariadb -uroot -pCloud000@ ECP_ADMIN < 20251106.mariadb.ECP_ADMIN.sql

Troubleshooting
“`bash
# Time out countermeasures
### E1215 12:29:09.296432 1320884 websocket.go:500] Websocket Ping failed: write tcp 10.255.57.88:56312->10.255.57.182:6443: use of closed network connection
### error: error reading from error stream: next reader: read tcp 10.255.57.88:56312->10.255.57.182:6443: i/o timeout

Send to each masternode

scp ./20251106.mariadb.ECP_ADMIN.sql root@dev-cp1:admin.sql
scp ./20251106.mariadb.ECP_ADMIN.sql root@stg-cp1:admin.sql

Run the following within each masternode

bash

kubectl cp ./admin.sql -n portal-system mariadb-primary-0:/bitnami/mariadb/admin.sql

bash

kubectl exec -it -n portal-system mariadb-primary-0 --container mariadb -- bash

## in container
cd /bitnami/mariadb
mariadb -uroot -pCloud000@ ECP_ADMIN < admin.sql
## exit
```

Step7. Install Portal-System (Middlewares)

bash

k apply -f dev/keycloak.yaml -f stg/keycloak.yaml

Step8. Install Portal

bash

GIT_ROOT=$(git rev-parse --show-toplevel2>/dev/null)
#helmfile.yaml.gotmpl에 통합됨
### cd $GIT_ROOT/_gitops/manifests/cmp
### kd apply -f browser-pv.yaml -f cronjob-sa.yaml
### ks apply -f browser-pv.yaml -f cronjob-sa.yaml
cd$GIT_ROOT/_gitops/manifests/vault
# 다음을 "secret/dev/portal/cronjob"."cronjob.accessToken"에 replace
kd get secret -n cronjob cronjob-admin-token -ojsonpath='{.data.token}'| base64 -d
# 다음을 "secret/prd/portal/cronjob"."cronjob.accessToken"에 replace
ks get secret -n cronjob cronjob-admin-token -ojsonpath='{.data.token}'| base64 -d
# cronjob sa 수정
vi ./20251215.renew-secrets.json
k apply -f dev/cmp.yaml -f stg/cmp.yaml

in NFS Server

bash

# softlink 생성
ln -s $(ls -l /srv/k8s-mgnt --format=single-column | grep apache) /root/apache_home

Application Uninstallation

`argo-cd` Management

bash

argocd app delete management-system/$argocd_appname

`helmfile` Management

bash

helmfile destroy

Features

Confirm Rancher Password

bash

#password 확인
kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}{{"\n"}}'

In case of Rancher Password error (under 12 Characters)

bash

#reset
kubectl -n cattle-system exec $(kubectl -n cattle-system get pods -l app=rancher | grep '1/1' | head -1 | awk '{ print $1 }') -- reset-password

Fredric News

Leave a Reply Cancel reply

Helmfile Declarative CD Archi: Multi-Env

Information

References

gitops repository

Cluster Version

Architectures

Cluster Installation

Global Setting

in NFS Server

NFS Server Installation

in Business-cluster HAProxy Servers

HAProxy Installation & setting

in Management-cluster

host setup

installation Management Cluster

installation Business Clusters

trouble shooting

When Business Clusters Installed (in Management-cluster)

Import Kubeconfig

all worker Nodes

Application Installation

Required

in Business-clusters

Step0. Prepare Worker Nodes

in Management-cluster

Step1. Helmfile Sync (To set ArgoCD)

Step2. Restore Image Registry

Step3. Install & Restore Vault

Step4. Etc.

Step5. Add Clusters to Argo CD

Step6. Install & Restore Database

Send to each masternode

Run the following within each masternode

Step7. Install Portal-System (Middlewares)

Step8. Install Portal

in NFS Server

Application Uninstallation

argo-cd Management

helmfile Management

Features

Confirm Rancher Password

In case of Rancher Password error (under 12 Characters)

Comments

Leave a Reply Cancel reply

`argo-cd` Management

`helmfile` Management