TACO 2.0 Infra: K8s, Ceph & OpenStack

💡 TACO 2.0 Infra: K8s, Ceph & OpenStack — Executive Summary

Technical architectural analysis, practical configuration manifests, and deployment instructions for TACO 2.0 Infra: K8s, Ceph & OpenStack.

This manual was written based on the EC2 environment created by AWS, and is an AIO (All-In-One) guide that installs Kubernetes, Ceph, and OpenStack all in one VM.

Installation Environment

Single Host Specification

Instance: AWS EC2
Flavor: m5.2xlarge
vCPU: 8
MEM: 32G \<–24G or more recommended

Volume Disk

Root Disk Volume: 200G \<–160G or more recommended
Additional Empty Volume: 50G \<–50G or more recommended

Network Settings

Port open: 80, 30608, 31000 (30001, 30009, 30018, 32000 when +LMA is installed)

Installing Guide

Account Settings

The account used in this manual is centos, and commands starting with 〈\$〉 mean that they are executed in the centos account.

Grant sudo permission

Set as follows so that you can use sudo without a password in your centos account.

txt

$ sudo visudo
## Allow root to run any commands anywhere
root    ALL=(ALL)       ALL
centos  ALL=(ALL:ALL) NOPASSWD: ALL

Ssh password settings

Change the variable below to yes to enable ssh login via password.

txt

$ sudo vi /etc/ssh/sshd_config
PasswordAuthentication yes

Set the password and reflect the changed sshd settings.

txt

$ sudo passwd centos
$ sudo systemctl restart sshd

Pre-installation

Download the packages and source code required to install TACO.

Update and download packages

txt

$ sudo yum update -y
$ sudo yum install -y bridge-utils epel-release git
$ sudo yum install -y python-pip
$ sudo yum update -y

Download tacoplay

tacoplay is a program that automatically installs TACO using the ansible playbook collection.

txt

$ git clone -b taco-v20.05 --single-branch https://github.com/openinfradev/tacoplay.git ~/tacoplay
$ cd $_

Download the packages and source code required for tacoplay.

txt

$ sudo pip install --upgrade pip
$ sudo pip install -r requirements.txt --upgrade --ignore-installed
$ ./fetch-sub-projects.sh

A bridge network must be configured to allocate the IP resources held by the AIO node to the VMs that will be created on OpenStack. For this, check the necessary local information using the method below.

{ ethernet_interface }, { interface_MAC }, { host_ip }

{ ethernet_interface }: ens5 \<–Line 7 of the output result below. If the instance is not created by AWS, it may be output with a name such as eth0.

{ interface_MAC } : 02\:ae\:fa\:f2\:88:84 \<–Line 8 of the output result below

{ host_ip } : 172.32.0.81 \<–Line 9 of the output below

txt

$ ip a
##(example)
1  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
2      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3      inet 127.0.0.1/8 scope host lo
4         valid_lft forever preferred_lft forever
5      inet6 ::1/128 scope host
6         valid_lft forever preferred_lft forever
7  2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
8      link/ether 02:ae:fa:f2:88:84 brd ff:ff:ff:ff:ff:ff
9      inet 172.32.0.81/24 brd 172.32.0.255 scope global dynamic ens5
10        valid_lft 3520sec preferred_lft 3520sec
11    inet6 fe80::ae:faff:fef2:8884/64 scope link
12        valid_lft forever preferred_lft forever

{ gateway }: 172.32.0.1 \<–You can find out through 《ip route》 below.

txt

$ ip route
##(example)
default via 172.32.0.1 dev ens5
172.32.0.0/24 dev ens5 proto kernel scope link src 172.32.0.81

Create a bridge network

Based on the values confirmed above, start creating a bridge network named br-data. The account used is root, and commands starting with 〈#〉 mean that they are executed in the root account. Anything starting with 〈##〉 means a comment.

txt

$ sudo su -
# cd /etc/sysconfig/network-scripts/

##{ } 안에 알맞은 값을 대입하여 아래 설정을 ifcfg-{ ethernet_interface }에 저장한다
# vi ifcfg-{ ethernet_interface }   ##should be edited
DEVICE={ ethernet_interface }   ##should be edited
ONBOOT=yes
TYPE=Ethernet
BRIDGE=br-data
BOOTPROTO=none
NM_CONTROLLED=no

##{ } 안에 알맞은 값을 대입하여 아래 설정을 ifcfg-br-data에 저장한다
# vi ifcfg-br-data
BOOTPROTO=none
DEFROUTE=yes
DEVICE=br-data
GATEWAY={ gateway }   ##should be edited
IPADDR={ host_ip }   ##should be edited
NETMASK=255.255.255.0
ONBOOT=yes
TYPE=Bridge
STP=no
NM_CONTROLLED=no

The settings are reflected based on the values confirmed above.

txt

# systemctl restart network
# ip link add veth0 type veth peer name veth1
# ip link set veth0 up
# ip link set veth1 up

##{ } 안에 알맞은 값을 대입하여 아래 명령을 수행한다. 두 명령을 ';'을 통해 연속적으로 수행하지 않으면 ssh 접속이 끊길 수 있으니 주의한다.
# brctl addif br-data veth1;ifconfig br-data hw ether { interface_MAC }   ##should be edited

Once you have completed configuring the bridge network, switch to the centos account.

txt

# su - centos

Edit Inventory

Check the local information needed to set up the inventory using the method below.

{ Additional_Empty_Volume } : nvme1n1 \<–Added 50G empty volume

txt

$ lsblk
##(example)
nvme0n1     259:0    0  200G  0 disk
└─nvme0n1p1 259:1    0  200G  0 part /
nvme1n1     259:2    0   50G  0 disk

{ host_ip } : 172.32.0.81 \<– Can be confirmed in line 9 of the output result below. When configuring a bridge network, use the { host_ip } (or the ip held by br-data) confirmed at the network configuration stage.
{ network_cidr }: 172.32.0.0/24 \<–The value of changing the 4th octet of 172.32.0.81/24 to 0, which can be seen in the 9th line output below. When a bridge network is configured, it is obtained through the { host_ip } (or the IP held by br-data) confirmed at the network configuration stage.

txt

$ ip a
##(example)br-data 구성하기 전
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:ae:fa:f2:88:84 brd ff:ff:ff:ff:ff:ff
    inet 172.32.0.81/24 brd 172.32.0.255 scope global dynamic ens5
       valid_lft 3520sec preferred_lft 3520sec
   inet6 fe80::ae:faff:fef2:8884/64 scope link
       valid_lft forever preferred_lft forever

If you have configured a bridge network, results similar to the following will be output. At this time, { host_ip } and { network_cidr } refer to br-data.

txt

$ ip a
##(example)br-data 구성한 후
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever.
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq master br-data state UP group default qlen 1000
    link/ether 02:ae:fa:f2:88:84 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ae:faff:fef2:8884/64 scope link
       valid_lft forever preferred_lft forever
3: br-data: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 02:ae:fa:f2:88:84 brd ff:ff:ff:ff:ff:ff
    inet 172.32.0.81/24 brd 172.32.0.255 scope global br-data
       valid_lft forever preferred_lft forever
    inet6 fe80::ae:faff:fef2:8884/64 scope link
       valid_lft forever preferred_lft forever
4: veth1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-data state UP group default qlen 1000
    link/ether 1e:88:df:ce:3a:43 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::1c88:dfff:fece:3a43/64 scope link
       valid_lft forever preferred_lft forever
5: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 06:12:1a:0a:65:25 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::412:1aff:fe0a:6525/64 scope link
       valid_lft forever preferred_lft forever

Inventory settings

Modify the values of the five items in the provided sample extra-vars.yml file as follows.

txt

##{ } 안에 알맞은 값을 대입하여 아래 설정을 extra-vars.yml에 저장한다.
$ vi ~/tacoplay/inventory/sample/aio/extra-vars.yml
taco_apps: ["openstack"]
monitor_interface: br-data
public_network: { network_cidr }   ##should be edited
cluster_network: { network_cidr }   ##should be edited
lvm_volumes:
  - data: /dev/{ Additional_Empty_Volume }   ##should be edited

(optional) Inventory settings for LMA (Logging, Monitoring, Alerting) installation

When LMA is installed, a dashboard is provided where you can check logs and usage status of resources managed by TACO.

Modify the value of one item in the provided sample extra-vars.yml as follows.

txt

##taco_apps의 value에 "lma"를 추가하면 자동으로 LMA를 설치한다.
$ vi ~/tacoplay/inventory/sample/aio/extra-vars.yml
taco_apps: ["openstack","lma"]

Run tacoplay

Once you have completed all of the above settings, run tacoplay.

txt

$ cd ~/tacoplay/
$ ansible-playbook -b -i inventory/sample/aio/hosts.ini -e @inventory/sample/aio/extra-vars.yml site.yml

Depending on the test environment specifications, deployment completion time may vary from 40 minutes to 2 hours.

Install Taco apps

Installation of OpenStack

taco_apps such as OpenStack through Tacoplay use [Decapod](https://github.com/openinfradev/decapod-flow.git). The manual below explains how to use Argo CLI. It can also be run through Argo UI (http://{ host_ip }:30004/). ..code-block:: bash

\$ git clone https://github.com/openinfradev/decapod-flow.git \$ cd decapod-flow/workflows \$ argo submit –from wftmpl/prepare-manifest -n argo \$ argo list -n argo // until prepare-manifest-XXX workflow completes Wait.

\$ argo submit openstack-infra-wf.yaml // Wait until the openstack-infra-XXX workflow is completed. \$ argo submit openstack-components-wf.yaml

(Optional) Customizing OpenStack

If you want to view or modify the configuration for OpenStack installed above, [decapod-base-yaml](https://github.com/openinfradev/decapod-base-yaml.git) and [decapod-site-yaml](https://github.com/openinfradev/decapod-site-yaml.git).

Fork [decapod-site-yaml](https://github.com/openinfradev/decapod-site-yaml.git).
Change the values in decapod-site-yaml/openstack/site/hanu-deploy-apps/site-values.yaml and commit.

3. Run prepare-manifest again with Argo CLI. ..code-block:: bash

References and Guidelines (NOTE)

\$ argo submit –from wftmpl/prepare-manifest -p site_yaml_url=https://github.com/{ your_repo }/decapod-site-yaml.git

4. Redistribute OpenStack. ..code-block:: bash

References and Guidelines (NOTE)

\$ argo submit openstack-infra-wf.yaml // Wait until the openstack-infra-XXX workflow is completed. \$ argo submit openstack-components-wf.yaml

txt

$ watch 'kubectl get pods -n openstack'   ##openstack을 구성하는 모든 파드를 모니터링
$ watch 'kubectl get pods -n openstack | grep -v Comp'   ##Completed 상태인 파드를 제외하고 모니터링
$ watch 'kubectl get pods -n openstack | grep -v Comp | grep -v Runn'   ##Completed 혹은 Running 상태인 파드를 제외하고 모니터링

$ watch 'kubectl get pods -n lma'   ##lma를 구성하는 모든 파드를 모니터링
$ watch 'kubectl get pods -n fed'   ##fed(fedaration) 관련 파드를 모니터링

$ watch 'kubectl get pods -A'   ##모든 K8s 파드를 모니터링(kube-system, openstack, lma, fed)

Check OpenStack installation

Connect to { host_ip }:31000 through a web browser and check whether the openstack web console appears.

domain : default

id: admin

pw : password

LMA access

If you have installed LMA, refer to the connection information below and connect using a web browser.

Kibana: http://{ host_ip }:30001/

ID / Password: elastic / tacoword

Grafana: http://{ host_ip }:30009/

ID / Password: admin / password

Configure network topology for creating OpenStack VMs

You must configure the network to create an instance in OpenStack. The following is the procedure for configuring an example network through the os_client created in the centos account.

txt

$ sh /home/centos/os_client.sh
~$ openstack network create private-net
~$ openstack subnet create --network private-net --subnet-range 172.30.1.0/24 --dns-nameserver 8.8.8.8 private-subnet

~$ openstack network create --external --share --provider-network-type flat --provider-physical-network provider provider-net
~$ openstack subnet create --network provider-net --subnet-range 192.168.97.0/24 --dns-nameserver 8.8.8.8 provider-subnet --allocation-pool 192.168.97.122=192.168.97.122,192.168.97.46=192.168.97.46,192.168.97.231=192.168.97.231,192.168.97.115=192.168.97.115


~$ openstack network create --external --share --provider-network-type flat --provider-physical-network provider provider-net
~$ openstack subnet create --network provider-net --subnet-range 192.168.97.0/24 --dns-nameserver 8.8.8.8 provider-subnet--allocation-pool 192.168.97.91=192.168.97.91,192.168.97.70=192.168.97.70,192.168.97.31=192.168.97.31,192.168.97.182=192.168.97.182

~$ openstack network create --external --share --provider-network-type flat --provider-physical-network provider provider-net
~$ openstack subnet create --network provider-net --subnet-range 192.168.97.0/24 --dns-nameserver 8.8.8.8 provider-subnet--allocation-pool 192.168.97.52=192.168.97.52,192.168.97.206=192.168.97.206,192.168.97.192=192.168.97.192,192.168.97.13=192.168.97.13


~$ openstack router create admin-router
~$ openstack router add subnet admin-router private-subnet
~$ openstack router set --external-gateway provider-net admin-router
~$ openstack router show admin-router

~$ openstack security group list --project admin | grep default | awk '{print $2}'
##출력되는 값을 { security_group } 이라고 하자

##{ }안에 알맞은 값을 대입하여 명령을 실행한다.
~$ openstack security group rule create --proto tcp --remote-ip 0.0.0.0/0 --dst-port 1:65535 --ingress { security_group }
~$ openstack security group rule create --protocol icmp --remote-ip 0.0.0.0/0 { security_group }
~$ openstack security group rule create --protocol icmp --remote-ip 0.0.0.0/0 --egress { security_group }

Once you have configured the network, you can connect to { host_ip }:31000 and add an instance in the Compute > Instance tab. If you created an instance using the provided cirros image, click the instance name to access the console tab. The default login information for cirros is cirros / gocubsgo. (If the console does not open normally, repeat refreshing the web page.)

Trouble Shooting

How to check ansible log

Check /tmp/ansible.log for the log created by default. If you want to manage the log separately, you can create the log in the desired file by adding the <> example_file.log_0> option.
If you add the -vvvv option to the ansible-playbook command, more specific logs will be recorded.

How to perform only some roles using tags when reinstalling ansible due to a problem occurring during installation

When running tacoplay, installation proceeds in the order of roles written in tacoplay/site.yml. Broadly speaking, installation proceeds in the following order: ceph – K8s – taco_app (OpenStack and LMA). If you want to partially install it, run the command below.

txt

##1. 초기 세팅 및 ceph의 설치를 진행하는 커맨드(ceph이 이미 설치된 경우 에러가 발생할 수 있으니 주의한다.)
$ ansible-playbook -b -i inventory/sample/aio/hosts.ini -e @inventory/sample/aio/extra-vars.yml site.yml --tags setup-os,ceph,ceph-post-install --skip-tags k8s

txt

##2. ceph이 정상적으로 설치되었을 때, K8s를 설치하는 커맨드(ceph을 중복으로 설치하게 되면 문제가 발생하여 스킵해준다)
$ ansible-playbook -b -i inventory/sample/aio/hosts.ini -e @inventory/sample/aio/extra-vars.yml site.yml --tags ceph-post-install,k8s,taco-clients --skip-tags ceph

txt

##3. K8s까지 정상적으로 설치되었을 때, taco_app(오픈스택 및 LMA)의 배포 혹은 남은 role을 수행하는 커맨드
$ ansible-playbook -b -i inventory/sample/aio/hosts.ini -e @inventory/sample/aio/extra-vars.yml site.yml --skip-tags ceph,k8s

When problems occur with K8s installation

Check whether K8s resources with the kube-system namespace are working properly.

txt

$ kubectl get pods -n kube-system
$ kubectl get services -n kube-system
$ kubectl get deployments -n kube-system

If a phrase like “The connection to the server localhost:8080 was refused – did you specify the right host or port?”

txt

$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

Execute the above commands sequentially. This occurs when the root account can access the K8s cluster, but a user account such as centos cannot. (Reference: https://snowdeer.github.io/kubernetes/2018/02/13/kubernetes-can-not-use-kubectl/)

When problems occur with OpenStack installation

Check /tmp/openstack-deployment.log for installation log.
Check whether K8s resources with the openstack namespace are working properly.

txt

$ kubectl get pods -n openstack

If there is a problem with the pod, check the events and log.

txt

$ kubectl get pods -n openstack   ##문제가 생긴 pod의 이름을 확인한다.
$ kubectl describe pods -n openstack example_pod_name
$ kubectl logs -n openstack example_pod_name

Check whether helm installation is normal. Installation of helm is carried out in tacoplay/kubespray/roles/kubernetes-apps/helm/tasks/main.yml.

txt

$ helm version
Client: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}

Check whether the helm chart has been deployed properly.

txt

$ helm list -a

When the horizon pod does not reach the ready state and restart is repeated

If the single host instance used has low specifications, problems may occur as the compress-related role within the horizon pod is processed late. Access the horizon deployment settings using the command below, find the two 〈initialDelaySeconds〉 items and change the value to 180, and find the two 〈periodSeconds〉 items and change the value to 600.

txt

$ kubectl edit deployment -n openstack horizon
  86         livenessProbe:
  87           failureThreshold: 3
  88           httpGet:
  89             path: /
  90             port: 80
  91             scheme: HTTP
  92           initialDelaySeconds: 180   ##HERE
  93           periodSeconds: 600   ##HERE
  94           successThreshold: 1
  95           timeoutSeconds: 5

 101         readinessProbe:
 102           failureThreshold: 3
 103           httpGet:
 104             path: /
 105             port: 80
 106             scheme: HTTP
 107           initialDelaySeconds: 180   ##HERE
 108           periodSeconds: 600   ##HERE
 109           successThreshold: 1
 110           timeoutSeconds: 1

Fredric News

Leave a Reply Cancel reply

TACO 2.0 Infra: K8s, Ceph & OpenStack

Comments

Leave a Reply Cancel reply