OpenShift 4: Libvirt UPI installation.

Hello everyone, just a couple of months ago, Red Hat has released a shiny OpenShift 4 [1] based on CoreOS technology. In this guide, we going to see how we can install OCP4 UPI on libvirt as baremetal type deployment.

A short definition of:
UPI: User Provided Infrastructure (User manually prepare machine for bootstrap)
IPI: Installer Provided Infrastructure (OCP Installer automatically terraforming machine for bootstrap)

1. Pre-Requisites

  • A Valid Red Hat OpenShift subscription
  • DNS Server (Helper Node)
  • HAProxy Server (Helper Node)
  • DHCP Server (Helper Node)
    • For DHCP Clustering refer here.
  • PXE/TFTP Server (Helper Node)
  • Libvirt Host
    • 1 x Helper Node
      -> To host above services
      -> 4vCPU, 4GB RAM
      -> 50 GB Disk
      -> RHEL 8.0
      -> helper.ocp4.techbeatly.com
    • 1 x Bootstrap Node
      -> 4 vCPU, 6GB RAM
      -> 50 GB Disk
      -> Empty Node
      -> PXE Booted
      -> bootstrap.ocp4.techbeatly.com
    • 3 x Master Node
      -> 4 vCPU, 8GB RAM
      -> 50 GB Disk
      -> Empty Node
      -> PXE Booted
      -> master[1-3].ocp4.techbeatly.com
    • 3 x Worker Node
      -> 4 vCPU, 8GB RAM
      -> 50 GB Disk
      -> Empty Node
      -> PXE Booted
      ->worker[1-3].ocp4.techbeatly.com
  • SDN Subnets
    • Cluster Network: 10.128.0.0/14 (host prefix /23)
    • Service Network: 172.30.0.0/16
  • FQDN
    • K8S API: api.ocp4.techbeatly.com
    • Machine API: api-int.ocp4.techbeatly.com
    • Wildcard subdomain: *.apps.ocp4.techbeatly.com

NOTE: Above sizing purely for testing purpose, for actual production load please engage Red Hat Consulting for proper design.

2. Configuration

2.1 Install and configure packages:

[[email protected] ~]# dnf -y install bind bind-utils dhcp-server tftp-server syslinux httpd haproxy
[[email protected] ~]# firewall-cmd --add-service={dhcp,tftp,http,https,dns} --permanent
[[email protected] ~]# firewall-cmd --add-port={6443/tcp,22623/tcp,8080/tcp} --permanent
[[email protected] ~]# firewall-cmd --reload

2.2 Configure DNS

1. Comment out below lines in /etc/named.conf:

#listen-on port 53 { 127.0.0.1; };
#listen-on-v6 port 53 { ::1; };

2. Allow DNS query from the subnet:

allow-query     { localhost;192.168.0.0/24; };

3. Now we need to create DNS hosted zone, append /etc/named.conf:

zone "techbeatly.com" IN {
    type master;
    file "techbeatly.com.db"; 
    allow-update { none; };
};

zone “50.168.192.in-addr.arpa” IN {
    type master;
    file “50.168.192.in-addr.arpa";
};

4. Create a zone database file in /var/named/techbeatly.com.db and /var/named/50.168.192.in-addr.arpa with below content:

[[email protected] ~]# cat /var/named/techbeatly.com.db 
 $TTL     1D
 @        IN  SOA dns.ocp4.techbeatly.com. root.techbeatly.com. (
                       2019022400 ; serial
                       3h         ; refresh
                       15         ; retry
                       1w         ; expire
                       3h         ; minimum
                                                                             )
                  IN  NS  dns.ocp4.techbeatly.com.
 dns.ocp4            IN  A   192.168.50.30
 bootstrap.ocp4            IN  A   192.168.50.60
 master01.ocp4            IN  A   192.168.50.61
 master02.ocp4            IN  A   192.168.50.62
 master03.ocp4            IN  A   192.168.50.63
 etcd-0.ocp4            IN  A   192.168.50.61
 etcd-1.ocp4            IN  A   192.168.50.62
 etcd-2.ocp4            IN  A   192.168.50.63
 api.ocp4               IN  A   192.168.50.30
 api-int.ocp4           IN  A   192.168.50.30
 *.apps.ocp4            IN  A   192.168.50.30
 worker01.ocp4            IN  A   192.168.50.64
 worker02.ocp4            IN  A   192.168.50.64
 worker03.ocp4            IN  A   192.168.50.66
 _etcd-server-ssl._tcp.ocp4    IN  SRV 0 10    2380 etcd-0.ocp4
 _etcd-server-ssl._tcp.ocp4      IN      SRV     0 10    2380 etcd-1.ocp4
 _etcd-server-ssl._tcp.ocp4      IN      SRV     0 10    2380 etcd-2.ocp4


[[email protected] ~]# cat /var/named/50.168.192.in-addr.arp
 $TTL     1D
 @        IN  SOA dns.ocp4.techbeatly.com. root.techbeatly.com. (
                       2019022400 ; serial
                       3h         ; refresh
                       15         ; retry
                       1w         ; expire
                       3h         ; minimum

                  IN  NS  dns.ocp4.techbeatly.com.
60 IN PTR bootstrap.ocp4.techbeatly.com.
61 IN PTR master01.ocp4.techbeatly.com.
62 IN PTR master02.ocp4.techbeatly.com.
63 IN PTR master03.ocp4.techbeatly.com.
64 IN PTR worker01.ocp4.techbeatly.com.
65 IN PTR worker02.ocp4.techbeatly.com.
66 IN PTR worker03.ocp4.techbeatly.com.

TIPS: Do not set PTR record for etcd. PTR record will be used to set the node hostname.

5. Restart named and test the DNS:

[[email protected] ~]# dig @localhost -t srv _etcd-server-ssl._tcp.ocp4.techbeatly.com
 ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-17.P2.el8_0 <<>> @localhost -t srv _etcd-server-ssl._tcp.ocp4.techbeatly.com
 ; (1 server found)
 ;; global options: +cmd
 ;; Got answer:
 ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26440
 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 1, ADDITIONAL: 5
 ;; OPT PSEUDOSECTION:
 ; EDNS: version: 0, flags:; udp: 4096
 ; COOKIE: ae2bc6642a496d42698637d95d1996a47c622a7ab26d73ef (good)
 ;; QUESTION SECTION:
 ;_etcd-server-ssl._tcp.ocp4.techbeatly.com. IN SRV
 ;; ANSWER SECTION:
 _etcd-server-ssl._tcp.ocp4.techbeatly.com. 86400 IN SRV    0 10 2380 etcd-0.ocp4.techbeatly.com.
 _etcd-server-ssl._tcp.ocp4.techbeatly.com. 86400 IN SRV    0 10 2380 etcd-1.ocp4.techbeatly.com.
 _etcd-server-ssl._tcp.ocp4.techbeatly.com. 86400 IN SRV    0 10 2380 etcd-2.ocp4.techbeatly.com.
 ;; AUTHORITY SECTION:
 techbeatly.com.        86400   IN  NS  dns.ocp4.techbeatly.com.
 ;; ADDITIONAL SECTION:
 etcd-0.ocp4.techbeatly.com. 86400 IN    A   192.168.50.61
 etcd-1.ocp4.techbeatly.com. 86400 IN    A   192.168.50.62
 etcd-2.ocp4.techbeatly.com. 86400 IN    A   192.168.50.63
 dns.ocp4.techbeatly.com. 86400    IN  A   192.168.50.30
 ;; Query time: 0 msec
 ;; SERVER: 127.0.0.1#53(127.0.0.1)
 ;; WHEN: Mon Jul 01 13:14:12 +08 2019
 ;; MSG SIZE  rcvd: 318

2.3 Configuring DHCP and PXE

  1. Setup /etc/dhcp/dhcpd.conf with below content (update this as per your environment!):
ddns-update-style interim;
ignore client-updates;
authoritative;
allow booting;
allow bootp;
deny unknown-clients;
default-lease-time -1;
max-lease-time -1;
subnet 192.168.50.0 netmask 255.255.255.0 {
         option routers 192.168.50.1;
         option domain-name-servers 192.168.50.30;
         option ntp-servers time.unisza.edu.my;
         option domain-search "techbeatly.com","ocp4.techbeatly.com";
         filename "pxelinux.0";
         next-server 192.168.50.30;
         host bootstrap { hardware ethernet 52:54:00:7d:2d:b1; fixed-address 192.168.50.60; }
         host master01 { hardware ethernet 52:54:00:7d:2d:b2; fixed-address 192.168.50.61; }
         host master02 { hardware ethernet 52:54:00:7d:2d:b3; fixed-address 192.168.50.62; }
         host master03 { hardware ethernet 52:54:00:7d:2d:b4; fixed-address 192.168.50.63; }
         host worker01 { hardware ethernet 52:54:00:7d:2d:b5; fixed-address 192.168.50.64; }
         host worker02 { hardware ethernet 52:54:00:7d:2d:b6; fixed-address 192.168.50.65; }
         host worker03 { hardware ethernet 52:54:00:7d:2d:c1; fixed-address 192.168.50.66; }

2. Populate /var/lib/tftpboot/pxelinux.cfg/default with below content(install syslinux package if not yet done):

[[email protected] 4.1.0]# cat /var/lib/tftpboot/pxelinux.cfg/default
default menu.c32
prompt 0
timeout 30
menu title **** OpenShift 4 PXE Boot Menu ****
 
label Install CoreOS 4.1.0 Bootstrap Node
 kernel /openshift4/4.1.0/rhcos-4.1.0-x86_64-installer-kernel
 append ip=dhcp rd.neednet=1 coreos.inst.install_dev=vda console=tty0 console=ttyS0 coreos.inst=yes coreos.inst.image_url=http://192.168.50.30:8080/openshift4/4.1.0/images/rhcos-4.1.0-x86_64-metal-bios.raw.gz coreos.inst.ignition_url=http://192.168.50.30:8080/openshift4/4.1.0/ignitions/bootstrap.ign initrd=/openshift4/4.1.0/rhcos-4.1.0-x86_64-installer-initramfs.img
label Install CoreOS 4.1.0 Master Node
 kernel /openshift4/4.1.0/rhcos-4.1.0-x86_64-installer-kernel
 append ip=dhcp rd.neednet=1 coreos.inst.install_dev=vda console=tty0 console=ttyS0 coreos.inst=yes coreos.inst.image_url=http://192.168.50.30:8080/openshift4/4.1.0/images/rhcos-4.1.0-x86_64-metal-bios.raw.gz coreos.inst.ignition_url=http://192.168.50.30:8080/openshift4/4.1.0/ignitions/master.ign initrd=/openshift4/4.1.0/rhcos-4.1.0-x86_64-installer-initramfs.img                                                                                                                                            
label Install CoreOS 4.1.0 Worker Node
 kernel /openshift4/4.1.0/rhcos-4.1.0-x86_64-installer-kernel
 append ip=dhcp rd.neednet=1 coreos.inst.install_dev=vda console=tty0 console=ttyS0 coreos.inst=yes coreos.inst.image_url=http://192.168.50.30:8080/openshift4/4.1.0/images/rhcos-4.1.0-x86_64-metal-bios.raw.gz coreos.inst.ignition_url=http://192.168.50.30:8080/openshift4/4.1.0/ignitions/worker.ign initrd=/openshift4/4.1.0/rhcos-4.1.0-x86_64-installer-initramfs.img

3 . Copy syslinux for PXE boot

 #> cp -rvf /usr/share/syslinux/* /var/lib/tftpboot

4. Start TFTP

#> systemctl start tftp

2.3.1 Optional MAC-Address based boot file

While above procedure is sufficient, you can extend it further to:

  • PXE to offer bootfile based on the MAC Address of the client. Further automate the PXE boot selection menu with zero touch.
  1. Create mac address based file e.g /var/lib/tftpboot/pxelinux.cfg/01-52-54-00-7d-2d-b1 (NOTE that 01- prefix in front is required) with below specific node content. Do this for each of the nodes with node specific mac address and boot content.
default menu.c32
prompt 0
timeout 2
menu title **** OpenShift 4 Bootstrap PXE Boot Menu ****
label Install CoreOS 4.3.0 Bootstrap Node
 kernel /openshift4/4.1.0/rhcos-4.1.0-x86_64-installer-kernel
 append ip=dhcp rd.neednet=1 coreos.inst.install_dev=vda console=tty0 console=ttyS0 coreos.inst=yes coreos.inst.image_url=http://192.168.50.30:8080/openshift4/4.1.0/images/rhcos-4.1.0-x86_64-metal-bios.raw.gz coreos.inst.ignition_url=http://192.168.50.30:8080/openshift4/4.1.0/ignitions/bootstrap.ign initrd=/openshift4/4.1.0/rhcos-4.1.0-x86_64-installer-initramfs.img

2. This will tell PXE server to serve this file content when it sees the MAC address are corresponding to the pxelinux.cfg hence we dont need to navigate the PXE menu, a zero touch.

2.4 Configure webserver to host RHCOS image

  1. Switch httpd to listen from 80 to 8080 in /etc/httpd/conf/httpd.conf:
Listen 8080
[[email protected] 4.1.0]# systemctl restart httpd

2. Create a new directory for hosting the kernel and initramfs for PXE boot:

 [[email protected] 4.1.0]# mkdir -p /var/lib/tftpboot/openshift4/4.1.0
 [[email protected] 4.1.0]# cd /var/lib/tftpboot/openshift4/4.1.0
 [[email protected] 4.1.0]# wget https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.1/latest/rhcos-4.1.0-x86_64-installer-kernel
 [[email protected] 4.1.0]# wget https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.1/latest/rhcos-4.1.0-x86_64-installer-initramfs.img
 [[email protected] 4.1.0]# restorecon -RFv .
 Relabeled /var/lib/tftpboot/openshift4/4.1.0 from unconfined_u:object_r:tftpdir_rw_t:s0 to system_u:object_r:tftpdir_rw_t:s0
 Relabeled /var/lib/tftpboot/openshift4/4.1.0/rhcos-4.1.0-x86_64-installer-kernel from unconfined_u:object_r:tftpdir_rw_t:s0 to system_u:object_r:tftpdir_rw_t:s0
 Relabeled /var/lib/tftpboot/openshift4/4.1.0/rhcos-4.1.0-x86_64-installer-initramfs.img from unconfined_u:object_r:tftpdir_rw_t:s0 to system_u:object_r:tftpdir_rw_t:s0

3. Now, we going to host the Red Hat CoreOS image:

[[email protected] html]# mkdir -p /var/www/html/openshift4/4.1.0/images/
 [[email protected] html]# cd  /var/www/html/openshift4/4.1.0/images/
 [[email protected] images]# wget https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.1/latest/rhcos-4.1.0-x86_64-metal-bios.raw.gz
 [[email protected] images]# restorecon -RFv .
 Relabeled /var/www/html/openshift4/4.1.0/images from unconfined_u:object_r:httpd_sys_content_t:s0 to system_u:object_r:httpd_sys_content_t:s0
 Relabeled /var/www/html/openshift4/4.1.0/images/rhcos-4.1.0-x86_64-metal-bios.raw.gz from unconfined_u:object_r:httpd_sys_content_t:s0 to system_u:object_r:httpd_sys_content_t:s0

2.5 Configure HAProxy as LB

1. A load balance required for balancing masters and ingress router, append /etc/haproxy/haproxy.conf :

---------------------------------------------------------------------
 round robin balancing for OCP Kubernetes API Server
 ---------------------------------------------------------------------
 frontend k8s_api
   bind *:6443
   mode tcp
   default_backend k8s_api_backend
 backend k8s_api_backend
   balance roundrobin
   mode tcp
   server bootstrap 192.168.50.60:6443 check
   server master01 192.168.50.61:6443 check
   server master02 192.168.50.62:6443 check
   server master03 192.168.50.63:6443 check
 ---------------------------------------------------------------------
 round robin balancing for OCP Machine Config Server
 ---------------------------------------------------------------------
 frontend machine_config
   bind *:22623
   mode tcp
   default_backend machine_config_backend
 backend machine_config_backend
   balance roundrobin
   mode tcp
   server bootstrap 192.168.50.60:22623 check
   server master01 192.168.50.61:22623 check
   server master02 192.168.50.62:22623 check
   server master03 192.168.50.63:22623 check
 ---------------------------------------------------------------------
 round robin balancing for OCP Ingress Insecure Port
 ---------------------------------------------------------------------
 frontend ingress_insecure
   bind *:80
   mode tcp
   default_backend ingress_insecure_backend
 backend ingress_insecure_backend
   balance roundrobin
   mode tcp
   server worker01 192.168.50.64:80 check
   server worker02 192.168.50.65:80 check
   server worker03 192.168.50.66:80 check
 ---------------------------------------------------------------------
 round robin balancing for OCP Ingress Secure Port
 ---------------------------------------------------------------------
 frontend ingress_secure
   bind *:443
   mode tcp
   default_backend ingress_secure_backend
 backend ingress_secure_backend
   balance roundrobin
   mode tcp
   server worker01 192.168.50.64:443 check
   server worker02 192.168.50.65:443 check
   server worker03 192.168.50.66:443 check
 ---------------------------------------------------------------------
 Exposing HAProxy Statistic Page
 ---------------------------------------------------------------------
 listen stats
     bind :32700
     stats enable
     stats uri /
     stats hide-version
     stats auth admin:password
 [[email protected] ~]#

2. Allow haproxy to bind to custom port with SELinux:

[[email protected] ~]# semanage port  -a 22623 -t http_port_t -p tcp
[[email protected] ~]# semanage port  -a 6443 -t http_port_t -p tcp
[[email protected] ~]# semanage port  -a 32700 -t http_port_t -p tcp
[[email protected] ~]# semanage port  -l  | grep -w http_port_t
 http_port_t                    tcp      22623, 32700, 6443, 80, 81, 443, 488, 8008, 8009, 8443, 9000

2.6 Configure OpenShift Installer and CLI binary

  1. To start creating ignition files and managing cluster, download installer and client:
[[email protected] ~]# wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/openshift-install-linux-4.1.3.tar.gz
[[email protected] ~]# wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/openshift-client-linux-4.1.3.tar.gz
[[email protected] ~]# tar -xvf openshift-client-linux-4.1.3.tar.gz 
[[email protected] ~]# tar -xvf openshift-install-linux-4.1.3.tar.gz 
[[email protected] ~]# cp -v {oc,kubectl,openshift-install} /usr/bin/
 'oc' -> '/usr/bin/oc'
 'kubectl' -> '/usr/bin/kubectl'
 'openshift-install' -> '/usr/bin/openshift-install'

2. If not yet, create a sshkey pair to use to access CoreOS node later on:

[[email protected] ocp4]# ssh-keygen -t rsa

3. Create installation working directory and start to prepare the ignition file:

NOTE: Replace “pullSecret” and “sshKey” using your own value.

[[email protected] ~]# mkdir -p ocp4
[[email protected] ~]# cd ocp4 
[[email protected] ~]# cat install-config-base.yaml 
apiVersion: v1
baseDomain: techbeatly.com
compute:
- hyperthreading: Enabled
  name: worker
  replicas: 0
controlPlane:
  hyperthreading: Enabled
  name: master
  replicas: 3
metadata:
  name: ocp4
networking:
  clusterNetworks:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16
platform:
  none: {}
pullSecret: 'GET FROM cloud.redhat.com'
sshKey: 'SSH PUBLIC KEY'

4. Now copy install-config-base.yaml to install-config.yaml. (install-config.yaml will be consumed by installer, hence copied from base file instead)

 [[email protected] ocp4]# cp install-config-base.yaml install-config.yaml 
 [[email protected] ocp4]# openshift-install create ignition-configs
 WARNING There are no compute nodes specified. The cluster will not fully initialize without compute nodes. 
 INFO Consuming "Install Config" from target directory 
 [[email protected] ocp4]# ll
 total 288
 drwxr-xr-x. 2 root root     50 Jul  1 16:46 auth
 -rw-r--r--. 1 root root 276768 Jul  1 16:46 bootstrap.ign
 -rw-r--r--. 1 root root   3545 Jul  1 16:42 install-config-base.yaml
 -rw-r--r--. 1 root root   1824 Jul  1 16:46 master.ign
 -rw-r--r--. 1 root root     96 Jul  1 16:46 metadata.json
 -rw-r--r--. 1 root root   1824 Jul  1 16:46 worker.ign
 [[email protected] ocp4]#

5. We need to copy ignition file to http hosted directory:

[[email protected] ocp4]# mkdir /var/www/html/openshift4/4.1.0/ignitions
 [[email protected] ocp4]# cp -v *.ign /var/www/html/openshift4/4.1.0/ignitions/
 'bootstrap.ign' -> '/var/www/html/openshift4/4.1.0/ignitions/bootstrap.ign'
 'master.ign' -> '/var/www/html/openshift4/4.1.0/ignitions/master.ign'
 'worker.ign' -> '/var/www/html/openshift4/4.1.0/ignitions/worker.ign'
 [[email protected] ocp4]# restorecon -RFv /var/www/html/
 Relabeled /var/www/html/openshift4 from unconfined_u:object_r:httpd_sys_content_t:s0 to system_u:object_r:httpd_sys_content_t:s0
 Relabeled /var/www/html/openshift4/4.1.0 from unconfined_u:object_r:httpd_sys_content_t:s0 to system_u:object_r:httpd_sys_content_t:s0
 Relabeled /var/www/html/openshift4/4.1.0/ignitions from unconfined_u:object_r:httpd_sys_content_t:s0 to system_u:object_r:httpd_sys_content_t:s0
 Relabeled /var/www/html/openshift4/4.1.0/ignitions/bootstrap.ign from unconfined_u:object_r:httpd_sys_content_t:s0 to system_u:object_r:httpd_sys_content_t:s0
 Relabeled /var/www/html/openshift4/4.1.0/ignitions/master.ign from unconfined_u:object_r:httpd_sys_content_t:s0 to system_u:object_r:httpd_sys_content_t:s0
 Relabeled /var/www/html/openshift4/4.1.0/ignitions/worker.ign from unconfined_u:object_r:httpd_sys_content_t:s0 to system_u:object_r:httpd_sys_content_t:s0
 [[email protected] ocp4]# 

6. At this moment we are done with the configuration, enable and start all services:

[[email protected] ocp4]# systemctl enable --now haproxy.service dhcpd httpd tftp named

7. We are now should be able to boot up all the nodes and let bootstrap installing the initial cluster!

For automating this via Ansible Playbook using Libvirt, example can be found here.

Muhammad Aizuddin Zali

Red Hat APAC-SEATH Senior Platform Consultant for OpenShift.

You may also like...

%d bloggers like this: