OpenShift 4: Libvirt UPI installation.

Hello everyone, just a couple of month ago, Red Hat has released a shiny OpenShift 4 [1] based on CoreOS technology. In this guide, we going to see how we can install OCP4 UPI on libvirt.

A short definition of:
UPI: User Provided Infrastructure (User manually prepare machine for bootstrap)
IPI: Installer Provided Infrastructure (OCP Installer automatically terraforming machine for bootstrap)

1. Pre-Requisites

  • A Valid Red Hat OpenShift subscription
  • DNS Server (Helper Node)
  • HAProxy Server (Helper Node)
  • DHCP Server (Helper Node)
  • PXE/TFTP Server (Helper Node)
  • Libvirt Host
    • 1 x Helper Node
      -> To host above services
      -> 4vCPU, 4GB RAM
      -> 50 GB Disk
      -> RHEL 8.0
      -> helper.ocp4.techbeatly.com
    • 1 x Bootstrap Node
      -> 4 vCPU, 6GB RAM
      -> 50 GB Disk
      -> Empty Node
      -> PXE Booted
      -> bootstrap.ocp4.techbeatly.com
    • 3 x Master Node
      -> 4 vCPU, 8GB RAM
      -> 50 GB Disk
      -> Empty Node
      -> PXE Booted
      -> master[1-3].ocp4.techbeatly.com
    • 3 x Worker Node
      -> 4 vCPU, 8GB RAM
      -> 50 GB Disk
      -> Empty Node
      -> PXE Booted
      ->worker[1-3].ocp4.techbeatly.com
  • SDN Subnets
    • Cluster Network: 10.128.0.0/14 (host prefix /23)
    • Service Network: 172.30.0.0/16
  • FQDN
    • K8S API: api.ocp4.techbeatly.com
    • Machine API: api-int.ocp4.techbeatly.com
    • Wildcard subdomain: *.apps.ocp4.techbeatly.com

NOTE: Above sizing purely for testing purpose, for actual production load please engage Red Hat Consulting for proper design.

2. Configuration

Install and configure packages:

[[email protected]helper ~]# dnf -y install bind bind-utils dhcp-server tftp-server syslinux httpd haproxy
[[email protected] ~]# firewall-cmd --add-service={dhcp,tftp,http,https,dns} --permanent
[[email protected] ~]# firewall-cmd --add-port={6443/tcp,22623/tcp,8080/tcp} --permanent
[[email protected] ~]# firewall-cmd --reload

Comment out below lines in /etc/named.conf:

#listen-on port 53 { 127.0.0.1; };
#listen-on-v6 port 53 { ::1; };

Enable ACL for request:

allow-query     { localhost;192.168.0.0/24; };

Now we need to create DNS hosted zone, append /etc/named.conf:

zone "techbeatly.com" IN {
    type master;
    file "techbeatly.com.db"; 
    allow-update { none; };
};

Create a zone database file in /var/named/techbeatly.com.db with below content:

[[email protected] ~]# cat /var/named/techbeatly.com.db 
 $TTL     1D
 @        IN  SOA dns.ocp4.techbeatly.com. root.techbeatly.com. (
                       2019022400 ; serial
                       3h         ; refresh
                       15         ; retry
                       1w         ; expire
                       3h         ; minimum
                                                                             )
                  IN  NS  dns.ocp4.techbeatly.com.
 dns.ocp4            IN  A   192.168.50.30
 bootstrap.ocp4            IN  A   192.168.50.60
 master01.ocp4            IN  A   192.168.50.61
 master02.ocp4            IN  A   192.168.50.62
 master03.ocp4            IN  A   192.168.50.63
 etcd-0.ocp4            IN  A   192.168.50.61
 etcd-1.ocp4            IN  A   192.168.50.62
 etcd-2.ocp4            IN  A   192.168.50.63
 api.ocp4               IN  A   192.168.50.30
 api-int.ocp4           IN  A   192.168.50.30
 *.apps.ocp4            IN  A   192.168.50.30
 worker01.ocp4            IN  A   192.168.50.64
 worker02.ocp4            IN  A   192.168.50.64
 worker03.ocp4            IN  A   192.168.50.66
 _etcd-server-ssl._tcp.ocp4    IN  SRV 0 10    2380 etcd-0.ocp4
 _etcd-server-ssl._tcp.ocp4      IN      SRV     0 10    2380 etcd-1.ocp4
 _etcd-server-ssl._tcp.ocp4      IN      SRV     0 10    2380 etcd-2.ocp4
 

Restart named and test the DNS:

[[email protected] ~]# dig @localhost -t srv _etcd-server-ssl._tcp.ocp4.techbeatly.com

 ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-17.P2.el8_0 <<>> @localhost -t srv _etcd-server-ssl._tcp.ocp4.techbeatly.com
 ; (1 server found)
 ;; global options: +cmd
 ;; Got answer:
 ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26440
 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 1, ADDITIONAL: 5
 ;; OPT PSEUDOSECTION:
 ; EDNS: version: 0, flags:; udp: 4096
 ; COOKIE: ae2bc6642a496d42698637d95d1996a47c622a7ab26d73ef (good)
 ;; QUESTION SECTION:
 ;_etcd-server-ssl._tcp.ocp4.techbeatly.com. IN SRV
 ;; ANSWER SECTION:
 _etcd-server-ssl._tcp.ocp4.techbeatly.com. 86400 IN SRV    0 10 2380 etcd-0.ocp4.techbeatly.com.
 _etcd-server-ssl._tcp.ocp4.techbeatly.com. 86400 IN SRV    0 10 2380 etcd-1.ocp4.techbeatly.com.
 _etcd-server-ssl._tcp.ocp4.techbeatly.com. 86400 IN SRV    0 10 2380 etcd-2.ocp4.techbeatly.com.
 ;; AUTHORITY SECTION:
 techbeatly.com.        86400   IN  NS  dns.ocp4.techbeatly.com.
 ;; ADDITIONAL SECTION:
 etcd-0.ocp4.techbeatly.com. 86400 IN    A   192.168.50.61
 etcd-1.ocp4.techbeatly.com. 86400 IN    A   192.168.50.62
 etcd-2.ocp4.techbeatly.com. 86400 IN    A   192.168.50.63
 dns.ocp4.techbeatly.com. 86400    IN  A   192.168.50.30
 ;; Query time: 0 msec
 ;; SERVER: 127.0.0.1#53(127.0.0.1)
 ;; WHEN: Mon Jul 01 13:14:12 +08 2019
 ;; MSG SIZE  rcvd: 318

Next, we going to setting up DHCP with PXE environment.

Setup /etc/dhcp/dhcpd.conf with below content (update this as per your environment!):

ddns-update-style interim;
 ignore client-updates;
 authoritative;
 allow booting;
 allow bootp;
 allow unknown-clients;
 subnet 192.168.50.0 netmask 255.255.255.0 {
         range 192.168.50.60 192.168.50.70;
         option routers 192.168.50.1;
         option domain-name-servers 192.168.50.30;
         option ntp-servers time.unisza.edu.my;
         option domain-search "techbeatly.com","ocp4.techbeatly.com";
         filename "pxelinux.0";
         next-server 192.168.50.30;
         host bootstrap { hardware ethernet 52:54:00:7d:2d:b1; fixed-address 192.168.50.60; option host-name "bootstrap"; }
         host master01 { hardware ethernet 52:54:00:7d:2d:b2; fixed-address 192.168.50.61; option host-name "master01"; }
         host master02 { hardware ethernet 52:54:00:7d:2d:b3; fixed-address 192.168.50.62; option host-name "master02"; }
         host master03 { hardware ethernet 52:54:00:7d:2d:b4; fixed-address 192.168.50.63; option host-name "master03"; }
         host worker01 { hardware ethernet 52:54:00:7d:2d:b5; fixed-address 192.168.50.64; option host-name "worker01"; }
         host worker02 { hardware ethernet 52:54:00:7d:2d:b6; fixed-address 192.168.50.65; option host-name "worker02"; }
         host worker03 { hardware ethernet 52:54:00:7d:2d:c1; fixed-address 192.168.50.66; option host-name "worker03"; }
 }

A load balance required for balancing masters and ingress router, append /etc/haproxy/haproxy.conf :

---------------------------------------------------------------------
 round robin balancing for OCP Kubernetes API Server
 ---------------------------------------------------------------------
 frontend k8s_api
   bind *:6443
   mode tcp
   default_backend k8s_api_backend
 backend k8s_api_backend
   balance roundrobin
   mode tcp
   server bootstrap 192.168.50.60:6443 check
   server master01 192.168.50.61:6443 check
   server master02 192.168.50.62:6443 check
   server master03 192.168.50.63:6443 check
 ---------------------------------------------------------------------
 round robin balancing for OCP Machine Config Server
 ---------------------------------------------------------------------
 frontend machine_config
   bind *:22623
   mode tcp
   default_backend machine_config_backend
 backend machine_config_backend
   balance roundrobin
   mode tcp
   server bootstrap 192.168.50.60:22623 check
   server master01 192.168.50.61:22623 check
   server master02 192.168.50.62:22623 check
   server master03 192.168.50.63:22623 check
 ---------------------------------------------------------------------
 round robin balancing for OCP Ingress Insecure Port
 ---------------------------------------------------------------------
 frontend ingress_insecure
   bind *:80
   mode tcp
   default_backend ingress_insecure_backend
 backend ingress_insecure_backend
   balance roundrobin
   mode tcp
   server worker01 192.168.50.64:80 check
   server worker02 192.168.50.65:80 check
   server worker03 192.168.50.66:80 check
 ---------------------------------------------------------------------
 round robin balancing for OCP Ingress Secure Port
 ---------------------------------------------------------------------
 frontend ingress_secure
   bind *:443
   mode tcp
   default_backend ingress_secure_backend
 backend ingress_secure_backend
   balance roundrobin
   mode tcp
   server worker01 192.168.50.64:443 check
   server worker02 192.168.50.65:443 check
   server worker03 192.168.50.66:443 check
 ---------------------------------------------------------------------
 Exposing HAProxy Statistic Page
 ---------------------------------------------------------------------
 listen stats
     bind :32700
     stats enable
     stats uri /
     stats hide-version
     stats auth admin:password
 [[email protected] ~]# 

Allow haproxy to bind to custom port with SELinux:

[[email protected] ~]# semanage port  -a 22623 -t http_port_t -p tcp
[[email protected] ~]# semanage port  -a 6443 -t http_port_t -p tcp
[[email protected] ~]# semanage port  -a 32700 -t http_port_t -p tcp
[[email protected] ~]# semanage port  -l  | grep -w http_port_t
 http_port_t                    tcp      22623, 32700, 6443, 80, 81, 443, 488, 8008, 8009, 8443, 9000

Populate /var/lib/tftpboot/pxelinux.cfg/default with below content:

[[email protected] 4.1.0]# cat /var/lib/tftpboot/pxelinux.cfg/default
default menu.c32
prompt 0
timeout 30
menu title **** OpenShift 4 PXE Boot Menu ****
 
label Install CoreOS 4.1.0 Bootstrap Node
 kernel /openshift4/4.1.0/rhcos-4.1.0-x86_64-installer-kernel
 append ip=dhcp rd.neednet=1 coreos.inst.install_dev=vda console=tty0 console=ttyS0 coreos.inst=yes coreos.inst.image_url=http://192.168.50.30:8080/openshift4/4.1.0/images/rhcos-4.1.0-x86_64-metal-bios.raw.gz coreos.inst.ignition_url=http://192.168.50.30:8080/openshift4/4.1.0/ignitions/bootstrap.ign initrd=/openshift4/4.1.0/rhcos-4.1.0-x86_64-installer-initramfs.img

label Install CoreOS 4.1.0 Master Node
 kernel /openshift4/4.1.0/rhcos-4.1.0-x86_64-installer-kernel
 append ip=dhcp rd.neednet=1 coreos.inst.install_dev=vda console=tty0 console=ttyS0 coreos.inst=yes coreos.inst.image_url=http://192.168.50.30:8080/openshift4/4.1.0/images/rhcos-4.1.0-x86_64-metal-bios.raw.gz coreos.inst.ignition_url=http://192.168.50.30:8080/openshift4/4.1.0/ignitions/master.ign initrd=/openshift4/4.1.0/rhcos-4.1.0-x86_64-installer-initramfs.img                                                                                                                                            

label Install CoreOS 4.1.0 Worker Node
 kernel /openshift4/4.1.0/rhcos-4.1.0-x86_64-installer-kernel
 append ip=dhcp rd.neednet=1 coreos.inst.install_dev=vda console=tty0 console=ttyS0 coreos.inst=yes coreos.inst.image_url=http://192.168.50.30:8080/openshift4/4.1.0/images/rhcos-4.1.0-x86_64-metal-bios.raw.gz coreos.inst.ignition_url=http://192.168.50.30:8080/openshift4/4.1.0/ignitions/worker.ign initrd=/openshift4/4.1.0/rhcos-4.1.0-x86_64-installer-initramfs.img

Switch httpd to listen from 80 to 8080 in /etc/httpd/conf/httpd.conf:

Listen 8080

[[email protected] 4.1.0]# systemctl restart httpd

Create a new directory for hosting the kernel and initramfs for PXE boot:

 [[email protected] 4.1.0]# mkdir -p /var/lib/tftpboot/openshift4/4.1.0
 [[email protected] 4.1.0]# cd /var/lib/tftpboot/openshift4/4.1.0
 [[email protected] 4.1.0]# wget https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.1/latest/rhcos-4.1.0-x86_64-installer-kernel
 [[email protected] 4.1.0]# wget https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.1/latest/rhcos-4.1.0-x86_64-installer-initramfs.img
 [[email protected] 4.1.0]# restorecon -RFv .
 Relabeled /var/lib/tftpboot/openshift4/4.1.0 from unconfined_u:object_r:tftpdir_rw_t:s0 to system_u:object_r:tftpdir_rw_t:s0
 Relabeled /var/lib/tftpboot/openshift4/4.1.0/rhcos-4.1.0-x86_64-installer-kernel from unconfined_u:object_r:tftpdir_rw_t:s0 to system_u:object_r:tftpdir_rw_t:s0
 Relabeled /var/lib/tftpboot/openshift4/4.1.0/rhcos-4.1.0-x86_64-installer-initramfs.img from unconfined_u:object_r:tftpdir_rw_t:s0 to system_u:object_r:tftpdir_rw_t:s0

Now, we going to host the Red Hat CoreOS image:

 [[email protected] html]# mkdir -p /var/www/html/openshift4/4.1.0/images/
 [[email protected] html]# cd  /var/www/html/openshift4/4.1.0/images/
 [[email protected] images]# wget https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.1/latest/rhcos-4.1.0-x86_64-metal-bios.raw.gz
 [[email protected] images]# restorecon -RFv .
 Relabeled /var/www/html/openshift4/4.1.0/images from unconfined_u:object_r:httpd_sys_content_t:s0 to system_u:object_r:httpd_sys_content_t:s0
 Relabeled /var/www/html/openshift4/4.1.0/images/rhcos-4.1.0-x86_64-metal-bios.raw.gz from unconfined_u:object_r:httpd_sys_content_t:s0 to system_u:object_r:httpd_sys_content_t:s0

To start creating ignition files and managing cluster, download installer and client:

[[email protected] ~]# wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/openshift-install-linux-4.1.3.tar.gz
[[email protected] ~]# wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/openshift-client-linux-4.1.3.tar.gz

[[email protected] ~]# tar -xvf openshift-client-linux-4.1.3.tar.gz 
[[email protected] ~]# tar -xvf openshift-install-linux-4.1.3.tar.gz 

[[email protected] ~]# cp -v {oc,kubectl,openshift-install} /usr/bin/
 'oc' -> '/usr/bin/oc'
 'kubectl' -> '/usr/bin/kubectl'
 'openshift-install' -> '/usr/bin/openshift-install'

If not yet, create a sshkey pair to use to access CoreOS node later on:

[[email protected] ocp4]# ssh-keygen -t rsa

Create installation working directory and start to prepare the ignition file:

[[email protected] ~]# mkdir -p ocp4
[[email protected] ~]# cd ocp4 
[[email protected] ~]# cat install-config-base.yaml 
apiVersion: v1
baseDomain: techbeatly.com
compute:
- hyperthreading: Enabled
  name: worker
  replicas: 0
controlPlane:
  hyperthreading: Enabled
  name: master
  replicas: 3
metadata:
  name: ocp4
networking:
  clusterNetworks:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16
platform:
  none: {}
pullSecret: 'GET FROM cloud.redhat.com'
sshKey: 'SSH PUBLIC KEY'

Now copy install-config-base.yaml to install-config.yaml. (install-config.yaml will be consumed by installer, hence copied from base file instead)

 [[email protected] ocp4]# cp install-config-base.yaml install-config.yaml 
 [[email protected] ocp4]# openshift-install create ignition-configs
 WARNING There are no compute nodes specified. The cluster will not fully initialize without compute nodes. 
 INFO Consuming "Install Config" from target directory 
 [[email protected] ocp4]# ll
 total 288
 drwxr-xr-x. 2 root root     50 Jul  1 16:46 auth
 -rw-r--r--. 1 root root 276768 Jul  1 16:46 bootstrap.ign
 -rw-r--r--. 1 root root   3545 Jul  1 16:42 install-config-base.yaml
 -rw-r--r--. 1 root root   1824 Jul  1 16:46 master.ign
 -rw-r--r--. 1 root root     96 Jul  1 16:46 metadata.json
 -rw-r--r--. 1 root root   1824 Jul  1 16:46 worker.ign
 [[email protected] ocp4]# 

We need to copy ignition file to http hosted directory:

 [[email protected] ocp4]# mkdir /var/www/html/openshift4/4.1.0/ignitions
 [[email protected] ocp4]# cp -v *.ign /var/www/html/openshift4/4.1.0/ignitions/
 'bootstrap.ign' -> '/var/www/html/openshift4/4.1.0/ignitions/bootstrap.ign'
 'master.ign' -> '/var/www/html/openshift4/4.1.0/ignitions/master.ign'
 'worker.ign' -> '/var/www/html/openshift4/4.1.0/ignitions/worker.ign'
 [[email protected] ocp4]# restorecon -RFv /var/www/html/
 Relabeled /var/www/html/openshift4 from unconfined_u:object_r:httpd_sys_content_t:s0 to system_u:object_r:httpd_sys_content_t:s0
 Relabeled /var/www/html/openshift4/4.1.0 from unconfined_u:object_r:httpd_sys_content_t:s0 to system_u:object_r:httpd_sys_content_t:s0
 Relabeled /var/www/html/openshift4/4.1.0/ignitions from unconfined_u:object_r:httpd_sys_content_t:s0 to system_u:object_r:httpd_sys_content_t:s0
 Relabeled /var/www/html/openshift4/4.1.0/ignitions/bootstrap.ign from unconfined_u:object_r:httpd_sys_content_t:s0 to system_u:object_r:httpd_sys_content_t:s0
 Relabeled /var/www/html/openshift4/4.1.0/ignitions/master.ign from unconfined_u:object_r:httpd_sys_content_t:s0 to system_u:object_r:httpd_sys_content_t:s0
 Relabeled /var/www/html/openshift4/4.1.0/ignitions/worker.ign from unconfined_u:object_r:httpd_sys_content_t:s0 to system_u:object_r:httpd_sys_content_t:s0
 [[email protected] ocp4]# 

At this moment we are done with the configuration, enable and start all services:

[[email protected] ocp4]# systemctl enable --now haproxy.service dhcpd httpd tftp named

We are now should be able to boot up all the nodes and let bootstrap installing the initial cluster!

Muhammad Aizuddin Zali

Red Hat APAC-SEATH Consultant for OpenShift Application Development and Platform Delivery Specialist.

You may also like...

2
Leave a Reply

avatar
1 Comment threads
1 Thread replies
1 Followers
 
Most reacted comment
Hottest comment thread
2 Comment authors
Muhammad Aizuddin Bin ZaliPeter Recent comment authors

This site uses Akismet to reduce spam. Learn how your comment data is processed.

  Subscribe  
newest oldest most voted
Notify of
Peter
Guest
Peter

I did follow you steps but I got
something like:
master:
ignition[xxx]: GET error: Get https://api-int.ocp.ocp4.lab:22623/config/master: EOF

worker:
ignition[xxx]: GET error: Get https://api-int.ocp.ocp4.lab:22623/config/woker: EOF

Muhammad Aizuddin Bin Zali
Guest
Muhammad Aizuddin Bin Zali

what curl -kv https://api-int.ocp.ocp4.lab:22623 telling you?

%d bloggers like this: