OpenShift 4: EgressIP for egress(outbound) connection

Introduction

By default, all namespaces egress networking will be using host IP where the pod sits as SNAT rule, similar to how our home router work to access the internet.

For access control and traceability, this pose some difficulty and security issue since all pods regardless which namespace it sit will use common host IP.

With OpenShift, we have a concept known as EgressIP. With EgressIP, we can assign an IP or set of IPs to be assigned to particular namespace that we required to have more fine grain control on egress route. Throughout the namespace lifecycle, it will use this EgressIP assigned for any egress connection to external network(out from SDN).

There are two types of EgressIP configuration:

  • Automatically assigned EgressIP
  • Manually assigned EgressIP

For details of the configuration kindly refer to official documentation.

Example of Typical Use Case

Firewall rule to access certain external target

EgressIP configured on Firewall to allow access to external database

Configuring all host IP to be allowed on firewall to access secured resource is less efficient to manage and furthermore pose greater attack surface (imagine if the cluster has hundreds worker in it, hence hundreds of IPs on the firewall to be allowed).

With EgressIP, we can assign one EgressIP and allow that IP to access the resource on the firewall.

Automatically Assigned EgressIP

EgressIP being assigned to primary interface of the node and will be automatically reassigned when the node is unreachable

As the name suggest, this type installation is the easiest way of letting EgressIP to be automatically assigned. Cluster administrator just need to assign an EgressCIDR to node required to host the EgressIP and then patch the netnamespace for the namespace to select which IP to be used.

IMPORTANT: Since the CIDR are being used as the same as primary network interface(interface of host IP as seen by control plane), IP address management is critical to avoid routing issue. As an example EgressIP selected already being used by some other node in the subnet, this will cause ARP issue since two same IPs exist on different MAC address. This can be controlled by using proper range in CIDR format.

Configuring netnamespace and hostsubnet:

#> oc patch netnamespace mywebserver --type=merge -p \
  '{"egressIPs": [ "192.168.50.120"]}'
  
#> oc patch hostsubnet worker01 --type=merge -p \
  '{"egressCIDRs": ["192.168.50.0/24"]}'
   
#> oc patch hostsubnet worker02 --type=merge -p \
  '{"egressCIDRs": ["192.168.50.0/24"]}'

EgressIP now automatically being assigned by SDN controller:

#> oc get hostsubnets.network.openshift.io 
NAME       HOST       HOST IP          SUBNET         EGRESS CIDRS        EGRESS IPS
master01   master01   192.168.50.141   10.10.0.0/23                       
master02   master02   192.168.50.142   10.9.0.0/23                        
master03   master03   192.168.50.143   10.8.0.0/23                        
worker01   worker01   192.168.50.144   10.11.0.0/23   [192.168.50.0/24]   
worker02   worker02   192.168.50.145   10.8.2.0/23    [192.168.50.0/24]   [192.168.50.120]

Now lets kill worker02 and observe the IP reassigned to next available node:

#>oc get nodes
NAME       STATUS     ROLES    AGE   VERSION
master01   Ready      master   23h   v1.17.1
master02   Ready      master   23h   v1.17.1
master03   Ready      master   23h   v1.17.1
worker01   Ready      worker   23h   v1.17.1
worker02   NotReady   worker   23h   v1.17.1

Now the EgressIP should move to worker01 (the only next available node):

#> oc get hostsubnets.network.openshift.io 
NAME       HOST       HOST IP          SUBNET         EGRESS CIDRS        EGRESS IPS
master01   master01   192.168.50.141   10.10.0.0/23                       
master02   master02   192.168.50.142   10.9.0.0/23                        
master03   master03   192.168.50.143   10.8.0.0/23                        
worker01   worker01   192.168.50.144   10.11.0.0/23   [192.168.50.0/24]   [192.168.50.120]
worker02   worker02   192.168.50.145   10.8.2.0/23    [192.168.50.0/24]   

Statically Assigned EgressIP

EgressIP manually assigned to each of the nodes, and netnamespace configured to use both IP for high availability.

Next available configuration is manually assigned EgressIP, cluster administrator need to assigned each of the IP that can be hosted by the node and use at least two of those IP for netnamespace to provide high availability.

Each node should have specific IP and that IP cant be assigned to another node it will lead to this error:

#> oc get hostsubnets.network.openshift.io 
NAME       HOST       HOST IP          SUBNET         EGRESS CIDRS   EGRESS IPS
master01   master01   192.168.50.141   10.10.0.0/23                  
master02   master02   192.168.50.142   10.9.0.0/23                   
master03   master03   192.168.50.143   10.8.0.0/23                   
worker01   worker01   192.168.50.144   10.11.0.0/23                  [192.168.50.120 192.168.50.121]
worker02   worker02   192.168.50.145   10.8.2.0/23                   [192.168.50.120 192.168.50.121]
#> oc logs -f sdn-8zhv7
E0509 08:09:03.956672    2649 egressip.go:370] Multiple EgressIPs (192.168.50.120, 192.168.50.121) for VNID 5281170 on node 192.168.50.144
E0509 08:09:03.956695    2649 egressip.go:370] Multiple EgressIPs (192.168.50.121, 192.168.50.120) for VNID 5281170 on node 192.168.50.144
E0509 08:09:04.747512    2649 egressip.go:370] Multiple nodes (192.168.50.144, 192.168.50.145) claiming EgressIP 192.168.50.120
E0509 08:09:04.747538    2649 egressip.go:370] Multiple nodes (192.168.50.144, 192.168.50.145) claiming EgressIP 192.168.50.121

To statically assigned EgressIP, patch netnamespace and hostsubnet:

#> oc patch netnamespace mywebserver --type=merge -p \
  '{"egressIPs": [ "192.168.50.120", "192.168.50.121"]}'
  
#> oc patch hostsubnet worker01 --type=merge -p \
  '{"egressIPs": [ "192.168.50.120"]}'
  
#> oc patch hostsubnet worker02 --type=merge -p \
  '{"egressIPs":[ "192.168.50.121"]}'

Now the pod in the namespace has two IPs attached to two different nodes that it can use for outbound connection, it will switch to next IP when first IP unreachable:

#> oc get hostsubnets.network.openshift.io 
NAME       HOST       HOST IP          SUBNET         EGRESS CIDRS   EGRESS IPS
master01   master01   192.168.50.141   10.10.0.0/23                  
master02   master02   192.168.50.142   10.9.0.0/23                   
master03   master03   192.168.50.143   10.8.0.0/23                   
worker01   worker01   192.168.50.144   10.11.0.0/23                  [192.168.50.120]
worker02   worker02   192.168.50.145   10.8.2.0/23                   [192.168.50.121]
#> oc get netnamespaces.network.openshift.io mywebserver 
NAME          NETID     EGRESS IPS
mywebserver   5281170   [192.168.50.120 192.168.50.121]

Summary

With EgressIP, it much more easier to control traffic and allow specific traffic to specific target(s). However this EgressIP only can be attached to primary interface (interface of host IP as seen by control plane), hence multi-homed node cant be attached to different interface except a default gateway, else we will see this error:

E0509 07:13:30.090807    2838 egressip.go:121] Error assigning Egress IP "10.10.10.81": egress IP "10.10.10.81" is not in local network 192.168.50.0/24 of interface enp1s0

Muhammad Aizuddin Zali

Red Hat APAC-SEATH Senior Platform Consultant for OpenShift.

You may also like...

%d bloggers like this: