DevSecOps with OpenShift – Image Signing

INTRODUCTION

Just like an RPM signing, container image singing can be used to verify authenticity of a container image created by users/developers throughout image life-cycle.

The concept of verification are similar to RPM gpgcheck verification.

Refer here for containerized image signing and scanning operator framework produced by Red Hat container-cop, automatically by requesting the signing/scanning service.

As for understanding purposes, steps in this blog are manual step.

CONFIGURATION

Blog Prerequisites

While OpenShift can have varieties of container run-time option. This blog will focus for below environment. The selection was due to nature that docker by default not like other run-time will not honor policy.json out-of-the-box.

  • OCP running on full RHEL (7.5)
  • Docker container run-time (not CRI-O or others)
  • OCP 3.11 (Tested on)

Configuration Steps

We going to use master01 for signing node, while for production purpose, it is a best practise to have particular node for signing and kept very secure.

Overview of steps will be taken

  1. Create GPG Key Pair.
  2. Enable V2 endpoint for OCP registry.
  3. Configure atomic trust.
  4. Enforce Docker signature verification.
  5. Sign, Push and Verify image from OCP.
  6. Docker GPG check pull test.

1. Create GPG Key Pair

Existing GPG keypair can be used for this activity, in case we dont have any keypair yet.

[[email protected] ~]# gpg2 --gen-key

List generated keypair:

[[email protected] ~]# gpg --list-secret-keys
/root/.gnupg/secring.gpg
sec 2048R/687C588F 2019-02-24
uid OCP Image Signer (Key for OCP Image) [email protected]
ssb 2048R/A31CA531 2019-02-24
[[email protected] ~]# gpg --list-key
/root/.gnupg/pubring.gpg
pub 2048R/687C588F 2019-02-24
uid OCP Image Signer (Key for OCP Image) [email protected]
sub 2048R/A31CA531 2019-02-24
[[email protected] ~]# gpg --fingerprint
/root/.gnupg/pubring.gpg
pub 2048R/687C588F 2019-02-24
Key fingerprint = 2AA9 DA41 1D89 A57D 9353 6F5B 4ABF BCBA 687C 588F
uid OCP Image Signer (Key for OCP Image) [email protected]
sub 2048R/A31CA531 2019-02-24
[[email protected] ~]#

On all nodes, ensure below public key content exported and exists. Copy /etc/pki/container/key.pub to all OPC nodes. This will act as verification key against signer key, just like RPM signing.

[[email protected] ~]# mkdir -p /etc/pki/containers/
[[email protected] ~]# gpg2 --armor --export --output /etc/pki/containers/key.pub [email protected]

2. Enable V2 endpoint for OCP registry

Let`s freeze the automatic rollout for docker-registry.

[[email protected] ~]# oc rollout pause deploymentconfig docker-registry -n default

Set registry environment to enable V2 schema.

[[email protected] ~]#  oc set env dc/docker-registry REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ACCEPTSCHEMA2=true

Resume docker-registry rollout

[[email protected] ~]# oc rollout  resume dc/docker-registry -n default
deploymentconfig.apps.openshift.io/docker-registry resumed
[[email protected] ~]# oc rollout latest dc/docker-registry -n default
deploymentconfig.apps.openshift.io/docker-registry rolled out

Now lets see the header (expecting 401 return since no token provided) and look for `X-Registry-Supports-Signatures: 1`:

[[email protected] ~]# curl -kv https://docker-registry.default.svc:5000/v2/
About to connect() to docker-registry.default.svc port 5000 (#0)
Trying 10.51.1.99…
Connected to docker-registry.default.svc (10.51.1.99) port 5000 (#0)
Initializing NSS with certpath: sql:/etc/pki/nssdb
skipping SSL peer certificate verification
SSL connection using TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
Server certificate:
subject: CN=10.51.1.99
start date: Feb 12 14:17:40 2019 GMT
expire date: Feb 11 14:17:41 2021 GMT
common name: 10.51.1.99
issuer: [email protected]
GET /v2/ HTTP/1.1
User-Agent: curl/7.29.0
Host: docker-registry.default.svc:5000
Accept: /
< HTTP/1.1 401 Unauthorized
< Content-Type: application/json; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
< Www-Authenticate: Bearer realm="https://docker-registry.default.svc:5000/openshift/token"
< X-Registry-Supports-Signatures: 1
< Date: Sun, 24 Feb 2019 13:33:45 GMT
< Content-Length: 87
<
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}
Connection #0 to host docker-registry.default.svc left intact
[[email protected] ~]#

Next we going to enable and configure atomic trust, hence only allowed registry and/or signed image allowed to be pulled into the node.

3. Configure atomic trust

This step can be automated via Ansible or other automation mean. In this example we are using master01 as configuration node and copy configs to all nodes.

As an example of Ansible play:

- hosts: OSEv3
tasks:
name: Create /etc/pki/containers directory
file: path=/etc/pki/containers state=directory
name: Create /etc/containers/registries.d directory
file: path=/etc/containers/registries.d state=directory
name: Copy trusted public keys
copy: src=/etc/pki/containers/ dest=/etc/pki/containers
name: Copy container trust policy
copy: src=/etc/containers/policy.json
dest=/etc/containers/policy.json
name: Copy signature server configuration files
copy: src=/etc/containers/registries.d/
dest=/etc/containers/registries.d/

As default policy, reject all:

[[email protected] ~]# atomic trust default reject

Now we add our internal OCP registry(or any registry required to pull/push image):

[[email protected] ~]# atomic trust add docker-registry-default.apps.bytewise.com.my  --pubkeys /etc/pki/containers/key.pub
[[email protected] ~]# atomic trust add docker-registry.default.svc:5000 --pubkeys /etc/pki/containers/key.pub
[[email protected] ~]# atomic trust add quay.bytewise.com.my -t insecureAcceptAnything

Let list our atomic trust:

[[email protected] ~]# atomic trust show
(default) reject
docker-registry-default.apps.bytewise.com.my signed [email protected]
docker-registry.default.svc:5000 signed [email protected]
quay.bytewise.com.my accept

/etc/containers/policy.json looks like this:

{
"default": [
{
"type": "reject"
}
],
"transports": {
"docker": {
"docker-registry-default.apps.bytewise.com.my": [
{
"keyType": "GPGKeys",
"type": "signedBy",
"keyData": "#########"
}
],
"quay.bytewise.com.my": [
{
"type": "insecureAcceptAnything"
}
],
"docker-registry.default.svc:5000": [
{
"keyType": "GPGKeys",
"type": "signedBy",
"keyData": "#########"
}
]
},
"docker-daemon": {
"": [
{
"type": "insecureAcceptAnything"
}
]
}
}
}

4. Enforce Docker signature verification

To enforce docker to verify image signature (on all OCP nodes):

NOTE: Ensure no duplicate line for ‘”signature-verification”: true’ in /etc/sysconfig/docker, or you may use configuration here as well instead of daemon.json.

[[email protected] ~]# cat /etc/docker/daemon.json
{
"log-driver": "journald",
"signature-verification": true
}
[[email protected] ~]# systemctl restart docker

5. Sign, Push and Verify image from OCP

Now we going to push one test image (note debug is on for testing purposes):

[[email protected] ~]# atomic --debug push --type atomic --sign-by [email protected] docker-registry.default.svc:5000/django/httpd:latest

Once the image pushed, inspect the imageStream, where you can see the `Status: Unverified`:

[[email protected] sigstore]#  oc describe istag httpd:latest
Image Name: sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33
Docker Image: docker-registry.default.svc:5000/django/[email protected]:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33
Name: sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33
Created: 29 minutes ago
Annotations: image.openshift.io/dockerLayersOrder=ascending
image.openshift.io/manifestBlobStored=true
openshift.io/image.managed=true
Image Size: 49.39MB in 5 layers
Layers: 22.5MB sha256:6ae821421a7debccb4151f7a50dc8ec0317674429bec0f275402d697047a8e96
153B sha256:0ceda4df88c8d23a0cd3c077725e3ac77c63786eca7bca89051e3cc2c7400aae
10.33MB sha256:24f08eb4db686b2136baac67538688d99064dc5a4a6a3b9ec821b7e4af71d6c3
16.55MB sha256:ddf4fc3180816e62e990fdb516292a3d4c20faf53e044d722848e5f0bdcc7d19
302B sha256:fc5812428ac05db13815379a1e08657f76d8906d9cd56fce6d1d161a10fdca80
Image Signatures:
Name: sha256:c[email protected]c902aa576b9214ded37ed4e89afebbcf
Type: atomic
Status: Unverified
Image Created: 11 days ago
Author:
Arch: amd64
Command: httpd-foreground
Working Dir: /usr/local/apache2
User:
Exposes Ports: 80/tcp
Docker Labels:
Environment: PATH=/usr/local/apache2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HTTPD_PREFIX=/usr/local/apache2
HTTPD_VERSION=2.4.38
HTTPD_SHA256=7dc65857a994c98370dc4334b260101a7a04be60e6e74a5c57a6dee1bc8f394a
HTTPD_PATCHES=
APACHE_DIST_URLS=https://www.apache.org/dyn/closer.cgi?action=download&filename= https://www-us.apache.org/dist/ https://www.apache.org/dist/ https://archive.apache.org/dist/

Now lets verified our image and save it in OCP:

[[email protected] ~]# oc adm verify-image-signature sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33 --expected-identity docker-registry.default.svc:5000/django/httpd:latest --public-key /etc/pki/containers/key.pub --save
image "sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33" identity is now confirmed (signed by GPG key "4ABFBCBA687C588F

Now let re-inspect out the imageStream, and it should shown ‘Status: Verified`:

[[email protected] pem]#  oc describe istag httpd:latest -n django
Image Name: sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33
Docker Image: docker-registry.default.svc:5000/django/[email protected]:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33
Name: sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33
Created: 34 minutes ago
Annotations: image.openshift.io/dockerLayersOrder=ascending
image.openshift.io/manifestBlobStored=true
openshift.io/image.managed=true
Image Size: 49.39MB in 5 layers
Layers: 22.5MB sha256:6ae821421a7debccb4151f7a50dc8ec0317674429bec0f275402d697047a8e96
153B sha256:0ceda4df88c8d23a0cd3c077725e3ac77c63786eca7bca89051e3cc2c7400aae
10.33MB sha256:24f08eb4db686b2136baac67538688d99064dc5a4a6a3b9ec821b7e4af71d6c3
16.55MB sha256:ddf4fc3180816e62e990fdb516292a3d4c20faf53e044d722848e5f0bdcc7d19
302B sha256:fc5812428ac05db13815379a1e08657f76d8906d9cd56fce6d1d161a10fdca80
Image Signatures:
Name: sha256:c[email protected]c902aa576b9214ded37ed4e89afebbcf
Type: atomic
Status: Verified
Issued By: 4ABFBCBA687C588F
: Signature is Trusted (verified by user "ocpadmin" on 2019-02-24 19:07:21 +0800 +08)
: Signature is ForImage ( on 2019-02-24 19:07:21 +0800 +08)
Image Created: 11 days ago
Author:
Arch: amd64
Command: httpd-foreground
Working Dir: /usr/local/apache2
User:
Exposes Ports: 80/tcp
Docker Labels:
Environment: PATH=/usr/local/apache2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HTTPD_PREFIX=/usr/local/apache2
HTTPD_VERSION=2.4.38
HTTPD_SHA256=7dc65857a994c98370dc4334b260101a7a04be60e6e74a5c57a6dee1bc8f394a
HTTPD_PATCHES=
APACHE_DIST_URLS=https://www.apache.org/dyn/closer.cgi?action=download&filename= https://www-us.apache.org/dist/ https://www.apache.org/dist/ https://archive.apache.org/dist/

6. Docker GPG check pull test

Signed image:

[[email protected] ~]# atomic trust show
(default) accept
docker-registry-default.apps.bytewise.com.my signed [email protected]
docker-registry.default.svc:5000 signed [email protected]
quay.bytewise.com.my accept
[[email protected] ~]# docker pull docker-registry.default.svc:5000/openshift/django-psql-example:latest
Trying to pull repository docker-registry.default.svc:5000/openshift/django-psql-example … sha256:1331148170ce4c0c7befc07d038560a7c01ed3d19d1b26f2e12c731cfe268157:
Pulling from docker-registry.default.svc:5000/openshift/django-psql-example23113ae36f8e: Already exists d134b18b98b0: Already exists e9c030a1a5e3:
Pull complete 784f9bf04822: Pull complete 8d1fafd69d24:
Pull complete 29d3ba629150: Pull complete Digest: sha256:1331148170ce4c0c7befc07d038560a7c01ed3d19d1b26f2e12c731cfe268157Status:
Downloaded newer image for docker-registry.default.svc:5000/openshift/django-psql-example:latest

Unsigned image:

[[email protected] ~]# atomic trust show
(default) accept
docker-registry-default.apps.bytewise.com.my signed [email protected]
docker-registry.default.svc:5000 signed [email protected]
quay.bytewise.com.my accept
[[email protected] ~]# docker pull docker-registry.default.svc:5000/openshift/hello-world:latestTrying to pull repository docker-registry.default.svc:5000/openshift/hello-world … docker-registry.default.svc:5000/openshift/hello-world:latest isn't allowed: A signature was required, but no signature exists

Test with wrong gpg public key:

[[email protected] ~]# atomic trust show
(default) reject
docker-registry.default.svc:5000 signed [email protected]
[[email protected] ~]# docker pull docker-registry.default.svc:5000/httpd/httpd:latest
Trying to pull repository docker-registry.default.svc:5000/httpd/httpd …
docker-registry.default.svc:5000/httpd/httpd:latest isn't allowed: Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"4ABFBCBA687C588F", Status:gpgme.Error{err:0x7000009}, Timestamp:time.Time{wall:0x0, ext:63686601828, loc:(time.Location)(0x2560640)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(time.Location)(0x2560640)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:2

CONCLUSION

1. Above configuration are true for docker run-time environment, while atomic or cri-o as an example will honour policy.json by default.

2. While above is manual steps, docker should able to block unverified image automatically via policy.json with signature verification sets to true.

3. If we do not want to configure docker signature check and only using OCP objects, the only way that possible is to check image signature via  `oc adm verify-image-signature` status and annnotate the image with  `images.openshift.io/deny-execution: true` via pipeline.

4. There is one issue/RFE created in upstream to automatically check image signature during deployment:

OpenShift should automatically verify signatures on images during pod deployments. · Issue #21689 · openshift/origin · G… 

Muhammad Aizuddin Zali

Red Hat APAC-SEATH Consultant for OpenShift Application Development and Platform Delivery Specialist.

You may also like...

Leave a Reply

avatar
  Subscribe  
Notify of
%d bloggers like this: