DevSecOps with OpenShift – Image Signing

INTRODUCTION

Just like an RPM signing, container image singing can be used to verify authenticity of a container image created by users/developers throughout image life-cycle.

The concept of verification are similar to RPM gpgcheck verification.

Refer here for containerized image signing and scanning operator framework produced by Red Hat container-cop, automatically by requesting the signing/scanning service.

As for understanding purposes, steps in this blog are manual step.

CONFIGURATION

Blog Prerequisites

While OpenShift can have varieties of container run-time option. This blog will focus for below environment. The selection was due to nature that docker by default not like other run-time will not honor policy.json out-of-the-box.

  • OCP running on full RHEL (7.5)
  • Docker container run-time (not CRI-O or others)
  • OCP 3.11 (Tested on)

Configuration Steps

We going to use master01 for signing node, while for production purpose, it is a best practise to have particular node for signing and kept very secure.

Overview of steps will be taken

  1. Create GPG Key Pair.
  2. Enable V2 endpoint for OCP registry.
  3. Configure atomic trust.
  4. Enforce Docker signature verification.
  5. Sign, Push and Verify image from OCP.
  6. Docker GPG check pull test.

1. Create GPG Key Pair

Existing GPG keypair can be used for this activity, in case we dont have any keypair yet.
[[email protected] ~]# gpg2 --gen-key

List generated keypair:

[[email protected] ~]# gpg --list-secret-keys
/root/.gnupg/secring.gpg
------------------------
sec   2048R/687C588F 2019-02-24
uid                  OCP Image Signer (Key for OCP Image) <[email protected]>
ssb   2048R/A31CA531 2019-02-24

[[email protected] ~]# gpg --list-key
/root/.gnupg/pubring.gpg
------------------------
pub   2048R/687C588F 2019-02-24
uid                  OCP Image Signer (Key for OCP Image) <[email protected]>
sub   2048R/A31CA531 2019-02-24

[[email protected] ~]# gpg --fingerprint
/root/.gnupg/pubring.gpg
------------------------
pub   2048R/687C588F 2019-02-24
      Key fingerprint = 2AA9 DA41 1D89 A57D 9353  6F5B 4ABF BCBA 687C 588F
uid                  OCP Image Signer (Key for OCP Image) <[email protected]>
sub   2048R/A31CA531 2019-02-24

[[email protected] ~]#

On all nodes, ensure below public key content exported and exists. Copy /etc/pki/container/key.pub to all OPC nodes. This will act as verification key against signer key, just like RPM signing.

[[email protected] ~]# mkdir -p /etc/pki/containers/
[[email protected] ~]# gpg2 --armor --export --output /etc/pki/containers/key.pub [email protected]

2. Enable V2 endpoint for OCP registry

Let`s freeze the automatic rollout for docker-registry.

[[email protected] ~]# oc rollout pause deploymentconfig docker-registry -n default

Set registry environment to enable V2 schema.

[[email protected] ~]#  oc set env dc/docker-registry REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ACCEPTSCHEMA2=true

Resume docker-registry rollout

[[email protected] ~]# oc rollout  resume dc/docker-registry -n default
deploymentconfig.apps.openshift.io/docker-registry resumed

[[email protected] ~]# oc rollout  latest dc/docker-registry -n default
deploymentconfig.apps.openshift.io/docker-registry rolled out

Now lets see the header (expecting 401 return since no token provided) and look for `X-Registry-Supports-Signatures: 1`:

[[email protected] ~]# curl -kv https://docker-registry.default.svc:5000/v2/
* About to connect() to docker-registry.default.svc port 5000 (#0)
*   Trying 10.51.1.99...
* Connected to docker-registry.default.svc (10.51.1.99) port 5000 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
* Server certificate:
*      subject: CN=10.51.1.99
*      start date: Feb 12 14:17:40 2019 GMT
*      expire date: Feb 11 14:17:41 2021 GMT
*      common name: 10.51.1.99
*      issuer: [email protected]
> GET /v2/ HTTP/1.1
> User-Agent: curl/7.29.0
> Host: docker-registry.default.svc:5000
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Content-Type: application/json; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
< Www-Authenticate: Bearer realm="https://docker-registry.default.svc:5000/openshift/token"
< X-Registry-Supports-Signatures: 1
< Date: Sun, 24 Feb 2019 13:33:45 GMT
< Content-Length: 87
<
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}
* Connection #0 to host docker-registry.default.svc left intact
[[email protected] ~]#

Next we going to enable and configure atomic trust, hence only allowed registry and/or signed image allowed to be pulled into the node.

3. Configure atomic trust

This step can be automated via Ansible or other automation mean. In this example we are using master01 as configuration node and copy configs to all nodes.

As an example of Ansible play:

- hosts: OSEv3
  tasks:
  - name: Create /etc/pki/containers directory
    file: path=/etc/pki/containers state=directory
  - name: Create /etc/containers/registries.d directory
    file: path=/etc/containers/registries.d state=directory
  - name: Copy trusted public keys
          copy: src=/etc/pki/containers/ dest=/etc/pki/containers
  - name: Copy container trust policy
    copy: src=/etc/containers/policy.json
          dest=/etc/containers/policy.json
  - name: Copy signature server configuration files
    copy: src=/etc/containers/registries.d/
          dest=/etc/containers/registries.d/

As default policy, reject all:

[[email protected] ~]# atomic trust default reject

Now we add our internal OCP registry(or any registry required to pull/push image):

[[email protected] ~]# atomic trust add docker-registry-default.apps.bytewise.com.my  --pubkeys /etc/pki/containers/key.pub
[[email protected] ~]# atomic trust add docker-registry.default.svc:5000 --pubkeys /etc/pki/containers/key.pub
[[email protected] ~]# atomic trust add quay.bytewise.com.my -t insecureAcceptAnything

Let list our atomic trust:

[[email protected] ~]# atomic trust show
* (default)                         reject                              
docker-registry-default.apps.bytewise.com.my signed [email protected]             
docker-registry.default.svc:5000    signed [email protected]             
quay.bytewise.com.my                accept 

/etc/containers/policy.json looks like this:

{
    "default": [
        {
            "type": "reject"
        }
    ],
    "transports": {
        "docker": {
            "docker-registry-default.apps.bytewise.com.my": [
                {
                    "keyType": "GPGKeys",
                    "type": "signedBy",
                    "keyData": "#########"
                }
            ],
            "quay.bytewise.com.my": [
                {
                    "type": "insecureAcceptAnything"
                }
            ],
            "docker-registry.default.svc:5000": [
                {
                    "keyType": "GPGKeys",
                    "type": "signedBy",
                    "keyData": "#########"
                }
            ]
        },
        "docker-daemon": {
            "": [
                {
                    "type": "insecureAcceptAnything"
                }
            ]
        }
    }
}

4. Enforce Docker signature verification

To enforce docker to verify image signature (on all OCP nodes):

NOTE: Ensure no duplicate line for ‘”signature-verification”: true’ in /etc/sysconfig/docker, or you may use configuration here as well instead of daemon.json.

[[email protected] ~]# cat /etc/docker/daemon.json
{
    "log-driver": "journald",
    "signature-verification": true
}

[[email protected] ~]# systemctl restart docker

5. Sign, Push and Verify image from OCP

Now we going to push one test image (note debug is on for testing purposes):

[[email protected] ~]# atomic --debug push --type atomic --sign-by [email protected] docker-registry.default.svc:5000/django/httpd:latest

Once the image pushed, inspect the imageStream, where you can see the `Status: Unverified`:

[[email protected] sigstore]#  oc describe istag httpd:latest
Image Name:          sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33
Docker Image:          docker-registry.default.svc:5000/django/[email protected]:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33
Name:               sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33
Created:          29 minutes ago
Annotations:          image.openshift.io/dockerLayersOrder=ascending
               image.openshift.io/manifestBlobStored=true
               openshift.io/image.managed=true
Image Size:          49.39MB in 5 layers
Layers:               22.5MB     sha256:6ae821421a7debccb4151f7a50dc8ec0317674429bec0f275402d697047a8e96
               153B     sha256:0ceda4df88c8d23a0cd3c077725e3ac77c63786eca7bca89051e3cc2c7400aae
               10.33MB     sha256:24f08eb4db686b2136baac67538688d99064dc5a4a6a3b9ec821b7e4af71d6c3
               16.55MB     sha256:ddf4fc3180816e62e990fdb516292a3d4c20faf53e044d722848e5f0bdcc7d19
               302B     sha256:fc5812428ac05db13815379a1e08657f76d8906d9cd56fce6d1d161a10fdca80
Image Signatures:     
               Name:     sha256:c[email protected]c902aa576b9214ded37ed4e89afebbcf
               Type:     atomic
               Status:     Unverified

Image Created:          11 days ago
Author:               <none>
Arch:               amd64
Command:          httpd-foreground
Working Dir:          /usr/local/apache2
User:               <none>
Exposes Ports:          80/tcp
Docker Labels:          <none>
Environment:          PATH=/usr/local/apache2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
               HTTPD_PREFIX=/usr/local/apache2
               HTTPD_VERSION=2.4.38
               HTTPD_SHA256=7dc65857a994c98370dc4334b260101a7a04be60e6e74a5c57a6dee1bc8f394a
               HTTPD_PATCHES=
               APACHE_DIST_URLS=https://www.apache.org/dyn/closer.cgi?action=download&filename=      https://www-us.apache.org/dist/      https://www.apache.org/dist/      https://archive.apache.org/dist/

Now lets verified our image and save it in OCP:

[[email protected] ~]# oc adm verify-image-signature sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33 --expected-identity docker-registry.default.svc:5000/django/httpd:latest --public-key /etc/pki/containers/key.pub --save
image "sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33" identity is now confirmed (signed by GPG key "4ABFBCBA687C588F

Now let re-inspect out the imageStream, and it should shown ‘Status: Verified`:

[[email protected] pem]#  oc describe istag httpd:latest -n django
Image Name:          sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33
Docker Image:          docker-registry.default.svc:5000/django/[email protected]:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33
Name:               sha256:ce3878bf368075638dcd5cce8770443dcbc29f8d9bbe30029e65d34cbf31dc33
Created:          34 minutes ago
Annotations:          image.openshift.io/dockerLayersOrder=ascending
               image.openshift.io/manifestBlobStored=true
               openshift.io/image.managed=true
Image Size:          49.39MB in 5 layers
Layers:               22.5MB     sha256:6ae821421a7debccb4151f7a50dc8ec0317674429bec0f275402d697047a8e96
               153B     sha256:0ceda4df88c8d23a0cd3c077725e3ac77c63786eca7bca89051e3cc2c7400aae
               10.33MB     sha256:24f08eb4db686b2136baac67538688d99064dc5a4a6a3b9ec821b7e4af71d6c3
               16.55MB     sha256:ddf4fc3180816e62e990fdb516292a3d4c20faf53e044d722848e5f0bdcc7d19
               302B     sha256:fc5812428ac05db13815379a1e08657f76d8906d9cd56fce6d1d161a10fdca80
Image Signatures:     
               Name:          sha256:c[email protected]c902aa576b9214ded37ed4e89afebbcf
               Type:          atomic
               Status:          Verified
               Issued By:     4ABFBCBA687C588F
               :          Signature is Trusted (verified by user "ocpadmin" on 2019-02-24 19:07:21 +0800 +08)
               :          Signature is ForImage ( on 2019-02-24 19:07:21 +0800 +08)
Image Created:          11 days ago
Author:               <none>
Arch:               amd64
Command:          httpd-foreground
Working Dir:          /usr/local/apache2
User:               <none>
Exposes Ports:          80/tcp
Docker Labels:          <none>
Environment:          PATH=/usr/local/apache2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
               HTTPD_PREFIX=/usr/local/apache2
               HTTPD_VERSION=2.4.38
               HTTPD_SHA256=7dc65857a994c98370dc4334b260101a7a04be60e6e74a5c57a6dee1bc8f394a
               HTTPD_PATCHES=
               APACHE_DIST_URLS=https://www.apache.org/dyn/closer.cgi?action=download&filename=      https://www-us.apache.org/dist/      https://www.apache.org/dist/      https://archive.apache.org/dist/

6. Docker GPG check pull test

Signed image:

[[email protected] ~]# atomic trust show
* (default)                         accept                              
docker-registry-default.apps.bytewise.com.my signed [email protected]             
docker-registry.default.svc:5000    signed [email protected]             
quay.bytewise.com.my                accept

[[email protected] ~]# docker pull docker-registry.default.svc:5000/openshift/django-psql-example:latest
Trying to pull repository docker-registry.default.svc:5000/openshift/django-psql-example ... sha256:1331148170ce4c0c7befc07d038560a7c01ed3d19d1b26f2e12c731cfe268157:
Pulling from docker-registry.default.svc:5000/openshift/django-psql-example23113ae36f8e: Already exists d134b18b98b0: Already exists e9c030a1a5e3:
Pull complete 784f9bf04822: Pull complete 8d1fafd69d24:
Pull complete 29d3ba629150: Pull complete Digest: sha256:1331148170ce4c0c7befc07d038560a7c01ed3d19d1b26f2e12c731cfe268157Status:
Downloaded newer image for docker-registry.default.svc:5000/openshift/django-psql-example:latest

Unsigned image:

[[email protected] ~]# atomic trust show
* (default)                         accept                              
docker-registry-default.apps.bytewise.com.my signed [email protected]             
docker-registry.default.svc:5000    signed [email protected]             
quay.bytewise.com.my                accept

[[email protected] ~]# docker pull docker-registry.default.svc:5000/openshift/hello-world:latestTrying to pull repository docker-registry.default.svc:5000/openshift/hello-world ... docker-registry.default.svc:5000/openshift/hello-world:latest isn't allowed: A signature was required, but no signature exists

Test with wrong gpg public key:

[[email protected] ~]# atomic trust show
* (default)                         reject                              
docker-registry.default.svc:5000    signed [email protected]
           
[[email protected] ~]# docker pull docker-registry.default.svc:5000/httpd/httpd:latest
Trying to pull repository docker-registry.default.svc:5000/httpd/httpd ...
docker-registry.default.svc:5000/httpd/httpd:latest isn't allowed: Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"4ABFBCBA687C588F", Status:gpgme.Error{err:0x7000009}, Timestamp:time.Time{wall:0x0, ext:63686601828, loc:(*time.Location)(0x2560640)}, ExpTimestamp:time.Time{wall:0x0, ext:62135596800, loc:(*time.Location)(0x2560640)}, WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:2


CONCLUSION

1. Above configuration are true for docker run-time environment, while atomic or cri-o as an example will honour policy.json by default.

2. While above is manual steps, docker should able to block unverified image automatically via policy.json with signature verification sets to true.

3. If we do not want to configure docker signature check and only using OCP objects, the only way that possible is to check image signature via  `oc adm verify-image-signature` status and annnotate the image with  `images.openshift.io/deny-execution: true` via pipeline.

4. There is one issue/RFE created in upstream to automatically check image signature during deployment:

OpenShift should automatically verify signatures on images during pod deployments. · Issue #21689 · openshift/origin · G… 

Muhammad Aizuddin Zali

Red Hat APAC-SEATH Consultant for OpenShift Application Development and Platform Delivery Specialist.

You may also like...

Leave a Reply

avatar
  Subscribe  
Notify of
%d bloggers like this: