All posts by ftweedal

Running Keycloak in OpenShift

At PyCon Australia in August I gave a presentation about federated and social identity. I demonstrated concepts using Keycloak, an Open Source, feature rich identity broker. Keycloak is deployed in JBoss, so I wasn’t excited about the prospect of setting up Keycloak from scratch. Fortunately there is an official Docker image for Keycloak, so with that as the starting point I took an opportunity to finally learn about OpenShift v3, too.

This post is simply a recounting of how I ran Keycloak on OpenShift. Along the way we will look at how to get the containerised Keycloak to trust a private certificate authority (CA).

One thing that is not discussed is how to get Keycloak to persist configuration and user records to a database. This was not required for my demo, but it will be important in a production deployment. Nevertheless I hope this article is a useful starting point for someone wishing to deploy Keycloak on OpenShift.

Bringing up a local OpenShift cluster

To deploy Keycloak on OpenShift, one must first have an OpenShift. OpenShift Online is Red Hat’s public PaaS offering. Although running the demo on a public PaaS was my first choice, OpenShift Online was experiencing issues at the time I was setting up my demo. So I sought a local solution. This approach would have the additional benefit of not being subject to the whims of conference networks (or, it was supposed to – but that is a story for another day!)

oc cluster up

Next I tried oc cluster up. oc is the official OpenShift client program. On Fedora, it is provided by the origin-clients package. oc cluster up command pulls required images and brings up an OpenShift cluster running on the system’s Docker infrastructure. The command takes no further arguments; it really is that simple! Or is it…?

% oc cluster up
-- Checking OpenShift client ... OK
-- Checking Docker client ... OK
-- Checking Docker version ... OK
-- Checking for existing OpenShift container ... OK
-- Checking for openshift/origin:v1.5.0 image ...
   Pulling image openshift/origin:v1.5.0
   Pulled 0/3 layers, 3% complete
   ...
   Pulled 3/3 layers, 100% complete
   Extracting
   Image pull complete
-- Checking Docker daemon configuration ... FAIL
   Error: did not detect an --insecure-registry argument on the Docker daemon
   Solution:

     Ensure that the Docker daemon is running with the following argument:
        --insecure-registry 172.30.0.0/16

OK, so it is not that simple. But it got a fair way along, and (kudos to the OpenShift developers) they have provided actionable feedback about how to resolve the issue. I added --insecure-registry 172.30.0.0/16 to the OPTIONS variable in /etc/sysconfig/docker, then restarted Docker and tried again:

% oc cluster up
-- Checking OpenShift client ... OK
-- Checking Docker client ... OK
-- Checking Docker version ... OK
-- Checking for existing OpenShift container ... OK
-- Checking for openshift/origin:v1.5.0 image ... OK
-- Checking Docker daemon configuration ... OK
-- Checking for available ports ... OK
-- Checking type of volume mount ...
   Using nsenter mounter for OpenShift volumes
-- Creating host directories ... OK
-- Finding server IP ...
   Using 192.168.0.160 as the server IP
-- Starting OpenShift container ...
   Creating initial OpenShift configuration
   Starting OpenShift using container 'origin'
   Waiting for API server to start listening
-- Adding default OAuthClient redirect URIs ... OK
-- Installing registry ... OK
-- Installing router ... OK
-- Importing image streams ... OK
-- Importing templates ... OK
-- Login to server ... OK
-- Creating initial project "myproject" ... OK
-- Removing temporary directory ... OK
-- Checking container networking ... OK
-- Server Information ... 
   OpenShift server started.
   The server is accessible via web console at:
       https://192.168.0.160:8443

   You are logged in as:
       User:     developer
       Password: developer

   To login as administrator:
       oc login -u system:admin

Success! Unfortunately, on my machine with several virtual network, oc cluster up messed a bit too much with the routing tables, and when I deployed Keycloak on this cluster it was unable to communicate with my VMs. No doubt these issues could have been solved, but being short on time and with other approaches to try, I abandoned this approach.

Minishift

Minishift is a tool that launches a single-node OpenShift cluster in a VM. It supports a variety of operating systems and hypervisors. On GNU+Linux it supports KVM and VirtualBox.

First install docker-machine and docker-machine-driver-kvm. (follow the instructions at the preceding links). Unfortunately these are not yet packaged for Fedora.

Download and extract the Minishift release for your OS from https://github.com/minishift/minishift/releases.

Run minishift start:

% ./minishift start
-- Installing default add-ons ... OK
Starting local OpenShift cluster using 'kvm' hypervisor...
Downloading ISO 'https://github.com/minishift/minishift-b2d-iso/releases/download/v1.0.2/minishift-b2d.iso'

... wait a while ...

It downloads a boot2docker VM image containing the openshift cluster, boots the VM, and the console output then resembles the output of oc cluster up. I deduce that oc cluster up is being executed on the VM.

At this point, we’re ready to go. Before I continue, it is important to note that once you have access to an OpenShift cluster, the user experience of creating and managing applications is essentially the same. The commands in the following sections are relevant, regardless whether you are running your app on OpenShift online, on a cluster running on your workstation, or anything in between.

Preparing the Keycloak image

The JBoss project provides official Docker images, including an official Docker image for Keycloak. This image runs fine in plain Docker but the directory permissions are not correct for running in OpenShift.

The Dockerfile for this image is found in the jboss-dockerfiles/keycloak repository on GitHub. Although they do not publish an official image for it, this repository also contains a Dockerfile for Keycloak on OpenShift! I was able to build that image myself and upload it to my Docker Hub account. The steps were as follows.

First clone the jboss-dockerfiles repo:

% git clone https://github.com/jboss-dockerfiles/keycloak docker-keycloak
Cloning into 'docker-keycloak'...
remote: Counting objects: 1132, done.
remote: Compressing objects: 100% (22/22), done.
remote: Total 1132 (delta 14), reused 17 (delta 8), pack-reused 1102
Receiving objects: 100% (1132/1132), 823.50 KiB | 158.00 KiB/s, done.
Resolving deltas: 100% (551/551), done.
Checking connectivity... done.

Next build the Docker image for OpenShift:

% docker build docker-keycloak/server-openshift
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM jboss/keycloak:latest
 ---> fb3fc6a18e16
Step 2 : USER root
 ---> Running in 21b672e19722
 ---> eea91ef53702
Removing intermediate container 21b672e19722
Step 3 : RUN chown -R jboss:0 $JBOSS_HOME/standalone &&     chmod -R g+rw $JBOSS_HOME/standalone
 ---> Running in 93b7d11f89af
 ---> 910dc6c4a961
Removing intermediate container 93b7d11f89af
Step 4 : USER jboss
 ---> Running in 8b8ccba42f2a
 ---> c21eed109d12
Removing intermediate container 8b8ccba42f2a
Successfully built c21eed109d12

Finally, tag the image into the repo and push it:

% docker tag c21eed109d12 registry.hub.docker.com/frasertweedale/keycloak-openshift

% docker login -u frasertweedale registry.hub.docker.com
Password:
Login Succeeded

% docker push registry.hub.docker.com/frasertweedale/keycloak-openshift
... wait for upload ...
latest: digest: sha256:c82c3cc8e3edc05cfd1dae044c5687dc7ebd9a51aefb86a4bb1a3ebee16f341c size: 2623

Adding CA trust

For my demo, I used a local FreeIPA installation to issue TLS certificates for the the Keycloak app. I was also going to carry out a scenario where I configure Keycloak to use that FreeIPA installation’s LDAP server to authenticate users. I wanted to use TLS everywhere (eat your own dog food!) I needed the Keycloak application to trust the CA of one of my local FreeIPA installations. This made it necessary to build another Docker image based on the keycloak-openshift image, with the appropriate CA trust built in.

The content of the Dockerfile is:

FROM frasertweedale/keycloak-openshift:latest
USER root
COPY ca.pem /etc/pki/ca-trust/source/anchors/ca.pem
RUN update-ca-trust
USER jboss

The file ca.pem contains the CA certificate to add. It must be in the same directory as the Dockerfile. The build copies the CA certificate to the appropriate location and executes update-ca-trust to ensure that applications – including Java programs – will trust the CA.

Following the docker build I tagged the new image into my hub.docker.com repository (tag: f25-ca) and pushed it. And with that, we are ready to deploy Keycloak on OpenShift.

Creating the Keycloak application in OpenShift

At this point we have a local OpenShift cluster (via Minishift) and a Keycloak image (frasertweedale/keycloak-openshift:f25-ca) to deploy. When deploying the app we need to set some environment variables:

KEYCLOAK_USER=admin

A username for the Keycloak admin account to be created

KEYCLOAK_PASSWORD=secret123

Passphrase for the admin user

PROXY_ADDRESS_FORWARDING=true

Because the application will be running behind OpenShift’s HTTP proxy, we need to tell Keycloak to use the "external" hostname when creating hyperlinks, rather than Keycloak’s own view.

Use the oc new-app command to create and deploy the application:

% oc new-app --docker-image frasertweedale/keycloak-openshift:f25-ca \
    --env KEYCLOAK_USER=admin \
    --env KEYCLOAK_PASSWORD=secret123 \
    --env PROXY_ADDRESS_FORWARDING=true
--> Found Docker image 45e296f (4 weeks old) from Docker Hub for "frasertweedale/keycloak-openshift:f25-ca"

    * An image stream will be created as "keycloak-openshift:f25-ca" that will track this image
    * This image will be deployed in deployment config "keycloak-openshift"
    * Port 8080/tcp will be load balanced by service "keycloak-openshift"
      * Other containers can access this service through the hostname "keycloak-openshift"

--> Creating resources ...
    imagestream "keycloak-openshift" created
    deploymentconfig "keycloak-openshift" created
    service "keycloak-openshift" created
--> Success
    Run 'oc status' to view your app.

The app gets created immediately but it is not ready yet. The download of the image and deployment of the container (or pod in OpenShift / Kubernetes terminology) will proceed in the background.

After a little while (depending on how long it takes to download the ~300MB Docker image) oc status will show that the deployment is up and running:

% oc status
In project My Project (myproject) on server https://192.168.42.214:8443

svc/keycloak-openshift - 172.30.198.217:8080
  dc/keycloak-openshift deploys istag/keycloak-openshift:f25-ca 
    deployment #2 deployed 3 minutes ago - 1 pod

View details with 'oc describe <resource>/<name>' or list everything with 'oc get all'.

(In my case, the first deployment failed because the 10-minute timeout elapsed before the image download completed; hence deployment #2 in the output above.)

Creating a secure route

Now the Keycloak application is running, but we cannot reach it from outside the Keycloak project itself. In order to be able to reach it there must be a route. The oc create route command lets us create a route that uses TLS (so clients can authenticate the service). We will use the domain name keycloak.ipa.local. The public/private keypair and certificate have already been generated (how to do that is outside the scope of this article). The certificate was signed by the CA we added to the image earlier. The service name – visible in the oc status output above – is svc/keycloak-openshift.

% oc create route edge \
  --service svc/keycloak-openshift \
  --hostname keycloak.ipa.local \
  --key /home/ftweedal/scratch/keycloak.ipa.local.key \
  --cert /home/ftweedal/scratch/keycloak.ipa.local.pem
route "keycloak-openshift" created

Assuming there is a DNS entry pointing keycloak.ipa.local to the OpenShift cluster, and that the system trusts the CA that issued the certificate, we can now visit our Keycloak application:

% curl https://keycloak.ipa.local/
<!--
  ~ Copyright 2016 Red Hat, Inc. and/or its affiliates
  ~ and other contributors as indicated by the @author tags.
  ~
  ~ Licensed under the Apache License, Version 2.0 (the "License");
  ~ you may not use this file except in compliance with the License.
  ~ You may obtain a copy of the License at
  ~
  ~ http://www.apache.org/licenses/LICENSE-2.0
  ~
  ~ Unless required by applicable law or agreed to in writing, software
  ~ distributed under the License is distributed on an "AS IS" BASIS,
  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  ~ See the License for the specific language governing permissions and
  ~ limitations under the License.
  -->
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">

<html>
<head>
    <meta http-equiv="refresh" content="0; url=/auth/" />
    <meta name="robots" content="noindex, nofollow">
    <script type="text/javascript">
        window.location.href = "/auth/"
    </script>
</head>
<body>
    If you are not redirected automatically, follow this <a href='/auth'>link</a>.
</body>
</html>

If you visit in a browser, you will be able to log in using the admin account credentials specified in the KEYCLOAK_USER and KEYCLOAK_PASSWORD environment variables specified when the app was created. And from there you can create and manage authentication realms, but that is beyond the scope of this article.

Conclusion

In this post I discussed how to run Keycloak in OpenShift, from bringing up an OpenShift cluster to building the Docker image and creating the application and route in OpenShift. I recounted that I found OpenShift Online unstable at the time I tried it, and that although oc cluster up did successfully bring up a cluster I had trouble getting the Docker and VM networks to talk to each other. Eventually I tried Minishift which worked well.

We saw that although there is no official Docker image for Keycloak in OpenShift, there is a Dockerfile that builds a working image. It is easy to further extend the image to add trust for private CAs.

Creating the Keycloak app in OpenShift, and adding the routes, is straightforward. There are a few important environment variables that must be set. The oc create route command was used to create a secure route to access the application from the outside.

We did not discuss how to set up Keycloak with a database for persisting configuration and user records. The deployment we created is ephemeral. This satisfied my needs for demonstration purposes but production deployments will require persistence. There are official JBoss Docker images that extend the base Keycloak image and add support for PostgreSQL, MySQL and MongoDB. I have not tried these but I’d suggest starting with one of these images if you are looking to do a production deployment. Keep in mind that these images may not include the changes that are required for deploying in OpenShift.

Installing FreeIPA with an Active Directory subordinate CA

FreeIPA is often installed in enterprise environments for managing Unix and Linux hosts and services. Most commonly, enterprises use Microsoft Active Directory for managing users, Windows workstations and Windows servers. Often, Active Directory is deployed with Active Directory Certificate Services (AD CS) which provides a CA and certificate management capabilities. Likewise, FreeIPA includes the Dogtag CA, and when deploying FreeIPA in an enterprise using AD CS, it is often desired to make the FreeIPA CA a subordinate CA of the AD CS CA.

In this blog post I’ll explain what is required to issue an AD sub-CA, and how to do it with FreeIPA, including a step-by-step guide to configuring AD CS.

AD CS certificate template overview

AD CS has a concept of certificate templates, which define the characteristics an issued certificate shall have. The same concept exists in Dogtag and FreeIPA except that in those projects we call them certificate profiles, and the mechanism to select which template/profile to use when issuing a certificate is different.

In AD CS, the template to use is indicated by an X.509 extension in the certificate signing request (CSR). The template specifier can be one of two extensions. The first, older extension has OID 1.3.6.1.4.1.311.20.2 and allows you to specify a template by name:

CertificateTemplateName ::= SEQUENCE {
   Name            BMPString
}

(Note that some documents specify UTF8String instead of BMPString. BMPString works and is used in practice. I am not actually sure if UTF8String even works.)

The second, Version 2 template specifier extension has OID 1.3.6.1.4.1.311.21.7 and allows you to specify a template by OID and version:

CertificateTemplate ::= SEQUENCE {
    templateID              EncodedObjectID,
    templateMajorVersion    TemplateVersion,
    templateMinorVersion    TemplateVersion OPTIONAL
}

TemplateVersion ::= INTEGER (0..4294967295)

Note that some documents also show templateMajorVersion as optional, but it is actually required.

When submitting a CSR for signing, AD CS looks for these extensions in the request, and uses the extension data to select the template to use.

External CA installation in FreeIPA

FreeIPA supports installation with an externally signed CA certificate, via ipa-server-install --external-ca or (for existing CA-less installations ipa-ca-install --external-ca). The installation takes several steps. First, a key is generated and a CSR produced:

$ ipa-ca-install --external-ca

Directory Manager (existing master) password: XXXXXXXX

Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/8]: configuring certificate server instance
The next step is to get /root/ipa.csr signed by your CA and re-run /sbin/ipa-ca-install as:
/sbin/ipa-ca-install --external-cert-file=/path/to/signed_certificate --external-cert-file=/path/to/external_ca_certificate

The installation program exits while the administrator submits the CSR to the external CA. After they receive the signed CA certificate, the administrator resumes the installation, giving the installation program the CA certificate and a chain of one or more certificates up to the root CA:

$ ipa-ca-install --external-cert-file ca.crt --external-cert-file ipa.crt
Directory Manager (existing master) password: XXXXXXXX

Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/29]: configuring certificate server instance
  ...
  [29/29]: configuring certmonger renewal for lightweight CAs
Done configuring certificate server (pki-tomcatd).

Recall, however, that if the external CA is AD CS, a CSR must bear one of the certificate template specifier extensions. There is an additional installation program option to add the template specifier:

$ ipa-ca-install --external-ca --external-ca-type=ms-cs

This adds a name-based template specifier to the CSR, with the name SubCA (this is the name of the default sub-CA template in AD CS).

Specifying an alternative AD CS template

Everything discussed so far is already part of FreeIPA. Until now, there is no way to specify a different template to use with AD CS.

I have been working on a feature that allows an alternative AD CS template to be specified. Both kinds of template specifier extension are supported, via the new --external-ca-profile installation program option:

$ ipa-ca-install --external-ca --external-ca-type=ms-cs \
  --external-ca-profile=1.3.6.1.4.1.311.21.8.8950086.10656446.2706058.12775672.480128.147.7130143.4405632:1

(Note: huge OIDs like the above are commonly used by Active Directory for installation-specific objects.)

To specify a template by name, the --external-ca-profile value should be:

--external-ca-profile=NAME

To specify a template by OID, the OID and major version must be given, and optionally the minor version too:

--external-ca-profile=OID:MAJOR[:MINOR]

Like --external-ca and --external-ca-type, the new --external-ca-profile option is available with both ipa-server-install and ipa-ca-install.

With this feature, it is now possible to specify an alternative or custom certificate template when using AD CS to sign the FreeIPA CA certificate. The feature has not yet been merged but there an open pull request. I have also made a COPR build for anyone interested in testing the feature.

The remainder of this post is a short guide to configuring Active Directory Certificate Services, defining a custom CA profile, and submitting a CSR to issue a certificate.

Renewing the certificate

FreeIPA provides the ipa-cacert-manage renew command for renewing an externally-signed CA certificate. Like installation with an externally-signed CA, this is a two-step procedure. In the first step, the command prompts Certmonger to generate a new CSR for the CA certificate, and saves the CSR so that the administrator can submit it to the external CA.

For renewing a certificate signed by AD CS, as in the installation case a template specifier extension is needed. Therefore the ipa-cacert-manage renew command has also learned the --external-ca-profile option:

# ipa-cacert-manage renew --external-ca-type ms-cs \
  --external-ca-profile MySubCA
Exporting CA certificate signing request, please wait
The next step is to get /var/lib/ipa/ca.csr signed by your CA and re-run ipa-cacert-manage as:
ipa-cacert-manage renew --external-cert-file=/path/to/signed_certificate --external-cert-file=/path/to/external_ca_certificate
The ipa-cacert-manage command was successful

The the above example the CSR that was generated will contain a version 1 template extension, using the name MySubCA. Like the installation commands, the version 2 extension is also supported.

This part of the feature requires some changes to Certmonger as well as FreeIPA. At time of writing these changes haven’t been merged. There is a Certmonger pull request and a Certmonger COPR build if you’d like to test the feature.

Appendix A: installing and configuring AD CS

Assuming an existing installation of Active Directory, AD CS installation and configuration will take 10 to 15 minutes. Open Server Manager, invoke the Add Roles and Features Wizard and select the AD CS Certification Authority role:

image

Proceed, and wait for the installation to complete…

image

After installation has finished, you will see AD CS in the Server Manager sidebar, and upon selecting it you will see a notification that Configuration required for Active Directory Certificate Services.

image

Click More…, and up will come the All Servers Task Details dialog showing that the Post-deployment Configuration action is pending. Click the action to continue:

image

Now comes the AD CS Configuration assistant, which contains several steps. Proceed past the Specify credentials to configure role services step.

In the Select Role Services to configure step, select Certification Authority then continue:

image

In the Specify the setup type of the CA step, choose Enterprise CA then continue:

image

The Specify the type of the CA step lets you choose whether the AD CS CA will be a root CA or chained to an external CA (just like how FreeIPA lets you create root or subordinate CA!) Installing AD CS as a Subordinate CA is outside the scope of this guide. Choose Root CA and continue:

image

The next step lets you Specify the type of the private key. You can use an existing private key or Create a new private key, the continue.

The Specify the cryptographic options step lets you specify the Key length and hash algorithm for the signature. Choose a key length of at least 2048 bits, and the SHA-256 digest:

image

Next, Specify the name of the CA. This sets the Subject Distinguished Name of the CA. Accept defaults and continue.

The next step is to Specify the validity period. CA certificates (especially root CAs) typically need a long validity period. Choose a value like 5 Years, then continue:

image

Accept defauts for the Specify the database locations step.

Finally, you will reach the Confirmation step, which summarises the chosen configuration options. Review the settings then Configure:

image

The configuration will take a few moments, then the Results will be displayed:

image

AD CS is now configured and you can begin issuing certificates.

Appendix B: creating a custom sub-CA certificate template

In this section we look at how to create a new certificate template for sub-CAs by duplicating an existing template, then modifying it.

To manage certificate templates, from Server Manager right-click the server and open the Certification Authority program:

image

In the sidebar tree view, right-click Certificate Templates then select Manage.

image

The Certificate Templates Console will open. The default profile for sub-CAs has the Template Display Name Subordinate Certification Authority. Right-click this template and choose Duplicate Template.

image

The new template is created and the Properties of New Template dialog appears, allowing the administrator to customise the template. You can set a new Template display name, Template name and so on:

image

You can also change various aspects of certificate issuance including which extensions will appear on the issued certificate, and the values of those extensions. In the following screenshot, we see a new Certificate Policies OID being defined for addition to certificates issued via this template:

image

Also under Extensions, you can discover the OID for this template by looking at the Certificate Template Information extension description.

Finally, having defined the new certificate template, we have to activate it for use with the AD CA. Back in the Certification Authority management window, right-click Certificate Templates and select Certificate Template to Issue:

image

This will pop up the Enable Certificate Templates dialog, containing a list of templates available for use with the CA. Select the new template and click OK. The new certificate template is now ready for use.

Appendix C: issuing a certificate

In this section we look at how to use AD CS to issue a certificate. It is assumed that the CSR to be signed exists and Active Directory can access it.

In the Certification Authority window, in the sidebar right-click the CA and select All Tasks >> Submit new request…:

image

This will bring up a file chooser dialog. Find the CSR and Open it:

image

Assuming all went well (including the CSR indicating a known certificate template), the certificate is immediately issued and the Save Certificate dialog appear, asking where to save the issued certificate.