Renewable Security: Steps to Save The Cyber Security Planet

Actually, this has nothing to-do with being green.  Although, that is a passion of mine.  This is more to-do with a paradigm that is becoming more popular in security architectures: that of being able to re-spin particular services to a known “safe” state after breach, or even as a preventative measure before a breach or vulnerability has been exploited.

Triple R's of Security


This falls into what is known as the “3 R’s of Security”.  A quick Google on that topic will result in a fair few decent explanations of what that can mean.  The TL;DR is basically, rotate (credentials), repair (vulnerabilities) and repave (services and servers to a known good state).  This approach is gaining popularity mainly due devops deployment models.  Or “secdevops”.  Or is it “devsecops”?  Containerization and highly automated “code to prod” pipelines make it a lot easier to get stuff into production, iterate and go again.  So how does security play into this?

Left-Shifting 


Well I want to back track a little, and tackle the age old issue of why security is generally applied as a post live issue.  Security practitioners, often evangelise on the “left shifting” of security.  Getting security higher up the production line, earlier in the software design life cycle and less as an audit/afterthought/pen testing exercise.  Why isn’t this really happening?  Well anecdotally, just look at the audit, pen testing and testing contractor rates.  They’re high and growing.  Sure, lots of dev teams and organisations are incorporating security architecture practices earlier in the dev cycle, but many find this too slow, expensive or inhibitive.  Many simply ship insecure software and assume external auditors will find the issues.

This I would say has resulted in variations of R3.  Dev as normal and simply flatten and rebuild in production in order to either prevent vulnerabilities being exploited, or recover from them faster.  Is this the approach many organisations are applying to newer architectures such as micro-services, server-less and IoT?

IoT, Microservices and Server-less


There are not many mature design patterns or vendors for things like micro-services security or even IoT security.  Yes, there are some interesting ideas, but the likes of Forrester, Gartner and other industry analysts, don’t to my knowledge, describe security for these areas as a known market size, or a level of repeatable maturity.  So what are the options?  These architectures ship with out security? Well, being a security guy, I would hope not.  So, what is the next best approach?  Maybe the triple R model is the next best thing.  Assume you’re going to breached – which CISO’s should be doing anyway – and focus on a remediation plan.

The triple R approach does assume a few things though.  The main one, is that you have a known-safe place.  Whether that is focused on images, virtual machines or new credentials, there needs to be a position which you can rollback or forward to, that is believed to be more secure than the version before.  That safe place, also needs to evolve.  There is no point in that safe place being unable to deliver the services needed to keep end users happy.

Options, Options, Options...


The main benefit of the triple R approach, is you have options – either as a response to a breach or vulnerability exposure, or as a preventative shortcut. It can bring other more pragmatic issues however.  If we’re referring to things like IoT security – how can devices, in the field and potentially aware from Internet connectivity – be hooked, rebuilt and re-keyed?  Can this be done in a hot-swappable model too, without interruptions to service?  If you need to rebuild a smart meter, you can’t possibly interrupt electricity supply to the property whilst that completes.

So the R3 model is certainly a powerful tool in the security architecture kit bag.  Is is suitable for all scenarios?  Probably not.  Is it a good “get out of jail” card in environments with highly optimized devops-esque process?  Absolutely.

Renewable Security: Steps to Save The Cyber Security Planet

Actually, this has nothing to-do with being green.  Although, that is a passion of mine.  This is more to-do with a paradigm that is becoming more popular in security architectures: that of being able to re-spin particular services to a known “safe” state after breach, or even as a preventative measure before a breach or vulnerability has been exploited.

Triple R's of Security


This falls into what is known as the “3 R’s of Security”.  A quick Google on that topic will result in a fair few decent explanations of what that can mean.  The TL;DR is basically, rotate (credentials), repair (vulnerabilities) and repave (services and servers to a known good state).  This approach is gaining popularity mainly due devops deployment models.  Or “secdevops”.  Or is it “devsecops”?  Containerization and highly automated “code to prod” pipelines make it a lot easier to get stuff into production, iterate and go again.  So how does security play into this?

Left-Shifting 


Well I want to back track a little, and tackle the age old issue of why security is generally applied as a post live issue.  Security practitioners, often evangelise on the “left shifting” of security.  Getting security higher up the production line, earlier in the software design life cycle and less as an audit/afterthought/pen testing exercise.  Why isn’t this really happening?  Well anecdotally, just look at the audit, pen testing and testing contractor rates.  They’re high and growing.  Sure, lots of dev teams and organisations are incorporating security architecture practices earlier in the dev cycle, but many find this too slow, expensive or inhibitive.  Many simply ship insecure software and assume external auditors will find the issues.

This I would say has resulted in variations of R3.  Dev as normal and simply flatten and rebuild in production in order to either prevent vulnerabilities being exploited, or recover from them faster.  Is this the approach many organisations are applying to newer architectures such as micro-services, server-less and IoT?

IoT, Microservices and Server-less


There are not many mature design patterns or vendors for things like micro-services security or even IoT security.  Yes, there are some interesting ideas, but the likes of Forrester, Gartner and other industry analysts, don’t to my knowledge, describe security for these areas as a known market size, or a level of repeatable maturity.  So what are the options?  These architectures ship with out security? Well, being a security guy, I would hope not.  So, what is the next best approach?  Maybe the triple R model is the next best thing.  Assume you’re going to breached – which CISO’s should be doing anyway – and focus on a remediation plan.

The triple R approach does assume a few things though.  The main one, is that you have a known-safe place.  Whether that is focused on images, virtual machines or new credentials, there needs to be a position which you can rollback or forward to, that is believed to be more secure than the version before.  That safe place, also needs to evolve.  There is no point in that safe place being unable to deliver the services needed to keep end users happy.

Options, Options, Options...


The main benefit of the triple R approach, is you have options – either as a response to a breach or vulnerability exposure, or as a preventative shortcut. It can bring other more pragmatic issues however.  If we’re referring to things like IoT security – how can devices, in the field and potentially aware from Internet connectivity – be hooked, rebuilt and re-keyed?  Can this be done in a hot-swappable model too, without interruptions to service?  If you need to rebuild a smart meter, you can’t possibly interrupt electricity supply to the property whilst that completes.

So the R3 model is certainly a powerful tool in the security architecture kit bag.  Is is suitable for all scenarios?  Probably not.  Is it a good “get out of jail” card in environments with highly optimized devops-esque process?  Absolutely.

Runtime configuration for ForgeRock Identity Microservices using Consul

The ForgeRock Identity Microservices are able to read all runtime configuration from environment variables. These could be statically provided as hard coded key-value pairs in a Kubernetes manifest or shell script, but a better solution is to manage the configuration in a centralized configuration store such as Hashicorp Consul.

Managing configuration outside the code as such has been routinely adopted as a practice by modern service providers such as Google, Azure and AWS and business that use both cloud as well as on-premise software.

In this post, I shall walk you through how Consul can be used to set configuration values, envconsul can be used to read these key-values from different namespaces and supply them as environment variables to a subprocess. Envconsul can even be used to trigger automatic restarts on configuration changes without any modifications needed to the ForgeRock Identity Microservices.

Perhaps even more significantly, this post shows how it is possible to use Docker to layer on top of a base microservice docker image from the Forgerock Bintray repo and then use a pre-built go binary for envconsul to spawn the microservice as a subprocess.

This github.com repo has all the docker and kubernetes artifacts needed to run this integration. I used a single cluster minikube setup.

Consul

Starting a Pod in Minikube

We must start with setting up consul as a docker container and this is just a simple docker pull consul followed by a kubectl create -f kube-consul.yaml that sets up a pod running consul in our minikube environment. Consul creates a volume called /consul/data on to which you can store configuration key-value pairs in JSON format that could be directly imported using consul kv import @file.json. consul kv export > file.json can be used to export all stored configuration separated into JSON blobs by namespace.

The consul commands can be run from inside the Consul container by starting  a shell in it using docker exec -ti <consul-container-name> /bin/sh

Configuration setup using Key Value pairs

The namespaces for each microservice are defined either manually in the UI or via REST calls such as:
consul kv put my-app/my-key may-value

 

Expanding the ms-authn namespace yields the following screen:

A sample configuration set for all three microservices is available in the github.com repo for this integration here. Use consul kv import @consul-export.json to import them into your consul server.

Docker Builds

Base Builds

First, I built base images for each of the ForgeRock Identity microservices without using an ENTRYPOINT instruction. These images are built using the openjdk:8-jre-alpine base image which are the smallest alpine-based openjdk image one can possibly find. The openjdk7 and openjdk9 images are larger. Alpine itself is a great choice for a minimalistic docker base image and provides packaging capabilities using apk. It is a great choice for a base image especially if these are to be layered on top of like in this demonstration.

An example for the Authentication microservice is:
FROM openjdk:8-jre-alpine
WORKDIR /opt/authn-microservice
ADD . /opt/authn-microservice
EXPOSE 8080

The command docker build -t authn:BASE-ONLY builds the base image but does not start the authentication listener since no CMD or ENTRYPOINT instruction was specified in the Dockerfile.

The graphic below shows the layers the microservices docker build added on top of the openjdk:8-jre-alpine base image:

Next, we need to define a launcher script that uses a pre-built envconsul go binary to setup communication with the Consul server and use the key-value namespace for the authentication microservice.

/usr/bin/envconsul -log-level debug -consul=192.168.99.100:31100 -pristine -prefix=ms-authn /bin/sh -c "$SERVICE_HOME"/bin/docker-entrypoint.sh

A lot just happened there!

We instructed the launcher script to start envconsul with an address for the Consul server and nodePort defined in the kube-consul.yaml previously. The switch -pristine tells envconsul to not merge extant environment variables with the ones read from the Consul server. The -prefix switch identifies the application namespace we want to retrieve configuration for and the shell spawns a new process for the docker-entrypoint.sh script which is an ENTRYPOINT instruction for the authentication microservice tagged as 1.0.0-SNAPSHOT. Note that this is not used in our demonstration, but instead we built a BASE-ONLY tagged image and layer the envconsul binary and launcher script (shown next) on top of it. The -c switch enables tracking of changes to configuration values in Consul and if detected the subprocess is restarted automatically!

Envconsul Build

With the envconsul.sh script ready as shown above, it is time to build a new image using the base-only tagged microservice image in the FROM instruction as shown below:

FROM forgerock-docker-public.bintray.io/microservice/authn:BASE-ONLY
MAINTAINER Javed Shah <javed.shah@forgerock.com>
COPY envconsul /usr/bin/
ADD envconsul.sh /usr/bin/envconsul.sh
RUN chmod +x /usr/bin/envconsul.sh
ENTRYPOINT ["/usr/bin/envconsul.sh"]

This Dockerfile includes the COPY instruction to add our pre-built envconsul go-binary that can run standalone to our new image and also adds the envconsul.sh script shown above to the /usr/bin/ directory. It also sets the +x permission on the script and makes it the ENTRYPOINT for the container.

At this time we used the docker build -t authn:ENVCONSUL . command to tag the new image correctly. This is available at the ForgeRock Bintray docker repo as well.

This graphic shows the layers we just added using our Dockerfile instruction set above to the new ENVCONSUL tagged image:

These two images are available for use as needed by the github.com repo.

Kubernetes Manifest

It is a matter of pulling down the correctly tagged image and specifying one key item, the hostAliases instruction to point to the OpenAM server running on the host. This server could be anywhere, just update the hostAliases accordingly. Manifests for the three microservices are available in the github.com repo.

Start the pod with kubectl create -f kube-authn-envconsul.yaml will cause the following things to happen, as seen in the logs using kubectl logs ms-authn-env.

This shows that envonsul started up with the address of the Consul server. We deliberately used the older -consul flag for more “haptic” feedback from the pod if you will. Now that envconsul started up, it will immediately reach out to the Consul server unless a Consul agent was provided in the startup configuration. It will fetch the key-value pairs for the namespace ms-authn and make them available as environment variables to the child process!

The configuration has been received from the server and at this time envconsul is ready to spawn the child process with these key-value pairs available as environment variables:

The Authentication microservice is able to startup and read its configuration from the environment provided to it by envconsul.

Change Detection

Lets change a value in the Consul UI for one of our keys, CLIENT_CREDENTIALS_STORE from json to ldap:

This change is immediately detected by envconsul in all running pods:

envconsul then wastes no time in restarting the subprocess microservice:

Pretty cool way to maintain a 12-factor app!

Tests

A quick test for the OAuth2 client credential grant reveals our endeavor has been successful. We receive a bearer access token from the newly started authn-microservice.

The token validation and token exchange microservices were also spun up using the procedures defined above.

Here is an example of validating a stateful access_token generated using the Authorization Code grant flow using OpenAM:

And finally the postman call to the token-exchange service for exchanging a user’s id_token with an access token granting delegation for an “actor” whose id_token is also supplied as input is shown here:

The resulting token is:

This particular use case is described in its own blog.

Whats coming next?

Watch this space for secrets integration with Vault and envconsul!

Identity Disorder Podcast, Episode 1

I’m excited to introduce a new podcast series hosted by Daniel Raskin and myself. The series will focus on (what we hope are!) interesting identity topics, news about ForgeRock, events, and much more. Take a listen to the debut episode below where we discuss why and how to get rid of passwords, how stateless OAuth2 tokens work, and some current events, too!

-Chris