IDM: Zero Downtime Upgrade Strategy Using a Blue/Green Deployment

Introduction

This article is a continuation of the previous article on a Zero Downtime Upgrade Strategy Using a Blue/Green Deployment for AM. Traditionally, ForgeRock Identity Management (IDM) upgrades are handled either in-place or by leveraging the migration service. As many deployments have constraints around this approach (zero downtime, immutable, etc.), a parallel deployment approach, or a blue/green strategy can be leveraged for upgrading ForgeRock IDM servers.

This article provides a high-level approach for using a blue/green methodology for updating ForgeRock IDM servers.

This corresponds to a Unit 4: IDM Upgrade in our overall ForgeRock approach to upgrading.

Unit 4: IDM Upgrade
Blue/Green Deployment

Determining the correct approach

IDM 6.5 is bundled with the migration service, which lets you migrate data from a blue environment to a green environment, table by table, and row by row. This leads to identical data on both sides. The constraint is that update traffic must be cut off in the blue environment to ensure consistent data in the green environment after the migration completes. It is not a zero downtime migration.

The method described here shows you how to perform a zero downtime migration.

This approach in this article should be seen as a foundation on which to build your migration strategy. As IDM dependencies are very elaborate and varied, it is very difficult to design a perfect strategy that will work with all kinds of deployments.

The table below shows the major differences between the migration service method and the scripted CREST method:

Before using the methodology described in this article, please confirm that you really need zero downtime. It may be that the amount of data is too big to fall within an overnight window, which prevents you from using the migration service. The solution to this may be overcome with the following measures:

  1. Verify that the blue system nodes are performance-optimized to ensure the lowest possible response time.
  2. Allocate more nodes in the green environment, and distribute the migration between the nodes by partitioning the mappings.

If neither of the above options are possible, then please read on.

Prerequisites/Assumptions

1. This approach assumes that your infrastructure processes have the ability to install a parallel deployment for upgrade, or you are already using blue/green deployment.

2. In above diagram, a blue cluster represents an existing IDM deployment (like 4.5.x version) and a green one represents a new IDM deployment (like 6.5.x version).

3. Review the Release Notes for all IDM versions between existing and target IDM deployment for new, deprecated features, bug fixes, etc. For an IDM 4.5 to IDM 6.5 upgrade, review the Release Notes for IDM 5.0, 5.5, 6.0, and 6.5 versions.

4. Verify that the deployment upgrade fits with a blue/green deployment. It requires copying repository data over. When data transformation is not required, such as upgrading from 6.0 to 6.5, sharing the repository, which is not a pure blue/green deployment, might be the best fit.

5. When external systems are the source of truth for data, the IDM repository can be rebuilt through a full reconciliation. Therefore, the methodology exposed in this paper is not relevant for this case.

Upgrade the Process Using the CREST Scripted Connector

1. Prepare the migration:

  • Clone the crestidmmigration project from Bitbucket, and read the project description so that you can decide whether this method fits the requirements. While implicit sync always occurs in the blue environment, decide which environment you will use to initiate the reconciliation.
  • Decide which strategy to employ for external resources (such as a ForgeRock Directory Server). As described in the project, you can migrate the repository links table using the migration service, as the CREST migration preserves the managed object IDs. Alternatively, you may perform a full reconciliation after the migration. In all cases, turn off implicit sync for all mappings in the green environment to avoid unnecessary (and perhaps, conflicting) traffic to the external provisioning systems.
  • Follow the instructions in the Installation Guide to prepare for a migration, except that you are not going to perform a migration.
  • Provide all property mappings in sync.json. For encrypted values such as “password”, step 2 in the Installation Guide ensures that both environments share the same encryption keys. Provide all the necessary configuration information to optimize the reconciliation (in particular, paged reconciliation). However, do not alter the correlationQuery.
  • Edit the onCreate and onUpdate scripts to add processing for all relationships. These scripts ensure that relationship properties are propagated to the target, but also filter out any relationships to resources that are not yet provisioned. This prevents duplicate entries.
  • Edit provisioner.openicf-scriptedcrest.json to include all properties that will be migrated. Be careful not to change the configuration for the _id property, and examine how relationship properties are configured (See “devices” for multivalued, and “owner” for single value).
  • Perform a blue/green upgrade to deploy the CREST scripted connector for the 4.x or 5.x nodes. Ensure that implicit sync is disabled for all mappings involving the CREST connectors. The resulting green environment becomes the blue environment in the next migration phase.
  • Deploy the green environment (for example, 6.5) prepared for migration (including the CREST connector, if reconciliation will be launched from the green nodes).

2. Turn on implicit sync in blue, and launch the reconciliation of each scripted CREST mapping, one after the other completes (so they are not running concurrently).

3. You may use the migration service to migrate other data, such as internal roles, internal users, and links.

4. Switch over to the new deployment:

  • After validating that the new deployment is working correctly, switch the load balancer from blue to green, and turn off implicit sync to external provisioning systems in blue. Then, turn on implicit sync to external provisioning systems in green, and perform a reconciliation with the external provisioning systems, or use a custom method if provisioning is not performed through reconciliation.
  • If there are any issues, you can always roll back to blue deployment.

Note: Any managed object or configuration change made after switchover should be applied to both blue and green deployments so that no change is lost during rollback.

Post Go-Live

  1. Stop the migration service on the green deployment.
  2. Stop the blue IDM servers.
  3. De-provision the blue deployment.

Conclusion

Although a blue/green deployment requires a high level of deployment maturity, this approach provides an elegant way to minimize downtime for ForgeRock deployment upgrades. It is always advisable to practice an upgrade strategy in lower environments like dev, stage before moving to the production environment.

Depending on the complexity of your deployment, multiple things may need to be considered for these upgrades, such as customizations, new FR features, etc. It is always recommended to break the entire upgrade process into multiple releases like “base upgrade”, followed by “leveraging new features” and so on.

When deploying IDM for the first time, it is always advisable to incorporate the upgrade strategy early on in the project, so that any designed feature allows for seamless migration in the future. Also, syncing to the green environment will impact the blue update performance, as implicit syncs are all executed synchronously on the request’s thread (on 4.x, 5.x, and 6.0). Fortunately, this does not apply any more to 6.5 when queued sync is enabled. The impact will be sensible with a high relationship cardinality, as the process requires interrogating the target system for each relationship before propagating it. This is why planning the upgrade strategy well in advance is important.

The provided CREST scripts and configuration are a starting point, as well as a proof of concept, which you can use as the basis to build your own upgrade based on your deployment requirements. The details are described in the crestidmmigration project. Note that a second alternative solution is proposed there; one that preserves relationships and lets you run the migration service from the green environment (CREST+MIGRATION folder). However, this may contribute to a higher performance hit on the blue environment. Please note that crestidmmigration is an evolving project, and as such, some other variants could be proposed in the future, so stay tuned!

References

DS: Zero Downtime Upgrade Strategy Using a Blue/Green Deployment

Introduction

This is the continuation of the previous blog about a Zero Downtime Upgrade Strategy Using a Blue/Green Deployment for AM. Traditionally, ForgeRock Directory Server (DS) upgrades are handled via a rolling upgrade strategy using an in-place update. As many deployments have constraints around this approach (zero downtime, immutable, etc.), a parallel deployment approach, also known as a blue/green strategy, can be leveraged for upgrading ForgeRock DS servers.

This blog provides a high-level approach for using a blue/green methodology for updating ForgeRock DS-UserStores.

This corresponds to Unit 3: DS-UserStores in our overall ForgeRock upgrade approach.

ForgeRock Upgrade Units
Unit 3: DS-Userstores Upgrade Process

Prerequisites/Assumptions

1. This approach assumes that your infrastructure processes have the ability to install a parallel deployment for an upgrade, or you are already using a blue/green deployment.

2. In the above diagram, the blue cluster reflects an existing DS deployment (like a 3.5.x version), and the green reflects a new DS deployment (like a 6.5.x version).

3. There are N+1 DS servers deployed in your existing deployment. N servers are used for your production workload and one server is reserved for maintenance activities like backup, upgrades, etc. If there is no maintenance server, then you may need to remove one server from the production cluster (thereby reducing production load capacity) or install an additional DS server node for this upgrade strategy.

4. Review release notes for all DS versions between existing and target DS deployments for new, deprecated features, bug fixes, and others. For a DS 3.5 to DS 6.5 upgrade, review the Release Notes for DS 5.0, 5.5, 6.0, and 6.5 versions.

Upgrade Process

1. Unconfigure replication for the DS-3 user store. Doing so ensures that the upgrade doesn’t impact your existing DS deployment.

2. Upgrade DS-3 in place using DS upgrade process.

3. Create a backup from DS-3 using the DS backup utility.

4. Configure green RS-1’s replication with the existing blue replication topology.

5. Configure green RS-2’s replication with the existing blue replication topology.

6. Install green DS-1 and restore data from backup using the DS restore utility.

7. Install green DS-2 and restore data from backup using the DS restore utility.

8. Install Green DS-3 and restore data from backup using the DS restore utility.

9. Configure Green DS-1’s replication with Green RS-1.

10. Configure Green DS-2’s replication with Green RS-1.

11. Configure Green DS-3’s replication with Green RS-1.

Switch Over to the New Deployment

12. After validating that the new deployment is working correctly, switch the load balancer from blue to green. This can also be done incrementally. If any issues occur, you can always roll back to the blue deployment.

If direct hostnames are used by DS clients, such as AM, IDM, etc., then those configurations need to be updated to leverage new green hostnames.

Post Go-Live

13. Unconfigure the blue RS1 replication server to remove this server from blue’s replication topology.

14. Unconfigure the blue RS2 replication server to remove this server from blue’s replication topology.

15. Stop the blue DS servers.

16. Stop the blue RS servers.

17. De-provision the blue deployment.

Conclusion

Although a blue/green deployment requires a high level of deployment maturity, this approach provides an elegant way to minimizing downtime for ForgeRock deployment upgrades. It is always advisable to try an upgrade strategy in lower environments like dev, stage before moving to a production environment.

Depending on the complexity of your deployment, there can be multiple things to be considered for these upgrades, such as customizations, new FR features, etc. It is always recommended to break the entire upgrade process into multiple releases like “base upgrade” followed by “leveraging new features”, and so on.

References

AM and IG: Zero Downtime Upgrade Strategy Using a Blue/Green Deployment

Introduction

The standard deployment for the ForgeRock Identity Platform consists of multiple ForgeRock products such as IG, AM, IDM, and DS. As newer ForgeRock versions are released, deployments using older versions need to be migrated before they reach their end of life. Also, newer versions of ForgeRock products provide features such as intelligent authentication and the latest OAuth standards, which help businesses implement complex use cases.

ForgeRock Deployment Components

Problem Statement

Traditionally, ForgeRock upgrades are handled via a rolling upgrade strategy using an in-place update. This strategy doesn’t suit all deployments due to the following constraints:

  • Many deployments don’t allow any downtime. This means production servers can’t be stopped for upgrade purposes.
  • Some deployments follow an immutable instances approach. This means no modification is allowed on the current running servers.

To resolve these constraints, a parallel deployment approach, or a blue/green strategy can be leveraged for upgrading ForgeRock servers.

Solution

This article provides a high-level approach for using a blue/green methodology for updating ForgeRock AM servers and related components like DS-ConfigStore, DS-CTS, AM-Agents, and IG servers. We plan to cover similar strategies for DS-UserStores and IDM in future articles.

In order to upgrade ForgeRock deployment, we need to first analyze the dependencies between various ForgeRock products and their impact on upgrade process:

Given the dependencies between ForgeRock products, it is generally advisable to upgrade AM before upgrading DS, AM agents, and others, as new versions of AM support older versions of DS and AM agents, but the converse may not be true.

Note: There can be some exceptions to this rule. For example:

  • Web policy agents 4.x are compatible with AM 6.0, but not with AM 6.5. This means the order of upgrade shall be existing version to AM 6.0 => AM Agent 4.x to 5.x => AM 6.0 to AM 6.5.x
  • If an AM-IDM integration is used, then both AM and IDM need to be upgraded at the same time.

Upgrade Units

ForgeRock Upgrade Units

A ForgeRock Identity Platform deployment can be divided into 4 units so that upgrade of these units can be handled individually:

  • Unit 1: AM and its related stores (DS-Config and DS-CTS)
  • Unit 2: AM-Agents/IG
  • Unit 3: DS-UserStores
  • Unit 4: IDM and its datastore

The order of upgrade used by our approach shall be Unit 1=>Unit 2=>Unit 3=>Unit 4.

Unit 1: AM Upgrade

Unit 1: AM Upgrade

Prerequisites/ Assumptions

1. This approach assumes that your infrastructure processes have the ability to install a parallel deployment for upgrade, or you are already using a blue/green deployment.

2. In the above diagram, the blue cluster reflects an existing AM deployment (like the 13.5.x version) and the green cluster reflects a new AM deployment (like the 6.5.x version).

3. There are N+1 AM servers and corresponding config stores deployed in your existing deployment. This means N servers are used for production load, and one server is reserved for maintenance activities like backup, upgrades, and others. If there is no such maintenance server, then you may need to remove one server from the production cluster (thereby reducing production load capacity) or install an additional node (AM server and corresponding config store) for this upgrade.

4. No sessions in CTS servers are replicated during blue/green switch; therefore, users are expected to re-authenticate after this migration. If your business use cases require users to remain authenticated, then these sessions (like OAuth Refresh tokens) need to be synced from the old to the new deployment. Mechanisms like ldif export/import or using IDM synchronization engine can be leveraged for syncing selective tokens from old to new deployments. Also, refer to the AM Release Notes on session compatibility across AM versions.

5. Review the Release Notes for all AM versions between existing and target AM deployment for new features, deprecated features, bug fixes, and so on for OpenAM 13.5 to AM 6.5 upgrade. Review the Release Notes for AM 5.0, 5.5, 6.0, and 6.5 versions.

Upgrade Process

1. Unconfigure replication for the DS-3 Config store. This ensures that the upgrade doesn’t impact an existing AM deployment.

2. Upgrade AM-3 in-place using the AM upgrade process. Note: You may need to handle new AM features in this process like AM 6.5 secrets, and others.

3. Export Amster configs from AM-3.

4. Transform Amster export so that the Amster export is aligned with a new green deployment such as DS hostname:port.

5. Install AM, DS-Config, and DS-CTS servers. Import the Amster export into a new green cluster. Note: For certain deployment patterns, such as ForgeRock immutable deployment, the Amster import needs to be executed for each AM node. If a shared config store is used, then the Amster import needs to be executed only once, and other nodes are required to be added to the existing AM site.

Switch Over to the New Deployment

6. After validating that the new deployment is working correctly, switch the load balancer from blue to green. This can also be done incrementally. If any issues occur, we can always roll back to the blue deployment.

Note: Any configuration changes made after the blue’s cluster’s Amster export should be applied to both blue and green deployments so that no configuration change is lost during switchover or rollback.

Post Go-Live

7. Stop the AM servers in the blue deployment.

8. Stop the Config and CTS DS servers in blue deployment.

9. De-provision the blue deployment.

Unit 2: AM-Agent/IG Upgrade

Unit 2: AM-Agent/IG Upgrade Process

AM-Agent

Prerequisites/ Assumptions

1. This approach assumes that your deployment (including applications protected by agents) has the ability to install a parallel deployment for upgrade, or you are already using a blue/green deployment.

2. In the above diagram, the blue cluster reflects an existing AM-Agent deployment and the green reflects new AM-Agent deployment.

3. A parallel base green deployment for protected app servers has already been created.

4. Create new Agent profiles for green deployment on AM servers.

5. This approach assumes both old and new AM-Agent versions are supported by the AM deployment version.

6. Refer to the Release Notes for latest and deprecated features in the new AM-Agent/IG version, such as the AM-Agent 5.6 Release Notes.

Upgrade Process

1. Install AM-Agents in the green deployment. Update agent profiles on the AM server (created in #4 above) for new agents deployed in the green deployment to match configurations used in agent profiles from the blue deployment. For certain AM versions, this process can be automated by retrieving existing Agent profiles and using these results to create new Agent profiles.

Switch Over to the New Deployment

2. After validating that the new deployment is working properly, switch the load balancer from blue to green.

Post Go-Live

3. Stop the app servers in the blue deployment.

4. Remove the blue agent profiles from AM deployment.

5. De-provision the blue deployment.

IG

Prerequisites/ Assumptions

1. This approach assumes that your deployment (including applications protected by agents) has the ability to install a parallel deployment for upgrade, or you are already using a blue/green deployment.

2. In the above diagram, the blue cluster reflects an existing IG deployment and the green reflects the new IG deployment.

3. This approach assumes both old and new IG versions are supported by the AM deployment version.

4. Create new Agent profiles for the green deployment on the AM servers required for IG servers.

5. Refer to the Release Notes for the latest and deprecated features in the new IG version, like IG 6.5 Release Notes.

Upgrade Process

1. Update the IG configs in the git repository as per the changes in the new version. You may create a different branch in your repository for the same.

2. Deploy the new green IG deployment by leveraging updated configurations.

Switch Over to the New Deployment

3. After validating that the new deployment is working fine, switch the load balancer from blue to green.

Post Go-Live

4. Stop the IG servers in the blue deployment.

5. De-provision the blue deployment.

Conclusion

Although a blue/green deployment requires a high level of deployment maturity, this approach provides an elegant way to minimize downtime for ForgeRock deployment upgrades. It is always advisable to practice an upgrade strategy in lower environments like dev, and stage before moving to a production environment.

Depending on the complexity of your deployment, there can be multiple things to be considered for these upgrade,s such as customizations, new FR features, migration to containers, and others. It is always recommended to break the entire upgrade process into multiple releases, like “base upgrade” followed by “leveraging new features”, and so on.

References

5 Indicators of Cyber Security Market Failure

6 Minute Read. By Simon Moffatt.


Let us start with some brief definitions to get us all on the same page. Firstly – what is meant by the term “market failure”? A textbook description would be something that articulated the “inefficient distribution of goods and services in a free market”. But how do we decide whether the distribution is inefficient or not? Perhaps, let us look at how "efficient" is described first, then work backwards.  An efficient market would probably display a scenario where goods and services are distributed, priced and made, in a manner which can not be improved upon, with the amount of waste minimised.

This requires analysing two distinct parties – the consumer of the good and the maker of the good. The consumer wants to purchase at the lowest price, that maximises their “utility” or satisfaction. The maker on the other hand, wants to maximise profits whilst simultaneously minimising costs.

If we start looking at the "good", as the manufacturing and procurement of cyber security software, services and consulting, are we confident we are operating at maximum efficiency? I would argue we are not.  I am going to pick five high level topics in which to dig a little deeper.

1) Labour Shortages

The 2019 ISC2 Cyber Workforce Study, identified a staggering 4.07 million unfilled cyber security positions – up from 2.93 million in 2018. The report highlighted this as a global problem too – with APAC sitting on a 2.6 million backlog of unfilled roles. There are probably numerous other reports and Google search nuggets, to back up the claim, that cyber security is one of the toughest skill sets to recruit for within technology in 2020.

But what does this prove? Mismatches in labour demand and supply are common in numerous professions – medical doctors being an obvious one. An excess in demand over supply can obviously create wage inflation, amongst other inefficiencies, but what about triggers from the supply side?

The classic causes of labour market imperfection are many – but some seem to easily apply to cyber. The inelastic supply of suitable candidates is a good starting place.


In-elasticity of the supply of cyber security candidates

In this basic example, the supply of cyber candidates is described as being highly inelastic – for example a change in salary, does not result in a proportional change in the supply of candidates. Why is this? Clearly training has a part to play. Skilled cyber practitioners are likely to require strong computer science, network and infrastructure skills, before being able to embark on more specialised training. This can take many years to obtain, effectively acting like barriers to entry for new and willing candidates.

As with many labour markets, immobility and lack of vacancy information may also hinder skills investment, especially if the candidate is not aware of the potential benefits the long term training can bring. The more common practice of remote working however, is certainly helping to reduce geographical immobility issues which often hamper more traditional industries.

The cyber security industry is still very much in its infancy too, which can contribute to a lack of repeatable candidate development. Only in 2019, did the UK’s Chartered Institute of Information Security receive its royal warrant. Compare that to the likes of the Soap Makers Company (1638), Needlemakers Company (1656), Coachmakers Company (1677), Fanmakers (1709) and the Royal Medical Society (1773) and there is a palpable level of professional immaturity to understand. 

This could be amplified by a lack of consistency surrounding certifications, curriculum and job role descriptions. Only within the last 3 months has the industry seen CyBoK – the cyber book of knowledge - published. This may go a little way in attempting to formalise training and certification of candidates globally.

2) Regulation

An interesting bi product of perceived market failure, is government intervention. External intervention can take many forms and is often used to simulate competition (eg the likes of OfCom, OfWat or OfRail in the UK) where monopolistic or quasi-public sector run industries would not necessarily deliver optimum allocative efficiency if left to their own devices.

Whilst the cyber security sector is not a monopolistic supplier or employer, it has seen numerous pieces of governmental regulation. A few basic examples in Europe would include the General Data Protection Regulation (GDPR) and the Network and Information Systems Directive (NIS). In the United States, at a state level at least, the California Consumer Privacy Act (CCPA) came into fruition with further amendments towards the end of 2019.

I am blurring the line between security and privacy with some of those regulations, but an interesting aspect, is the consumer protection angle of the likes of the GDPR and CCPA. If the market where left to its own devices, the consumer of 3rd party on line goods and services, is not deemed to be protected to a satisfactory level. The regulatory intervention is not to rectify a negative externality affecting 3rd parties, more to protect the first party user. During the exchange of goods and services to the individual, it seems the requisite level of privacy and security that benefits the group of users as a whole is not utilitarian. A major aim the current cyber legislation is trying to achieve, is to overcome the information asymmetries that exist when a user signs up for a service or makes an online purchase or interaction.

With respect to consumer privacy, a concept of information asymmetry known as adverse selection may exist - where the buyer is not fully aware of the use and value of their personal data in relation to the supplier of a good or service, who may use their data in ways not fully understood or even disclosed to the user.

The likes of the NIS directive, seems more focused upon reducing the impact of an externality - basically a negative impact to a wider group of users. Perhaps, due to a data breach, service disruption or degradation, that may have occurred due to a lack of cyber security controls. A simple example could be the lack of power generation to an entire town if a nuclear power station is knocked offline due to a malware attack.

3) Product Hyper Augmentation

The cyber security market is broad, complex and ever evolving. The number of product categories grows daily. Gartner has at least 20 security related magic quadrants. CISO's and CIO's have to make incredibly complex decisions regarding product and service procurement.
Certainly, there is another set of information asymmetries at play here, but those are likely to exist in other complex software markets. With respect to cyber, there seems to an accelerated augmentation of features and messaging. When does a next generation web application firewall become a dedicated API security gateway? When does a security orchestration automation and response platform become an integrated event driven identity and access management workflow? Is there space for such niche upon niches, or are we entering a phase of largely merged and consolidated markets, where buyer procurement decision making is simply based on non-features such as brand affiliation and procurement ease?


Product direction via vertical and horizontal augmentation

Many mature markets often reach a position where suppliers augment products to a position of mediocrity and consumer apathy and confusion. Youngme Moon from Harvard Business School articulates this concept extremely well in her book Different - which focuses on competitive strategies. It seems the market for cyber security products especially (maybe less so for services and consultancy) is rapidly moving to a position, where core products are being blurred via augmentation, add on services and proprietary market descriptions. This is creating difficulties when it comes to calculating product purchase versus return on investment reporting.

4) Breach Increase

A continuation of the purchase/RoI analysis pattern, is to analyse what "success" looks like for numerous cyber investments. Whether those investments are people, process or technology related, most procurement decisions, end up being mapped to a success criteria. Value for money. Return on Investment. Call it what you will, but many organisations will look to describe what success looks like when it comes to security investments.

Is it a reduction in data breaches? Is it having fewer installed products with known CVE (common vulnerability & exposures) due to faster patch roll out? Is it having more end users signing up with second factor authentication? This can tie neatly into the controls -v- outcomes discussion where risk, security and privacy management for an organisation needs to identify tangible and SMART (specific measurable assignable realistic time-bound) metrics for implied cyber investment. The ultimate goal of cyber is to support the CIA (confidentiality integrity availability) triad, either singularly or collectively.

A major source of cyber investment success discussion, is associated with data breach trends. There are numerous pieces of data to support the claim, that breaches are increasing. Increasing in volume (from 157 in 2005 to 783 in 2014), breadth and complexity. Many of the articles could admittedly be FUD raised by vendors to accelerate product adoption, but there is no denying the popularity of sites like HaveIBeenPwned, where the number of breached credentials is substantial and increasing. If cyber investment was efficient, shouldn't these metrics be reducing?

This starts to generate two questions: either buyers are buying and using the wrong products or those products are not providing a decent return on investment.

5) Corporate Information Failure

But are products really to blame? The entire thread of this article, is to discuss market failure points. Information is a key component of effective free market development. Many information barriers seem to exist within the cyber security sector. Think of the following:
  • RoI on cyber product investment
  • Cost of personal data protection
  • The societal impact of critical infrastructure failures
  • Risk management success criteria
  • Cyber security certification benefit to corporations
There are likely several other angles to take on this, but full information with regards to the upholding of the confidentiality, availability and integrity of data is unlikely to occur. Many private sector organisations have undergone digital transformation over the last 10 years. These "corporation.next" operations, have created new challenges with respect to data protection. Data relating to customers, employees, intellectual property, products, suppliers, transactions and products.

But how do organisations a) know what to protect b) know how to protect it and c) innovate and manage investment strategies with respect to the protection?

There are many strategies used to manage cyber corporate investment. Some are driven by vendor FUD - aka breach threat - right through to modern risk management strategies, driven by mapping information protection to a higher level corporate strategy. 

If the corporate strategy is known and well communicated, it can become easier to overlay information protection decisions, that the business owners are willing to accept, monitor and iterate against. Risk transparency can help to provide a deeper understanding to what investments should be made and whether those investments are personnel, training or product related.

Summary

Cyber security is a growing, complex and multi faceted market. Many aspects are emerging, with new vendors, design patterns and attack vectors being created monthly. Other aspects, such as risk management and core protection of critical assets are relatively mature and well understood, in comparison to the computational age.

The investment and usage patterns associated with cyber security technology however, are seemingly plagued with numerous information failures, resulting in complex procurement, skills and personnel misalignment.

A value driven approach is needed, where explicit investment decisions (on both the skills provider, procurer and end user side) are weighed against short and long term returns.

5 Indicators of Cyber Security Market Failure

6 Minute Read. By Simon Moffatt.


Let us start with some brief definitions to get us all on the same page. Firstly – what is meant by the term “market failure”? A textbook description would be something that articulated the “inefficient distribution of goods and services in a free market”. But how do we decide whether the distribution is inefficient or not? Perhaps, let us look at how "efficient" is described first, then work backwards.  An efficient market would probably display a scenario where goods and services are distributed, priced and made, in a manner which can not be improved upon, with the amount of waste minimised.

This requires analysing two distinct parties – the consumer of the good and the maker of the good. The consumer wants to purchase at the lowest price, that maximises their “utility” or satisfaction. The maker on the other hand, wants to maximise profits whilst simultaneously minimising costs.

If we start looking at the "good", as the manufacturing and procurement of cyber security software, services and consulting, are we confident we are operating at maximum efficiency? I would argue we are not.  I am going to pick five high level topics in which to dig a little deeper.

1) Labour Shortages

The 2019 ISC2 Cyber Workforce Study, identified a staggering 4.07 million unfilled cyber security positions – up from 2.93 million in 2018. The report highlighted this as a global problem too – with APAC sitting on a 2.6 million backlog of unfilled roles. There are probably numerous other reports and Google search nuggets, to back up the claim, that cyber security is one of the toughest skill sets to recruit for within technology in 2020.

But what does this prove? Mismatches in labour demand and supply are common in numerous professions – medical doctors being an obvious one. An excess in demand over supply can obviously create wage inflation, amongst other inefficiencies, but what about triggers from the supply side?

The classic causes of labour market imperfection are many – but some seem to easily apply to cyber. The inelastic supply of suitable candidates is a good starting place.


In-elasticity of the supply of cyber security candidates

In this basic example, the supply of cyber candidates is described as being highly inelastic – for example a change in salary, does not result in a proportional change in the supply of candidates. Why is this? Clearly training has a part to play. Skilled cyber practitioners are likely to require strong computer science, network and infrastructure skills, before being able to embark on more specialised training. This can take many years to obtain, effectively acting like barriers to entry for new and willing candidates.

As with many labour markets, immobility and lack of vacancy information may also hinder skills investment, especially if the candidate is not aware of the potential benefits the long term training can bring. The more common practice of remote working however, is certainly helping to reduce geographical immobility issues which often hamper more traditional industries.

The cyber security industry is still very much in its infancy too, which can contribute to a lack of repeatable candidate development. Only in 2019, did the UK’s Chartered Institute of Information Security receive its royal warrant. Compare that to the likes of the Soap Makers Company (1638), Needlemakers Company (1656), Coachmakers Company (1677), Fanmakers (1709) and the Royal Medical Society (1773) and there is a palpable level of professional immaturity to understand. 

This could be amplified by a lack of consistency surrounding certifications, curriculum and job role descriptions. Only within the last 3 months has the industry seen CyBoK – the cyber book of knowledge - published. This may go a little way in attempting to formalise training and certification of candidates globally.

2) Regulation

An interesting bi product of perceived market failure, is government intervention. External intervention can take many forms and is often used to simulate competition (eg the likes of OfCom, OfWat or OfRail in the UK) where monopolistic or quasi-public sector run industries would not necessarily deliver optimum allocative efficiency if left to their own devices.

Whilst the cyber security sector is not a monopolistic supplier or employer, it has seen numerous pieces of governmental regulation. A few basic examples in Europe would include the General Data Protection Regulation (GDPR) and the Network and Information Systems Directive (NIS). In the United States, at a state level at least, the California Consumer Privacy Act (CCPA) came into fruition with further amendments towards the end of 2019.

I am blurring the line between security and privacy with some of those regulations, but an interesting aspect, is the consumer protection angle of the likes of the GDPR and CCPA. If the market where left to its own devices, the consumer of 3rd party on line goods and services, is not deemed to be protected to a satisfactory level. The regulatory intervention is not to rectify a negative externality affecting 3rd parties, more to protect the first party user. During the exchange of goods and services to the individual, it seems the requisite level of privacy and security that benefits the group of users as a whole is not utilitarian. A major aim the current cyber legislation is trying to achieve, is to overcome the information asymmetries that exist when a user signs up for a service or makes an online purchase or interaction.

With respect to consumer privacy, a concept of information asymmetry known as adverse selection may exist - where the buyer is not fully aware of the use and value of their personal data in relation to the supplier of a good or service, who may use their data in ways not fully understood or even disclosed to the user.

The likes of the NIS directive, seems more focused upon reducing the impact of an externality - basically a negative impact to a wider group of users. Perhaps, due to a data breach, service disruption or degradation, that may have occurred due to a lack of cyber security controls. A simple example could be the lack of power generation to an entire town if a nuclear power station is knocked offline due to a malware attack.

3) Product Hyper Augmentation

The cyber security market is broad, complex and ever evolving. The number of product categories grows daily. Gartner has at least 20 security related magic quadrants. CISO's and CIO's have to make incredibly complex decisions regarding product and service procurement.
Certainly, there is another set of information asymmetries at play here, but those are likely to exist in other complex software markets. With respect to cyber, there seems to an accelerated augmentation of features and messaging. When does a next generation web application firewall become a dedicated API security gateway? When does a security orchestration automation and response platform become an integrated event driven identity and access management workflow? Is there space for such niche upon niches, or are we entering a phase of largely merged and consolidated markets, where buyer procurement decision making is simply based on non-features such as brand affiliation and procurement ease?


Product direction via vertical and horizontal augmentation

Many mature markets often reach a position where suppliers augment products to a position of mediocrity and consumer apathy and confusion. Youngme Moon from Harvard Business School articulates this concept extremely well in her book Different - which focuses on competitive strategies. It seems the market for cyber security products especially (maybe less so for services and consultancy) is rapidly moving to a position, where core products are being blurred via augmentation, add on services and proprietary market descriptions. This is creating difficulties when it comes to calculating product purchase versus return on investment reporting.

4) Breach Increase

A continuation of the purchase/RoI analysis pattern, is to analyse what "success" looks like for numerous cyber investments. Whether those investments are people, process or technology related, most procurement decisions, end up being mapped to a success criteria. Value for money. Return on Investment. Call it what you will, but many organisations will look to describe what success looks like when it comes to security investments.

Is it a reduction in data breaches? Is it having fewer installed products with known CVE (common vulnerability & exposures) due to faster patch roll out? Is it having more end users signing up with second factor authentication? This can tie neatly into the controls -v- outcomes discussion where risk, security and privacy management for an organisation needs to identify tangible and SMART (specific measurable assignable realistic time-bound) metrics for implied cyber investment. The ultimate goal of cyber is to support the CIA (confidentiality integrity availability) triad, either singularly or collectively.

A major source of cyber investment success discussion, is associated with data breach trends. There are numerous pieces of data to support the claim, that breaches are increasing. Increasing in volume (from 157 in 2005 to 783 in 2014), breadth and complexity. Many of the articles could admittedly be FUD raised by vendors to accelerate product adoption, but there is no denying the popularity of sites like HaveIBeenPwned, where the number of breached credentials is substantial and increasing. If cyber investment was efficient, shouldn't these metrics be reducing?

This starts to generate two questions: either buyers are buying and using the wrong products or those products are not providing a decent return on investment.

5) Corporate Information Failure

But are products really to blame? The entire thread of this article, is to discuss market failure points. Information is a key component of effective free market development. Many information barriers seem to exist within the cyber security sector. Think of the following:
  • RoI on cyber product investment
  • Cost of personal data protection
  • The societal impact of critical infrastructure failures
  • Risk management success criteria
  • Cyber security certification benefit to corporations
There are likely several other angles to take on this, but full information with regards to the upholding of the confidentiality, availability and integrity of data is unlikely to occur. Many private sector organisations have undergone digital transformation over the last 10 years. These "corporation.next" operations, have created new challenges with respect to data protection. Data relating to customers, employees, intellectual property, products, suppliers, transactions and products.

But how do organisations a) know what to protect b) know how to protect it and c) innovate and manage investment strategies with respect to the protection?

There are many strategies used to manage cyber corporate investment. Some are driven by vendor FUD - aka breach threat - right through to modern risk management strategies, driven by mapping information protection to a higher level corporate strategy. 

If the corporate strategy is known and well communicated, it can become easier to overlay information protection decisions, that the business owners are willing to accept, monitor and iterate against. Risk transparency can help to provide a deeper understanding to what investments should be made and whether those investments are personnel, training or product related.

Summary

Cyber security is a growing, complex and multi faceted market. Many aspects are emerging, with new vendors, design patterns and attack vectors being created monthly. Other aspects, such as risk management and core protection of critical assets are relatively mature and well understood, in comparison to the computational age.

The investment and usage patterns associated with cyber security technology however, are seemingly plagued with numerous information failures, resulting in complex procurement, skills and personnel misalignment.

A value driven approach is needed, where explicit investment decisions (on both the skills provider, procurer and end user side) are weighed against short and long term returns.