DS: Zero Downtime Upgrade Strategy Using a Blue/Green Deployment

Introduction

This is the continuation of the previous blog about a Zero Downtime Upgrade Strategy Using a Blue/Green Deployment for AM. Traditionally, ForgeRock Directory Server (DS) upgrades are handled via a rolling upgrade strategy using an in-place update. As many deployments have constraints around this approach (zero downtime, immutable, etc.), a parallel deployment approach, also known as a blue/green strategy, can be leveraged for upgrading ForgeRock DS servers.

This blog provides a high-level approach for using a blue/green methodology for updating ForgeRock DS-UserStores.

This corresponds to Unit 3: DS-UserStores in our overall ForgeRock upgrade approach.

ForgeRock Upgrade Units
Unit 3: DS-Userstores Upgrade Process

Prerequisites/Assumptions

1. This approach assumes that your infrastructure processes have the ability to install a parallel deployment for an upgrade, or you are already using a blue/green deployment.

2. In the above diagram, the blue cluster reflects an existing DS deployment (like a 3.5.x version), and the green reflects a new DS deployment (like a 6.5.x version).

3. There are N+1 DS servers deployed in your existing deployment. N servers are used for your production workload and one server is reserved for maintenance activities like backup, upgrades, etc. If there is no maintenance server, then you may need to remove one server from the production cluster (thereby reducing production load capacity) or install an additional DS server node for this upgrade strategy.

4. Review release notes for all DS versions between existing and target DS deployments for new, deprecated features, bug fixes, and others. For a DS 3.5 to DS 6.5 upgrade, review the Release Notes for DS 5.0, 5.5, 6.0, and 6.5 versions.

Upgrade Process

1. Unconfigure replication for the DS-3 user store. Doing so ensures that the upgrade doesn’t impact your existing DS deployment.

2. Upgrade DS-3 in place using DS upgrade process.

3. Create a backup from DS-3 using the DS backup utility.

4. Configure green RS-1’s replication with the existing blue replication topology.

5. Configure green RS-2’s replication with the existing blue replication topology.

6. Install green DS-1 and restore data from backup using the DS restore utility.

7. Install green DS-2 and restore data from backup using the DS restore utility.

8. Install Green DS-3 and restore data from backup using the DS restore utility.

9. Configure Green DS-1’s replication with Green RS-1.

10. Configure Green DS-2’s replication with Green RS-1.

11. Configure Green DS-3’s replication with Green RS-1.

Switch Over to the New Deployment

12. After validating that the new deployment is working correctly, switch the load balancer from blue to green. This can also be done incrementally. If any issues occur, you can always roll back to the blue deployment.

If direct hostnames are used by DS clients, such as AM, IDM, etc., then those configurations need to be updated to leverage new green hostnames.

Post Go-Live

13. Unconfigure the blue RS1 replication server to remove this server from blue’s replication topology.

14. Unconfigure the blue RS2 replication server to remove this server from blue’s replication topology.

15. Stop the blue DS servers.

16. Stop the blue RS servers.

17. De-provision the blue deployment.

Conclusion

Although a blue/green deployment requires a high level of deployment maturity, this approach provides an elegant way to minimizing downtime for ForgeRock deployment upgrades. It is always advisable to try an upgrade strategy in lower environments like dev, stage before moving to a production environment.

Depending on the complexity of your deployment, there can be multiple things to be considered for these upgrades, such as customizations, new FR features, etc. It is always recommended to break the entire upgrade process into multiple releases like “base upgrade” followed by “leveraging new features”, and so on.

References

AM and IG: Zero Downtime Upgrade Strategy Using a Blue/Green Deployment

Introduction

The standard deployment for the ForgeRock Identity Platform consists of multiple ForgeRock products such as IG, AM, IDM, and DS. As newer ForgeRock versions are released, deployments using older versions need to be migrated before they reach their end of life. Also, newer versions of ForgeRock products provide features such as intelligent authentication and the latest OAuth standards, which help businesses implement complex use cases.

ForgeRock Deployment Components

Problem Statement

Traditionally, ForgeRock upgrades are handled via a rolling upgrade strategy using an in-place update. This strategy doesn’t suit all deployments due to the following constraints:

  • Many deployments don’t allow any downtime. This means production servers can’t be stopped for upgrade purposes.
  • Some deployments follow an immutable instances approach. This means no modification is allowed on the current running servers.

To resolve these constraints, a parallel deployment approach, or a blue/green strategy can be leveraged for upgrading ForgeRock servers.

Solution

This article provides a high-level approach for using a blue/green methodology for updating ForgeRock AM servers and related components like DS-ConfigStore, DS-CTS, AM-Agents, and IG servers. We plan to cover similar strategies for DS-UserStores and IDM in future articles.

In order to upgrade ForgeRock deployment, we need to first analyze the dependencies between various ForgeRock products and their impact on upgrade process:

Given the dependencies between ForgeRock products, it is generally advisable to upgrade AM before upgrading DS, AM agents, and others, as new versions of AM support older versions of DS and AM agents, but the converse may not be true.

Note: There can be some exceptions to this rule. For example:

  • Web policy agents 4.x are compatible with AM 6.0, but not with AM 6.5. This means the order of upgrade shall be existing version to AM 6.0 => AM Agent 4.x to 5.x => AM 6.0 to AM 6.5.x
  • If an AM-IDM integration is used, then both AM and IDM need to be upgraded at the same time.

Upgrade Units

ForgeRock Upgrade Units

A ForgeRock Identity Platform deployment can be divided into 4 units so that upgrade of these units can be handled individually:

  • Unit 1: AM and its related stores (DS-Config and DS-CTS)
  • Unit 2: AM-Agents/IG
  • Unit 3: DS-UserStores
  • Unit 4: IDM and its datastore

The order of upgrade used by our approach shall be Unit 1=>Unit 2=>Unit 3=>Unit 4.

Unit 1: AM Upgrade

Unit 1: AM Upgrade

Prerequisites/ Assumptions

1. This approach assumes that your infrastructure processes have the ability to install a parallel deployment for upgrade, or you are already using a blue/green deployment.

2. In the above diagram, the blue cluster reflects an existing AM deployment (like the 13.5.x version) and the green cluster reflects a new AM deployment (like the 6.5.x version).

3. There are N+1 AM servers and corresponding config stores deployed in your existing deployment. This means N servers are used for production load, and one server is reserved for maintenance activities like backup, upgrades, and others. If there is no such maintenance server, then you may need to remove one server from the production cluster (thereby reducing production load capacity) or install an additional node (AM server and corresponding config store) for this upgrade.

4. No sessions in CTS servers are replicated during blue/green switch; therefore, users are expected to re-authenticate after this migration. If your business use cases require users to remain authenticated, then these sessions (like OAuth Refresh tokens) need to be synced from the old to the new deployment. Mechanisms like ldif export/import or using IDM synchronization engine can be leveraged for syncing selective tokens from old to new deployments. Also, refer to the AM Release Notes on session compatibility across AM versions.

5. Review the Release Notes for all AM versions between existing and target AM deployment for new features, deprecated features, bug fixes, and so on for OpenAM 13.5 to AM 6.5 upgrade. Review the Release Notes for AM 5.0, 5.5, 6.0, and 6.5 versions.

Upgrade Process

1. Unconfigure replication for the DS-3 Config store. This ensures that the upgrade doesn’t impact an existing AM deployment.

2. Upgrade AM-3 in-place using the AM upgrade process. Note: You may need to handle new AM features in this process like AM 6.5 secrets, and others.

3. Export Amster configs from AM-3.

4. Transform Amster export so that the Amster export is aligned with a new green deployment such as DS hostname:port.

5. Install AM, DS-Config, and DS-CTS servers. Import the Amster export into a new green cluster. Note: For certain deployment patterns, such as ForgeRock immutable deployment, the Amster import needs to be executed for each AM node. If a shared config store is used, then the Amster import needs to be executed only once, and other nodes are required to be added to the existing AM site.

Switch Over to the New Deployment

6. After validating that the new deployment is working correctly, switch the load balancer from blue to green. This can also be done incrementally. If any issues occur, we can always roll back to the blue deployment.

Note: Any configuration changes made after the blue’s cluster’s Amster export should be applied to both blue and green deployments so that no configuration change is lost during switchover or rollback.

Post Go-Live

7. Stop the AM servers in the blue deployment.

8. Stop the Config and CTS DS servers in blue deployment.

9. De-provision the blue deployment.

Unit 2: AM-Agent/IG Upgrade

Unit 2: AM-Agent/IG Upgrade Process

AM-Agent

Prerequisites/ Assumptions

1. This approach assumes that your deployment (including applications protected by agents) has the ability to install a parallel deployment for upgrade, or you are already using a blue/green deployment.

2. In the above diagram, the blue cluster reflects an existing AM-Agent deployment and the green reflects new AM-Agent deployment.

3. A parallel base green deployment for protected app servers has already been created.

4. Create new Agent profiles for green deployment on AM servers.

5. This approach assumes both old and new AM-Agent versions are supported by the AM deployment version.

6. Refer to the Release Notes for latest and deprecated features in the new AM-Agent/IG version, such as the AM-Agent 5.6 Release Notes.

Upgrade Process

1. Install AM-Agents in the green deployment. Update agent profiles on the AM server (created in #4 above) for new agents deployed in the green deployment to match configurations used in agent profiles from the blue deployment. For certain AM versions, this process can be automated by retrieving existing Agent profiles and using these results to create new Agent profiles.

Switch Over to the New Deployment

2. After validating that the new deployment is working properly, switch the load balancer from blue to green.

Post Go-Live

3. Stop the app servers in the blue deployment.

4. Remove the blue agent profiles from AM deployment.

5. De-provision the blue deployment.

IG

Prerequisites/ Assumptions

1. This approach assumes that your deployment (including applications protected by agents) has the ability to install a parallel deployment for upgrade, or you are already using a blue/green deployment.

2. In the above diagram, the blue cluster reflects an existing IG deployment and the green reflects the new IG deployment.

3. This approach assumes both old and new IG versions are supported by the AM deployment version.

4. Create new Agent profiles for the green deployment on the AM servers required for IG servers.

5. Refer to the Release Notes for the latest and deprecated features in the new IG version, like IG 6.5 Release Notes.

Upgrade Process

1. Update the IG configs in the git repository as per the changes in the new version. You may create a different branch in your repository for the same.

2. Deploy the new green IG deployment by leveraging updated configurations.

Switch Over to the New Deployment

3. After validating that the new deployment is working fine, switch the load balancer from blue to green.

Post Go-Live

4. Stop the IG servers in the blue deployment.

5. De-provision the blue deployment.

Conclusion

Although a blue/green deployment requires a high level of deployment maturity, this approach provides an elegant way to minimize downtime for ForgeRock deployment upgrades. It is always advisable to practice an upgrade strategy in lower environments like dev, and stage before moving to a production environment.

Depending on the complexity of your deployment, there can be multiple things to be considered for these upgrade,s such as customizations, new FR features, migration to containers, and others. It is always recommended to break the entire upgrade process into multiple releases, like “base upgrade” followed by “leveraging new features”, and so on.

References

Configuring ForgeRock AM Active/Active Deployment Routing Using IG

Introduction

The standard deployment pattern for ForgeRock Identity Platform is to deploy the entire platform in multiple data centers/cloud regions. This is ensures the availability of services in case of an outage in one data center. This approach also provides performance benefits, as the load can be distributed among multiple data centers. Below is the example diagram for Active/Active deployment:

Problem Statement

AM provides both stateful/CTS-based and stateless/client-based sessions. Global deployment use cases require a seamless, single sign-on (SSO) experience among all applications with following constraints:

  • Certain deployments have distributed applications, such as App-A, deployed only in Data Center-A, and App-B, deployed only in Data Center-B.
  • The end user may travel to different locations, such as from the East coast to the West coast in the U.S. This means that application access requests will be handled by different data centers.

To achieve these use cases, CTS replication has to be enabled across multiple data centers/cloud regions.

In some situations, a user may try to access an application hosted in a specific data center before their corresponding sessions have been replicated. This can result in the user being prompted to re-authenticate, thereby degrading the user experience:

Note: This problem may be avoided if client-based sessions are leveraged, but many deployments have to use CTS-based sessions due to current limitations in client-based sessions. Also, when CTS-based sessions are used, the impact of CTS replication is much more than in client-based sessions.

In this article, we leverage IG to intelligently route session validation requests to a single data center, irrespective of the application being accessed.

Solution

IG can route session validation requests to a specific data center/region, depending on an additional site cookie generated during user’s authentication.

This approach ensures that the AM data center that issued the user’s session is used for corresponding session validation calls. This also means that CTS replication is not required across multiple data centers/ cloud regions:

Configure AM

  • Install AM 6.5.x and corresponding DS stores, Amster, and others. Following is a sample Amster install command:
install-openam — serverUrl http://am-A.example.com:8094/am — adminPwd cangetinam — acceptLicense — userStoreDirMgr “cn=Directory Manager” — userStoreDirMgrPwd “cangetindj” — userStoreHost uds1.example.com — userStoreType LDAPv3ForOpenDS — userStorePort 1389 — userStoreRootSuffix dc=example,dc=com — cfgStoreAdminPort 18092 — cfgStorePort 28092 — cfgStoreJmxPort 38092 — cfgStoreSsl SIMPLE — cfgStoreHost am-A.example.com — cfgDir /home/forgerock/am11 — cookieDomain example.com
am> connect http://am-A.example.com:8094/am -i
Sign in
User Name: amadmin
Password: **********
amster am-A.example.com:8094> import-config — path /home/forgerock/work/amster
Importing directory /home/forgerock/work/amster
Imported /home/forgerock/work/amster/global/Realms/root-employees.json
Imported /home/forgerock/work/amster/realms/root-employees/CoookieSetterNode/e4c11a8e-6c3b-455d-a875–4a1c29547716.json
Imported /home/forgerock/work/amster/realms/root-employees/DataStoreDecision/6bc90a3d-d54d-4857-a226-fb99df08ff8c.json
Imported /home/forgerock/work/amster/realms/root-employees/PasswordCollector/013d8761–2267–43cf-9e5e-01a794bd6d8d.json
Imported /home/forgerock/work/amster/realms/root-employees/UsernameCollector/31ce613e-a630–4c64–84ee-20662fb4e15e.json
Imported /home/forgerock/work/amster/realms/root-employees/PageNode/55f2d83b-724b-4e3a-87cc-247570c7020e.json
Imported /home/forgerock/work/amster/realms/root-employees/AuthTree/LDAPTree.json
Imported /home/forgerock/work/amster/realms/root/J2eeAgents/IG.json
Import completed successfully
- Creates /root realm aliases: am-A.example.com and am-B.example.com- AM Agent to be used by IG in /root realm- LDAPTree to create cookie after authentication. Update Cookie value as DC-A or DC-B, dependending on datacenter being used
  • Repeat the previous steps for configuring AM in all data centers:

Configure IG

- frProps.json to specify AM primary and secondary DC endpoints. Refer frProps-DC-A for DC-A and frProps-DC-B for DC-B.- config.json to declare primary and secondary AmService objects- 01-pep-dc-igApp.json to route session validation to specific datacenter, depending on “DataCenterCookie” value.
  • Repeat the previous steps for deploying IG in all data centers.

Test the use cases

The user accesses an application deployed in DC-A first

  1. The user accesses app1.example.com, deployed in DC-A.
  2. IG, deployed in DC-A, redirects the request to AM, deployed in DC-A for authentication.
  3. A DataCenterCookie is issued with a DC-A value.
  4. The user accesses app2.example.com, deployed in DC-B.
  5. IG, deployed in DC-B, redirects the request to AM, deployed in DC-A, for session validation.

The user accesses an application deployed in DC-B first

  1. The user accesses app2.example.com deployed in DC-B.
  2. IG, deployed in DC-B, redirects the request to AM deployed in DC-B, for authentication.
  3. A DataCenterCookie is issued with a DC-B value.
  4. The user accesses app1.example.com, deployed in DC-A.
  5. IG, deployed in DC-A, redirects request to AM, deployed in DC-B, for session validation.

Extend AM to OAuth/OIDC use cases

OAuth: AM 6.5.2 provides option to modify Access tokens using scripts. This allows additional metadata in stateless OAuth tokens, such as dataCenter. This information can be leveraged by IG OAuth RS to invoke the appropriate data center’s AmService objects for tokenInfo/ introspection endpoints:

{
 “sub”: “user.88”,
 “cts”: “OAUTH2_STATELESS_GRANT”,
 “auth_level”: 0,
 “iss”: “http://am6521.example.com:8092/am/oauth2/employees",
 …
 “dataCenter”: “DC-A”
}

OIDC: AM allows additional claims in OIDC tokens using scripts. This information can be leveraged by IG to invoke appropriate dataceter’s AmService objects

References

Extending IG as a complete UMA-RS

Both AM and IG support UMA 1.0.1 where AM acts as UMA Authorization Server (AS) and IG as UMA Resource Server (RS).

Currently there are some limitations in UMA support in IG, one of the most important is: PAT is stored in IG memory and is not persisted and if IG is restarted then the resource owner must perform the entire share process again.

Note: This post is based on UMA 1.0.1  (Support for UMA 1.0 and UMA 1.0.1 will be removed in a future version of ForgeRock Access Management) 

Solution

Versions used for this implementation: IG 5, AM 5.1 and DS 5

We can overcome some of these limitations by extending IG-UMA filter:

Some of the features of this extension:

  • Realm support
  • Extend IG-UMA REST endpoint: Authentication using PAT
  • User friendly UMA Resource name
  • Persisting UMA ResourceSet id and PAT in DS/OpenDJ:

UMA Flows

  • Alice share UMA resource
  • Bob access UMA resource

Deploy

See Also

This blog post was first published @ theinfinitelooper.blogspot.com, included here with permission.

Unlock user account using OpenAM Forgot Password flow

OpenAM provides “Account Lockout” functionality which can be used to configure various lockout parameters such as failure count, lockout interval etc.

Note that OpenDJ also provides Account Lockout functionality, this article is based on OpenAM Account Lockout policies. Refer this KB article for more differences between OpenAM and OpenDJ lockout polices.

Using OpenAM “Account Lockout” policies, users may get locked out with invalid login attempts. OpenAM offers both Memory and Physical lockouts. Using memory lockout, users get unlocked automatically after specified duration.

Many deployments use “Physical lockout” due to security requirements. When this lockout mode is used then there should be some Self-service flow so that user can unlock themselves. Why not use OpenAM forgot password self-service flow ?

OpenAM forgot password allows user to reset password after successfully completing various stages (such as KBA, email confirmation, reCaptcha etc). Unfortunately, the problem is that the account is not unlocked when this flow is used. There is already an open RFE for this issue.

Solution

Versions used for this implementation: OpenAM 13.5, OpenDJ 3.5
One of the solution can include extending out of the box OpenAM’s forgot password self-service flow by adding custom stage to unlock user’s account:
  • Implement ForgottenPasswordConfigProviderExt to include account unlock stage.
  • Implement unlock custom stage
  • Extend selfServiceExt.xml to include custom provider.

Deploy

  • Build the custom stage by using maven.
  • Delete all instances of User Self-Service from all realms.
  • Remove existing selfService
./ssoadm delete-svc --adminid amadmin --password-file /tmp/pwd.txt -s selfService
  • Restart OpenAM
  • Register custom selfService
  • Restart OpenAM
./ssoadm create-svc --adminid amadmin --password-file /tmp/pwd.txt --xmlfile ~/softwares/selfServiceExt.xml
  • Add User Self-Service to specified realm and enable forgot password flow.

Testing

  1. Lock user by authenticating using wrong password till user is locked out.
  2. Follow forgot password flow to reset password and unlock account.
  3. Try authenticating again with new password. This should succeed.

This blog post was first published @ theinfinitelooper.blogspot.com, included here with permission.

Extending OpenAM HOTP module to display OTP delivery details

OpenAM provide HOTP authentication module which can send OTP to user’s email address and/or telephone number. By default, OpenAM doesn’t displays user’s email address and/or telephone number while sending this OTP.

Solution

Versions used for this implementation: OpenAM 13.5, OpenDJ 3.5
One of the solution can include extending out of the box OpenAM’s HOTP module:
  • Extend HOTP auth module (openam-auth-hotp).
  • Update below property in extended amAuthHOTP.properties: send.success=Please enter your One Time Password sent at
  • Extend HOTPService appropriately to retrieve user profile details.
  • Change extended HOTP module code as per below (both for auto send and on request):

substituteHeader(START_STATE, bundle.getString("send.success") + <Get User contact details from HOTPService>);

Deploy

Register service and module (Note that for OpenAM v12 use amAuthHOTPExt-12.xml) :
$ ./ssoadm create-svc --adminid amadmin --password-file /tmp/pwd.txt --xmlfile ~/softwares/amAuthHOTPExt.xml
$ ./ssoadm register-auth-module --adminid amadmin --password-file /tmp/pwd.txt --authmodule com.sun.identity.authentication.modules.hotp.HOTPExt

UnRegister service and module (in case module needs to be uninstalled) : 
$ ./ssoadm unregister-auth-module --adminid amadmin --password-file /tmp/pwd.txt --authmodule com.sun.identity.authentication.modules.hotp.HOTPExt
$ ./ssoadm delete-svc --adminid amadmin --password-file /tmp/pwd.txt -s sunAMAuthHOTPExtService
  • Configure HOTPExt module with required SMTP server. Enable both SMS and Email.
  • Create a chain(otpChain) with (LDAP:Required, HOTPExt:Required). Set this chain as default for “Organization Authentication”
  • Restart OpenAM
  • Invoke HOTP module and appropriate message is displayed on screen with user’s email address and/or telephone number:

 

This blog post was first published @ theinfinitelooper.blogspot.com, included here with permission.

OpenAM SP SAML Attribute Mapper extension for updating profile attributes

OpenAM can act as both SP and IdP for SAML webSSO flows. OpenAM also provides ability to dynamically create user profiles.

When OpenAM is acting as SAML SP and Dynamic user profile is enabled, if user profile doesn’t exist on OpenAM then OpenAM dynamically creates this profile from attributes in SAML assertion.
The problem comes if user profile is updated at IdP side, all subsequent SAML webSSO flows doesn’t update these changes at OpenAM SP side. More details here: OPENAM-8340

Solution

Versions used for this implementation: OpenAM 13.5, OpenDJ 3.5

One of the solution can include extending OpenAM SP Attribute Mapper. This extension may include just checking if user profile exists in OpenAM SP and updating any modified or new attributes in OpenAM datastore. Some tips for this implementation:

  1. Extend DefaultSPAttributeMapper and override getAttributes()
  2. Get datastore provider from SAML2Utils.getDataStoreProvider()
  3. Check if user exists: dataStoreProvider.isUserExists(userID)
  4. Get existing user attributes: dataStoreProvider.getAttributes()
  5. Compare attributes in SAML assertion with existing user attributes.
  6. Finally persist any new and updated attributes: dataStoreProvider.setAttributes()

Deploy

  • Compile and deploy this extension in OpenAM under  (OpenAM-Tomcat)/webapps/openam/WEB-INF/lib
  • Change SAML attribute setting in OpenAM. Navigate to Federation > Entity Providers > (SP Hosted Entity) > Assertion Processing. Specify ‘org.forgerock.openam.saml2.plugins.examples.UpdateDynamicUserSPAttMapper’ under Attribute Mapper.
  • Restart OpenAM
  • And we are good to go! Any changes in user profile attributes in SAML assertion will now be persisted in OpenAM datastore.

Note that ideally attributes between different sources should be synced by using some tool like OpenIDM 

See Also

Get code: https://github.com/CharanMann/OpenAM-SAMLSP-updateDynamicUser
OpenAM User Profile settings: https://backstage.forgerock.com/docs/openam/13.5/admin-guide#auth-core-realm-attributes
OpenAM SAML configuration: https://backstage.forgerock.com/docs/openam/13.5/admin-guide#chap-federation

This blog post was first published @ theinfinitelooper.blogspot.com, included here with permission.