• Summary
• Motivation
  • Goals
• Proposal
• Design and implementation details
  • Phase one: Zuora Subscription Cache
    • Proposed DB schema
    • Notes
    • Keeping data in sync with Zuora
    • Read only models
    • Rollout of Zuora Cache models
  • Phase two: Utilize Zuora Cache Models
  • Phase three: Transition from Order to Subscription
    • Proposed DB schema
    • Notes
    • Keeping data in sync with Zuora
    • Unknowns
      • Trial data in Subscription?
  • Resources

Align CustomersDot Orders with Zuora Orders

Summary

The GitLab Customers Portal is an application separate from the GitLab product that allows GitLab Customers to manage their account and subscriptions, tasks like purchasing additional seats. More information about the Customers Portal can be found in the GitLab docs. Internally, the application is known as CustomersDot (also known as CDot).

GitLab uses Zuora’s platform to manage their subscription-based services. CustomersDot integrates directly with Zuora Billing and treats Zuora Billing as the single source of truth for subscription data.

CustomersDot stores some subscription and order data locally, in the form of the orders database table, which at times can be out of sync with Zuora Billing. The main objective for this blueprint is to lay out a plan for improving the integration with Zuora Billing, making it more reliable, accurate, and performant.

Motivation

Working with the Order model in CustomersDot has been a challenge for Fulfillment engineers. It is difficult to trust Order data as it can get out of sync with the single source of truth for subscription data, Zuora Billing. This has led to bugs, confusion and delays in feature development. An epic exists for aligning CustomersDot Orders with Zuora objects which lists a variety of issues related to these data integrity problems. The motivation of this blueprint is to develop a better data architecture in CustomersDot for Subscriptions and associated data models which builds trust and reduces bugs.

Goals

This re-architecture project has several multifaceted objectives.

• Increase the accuracy of CustomersDot data pertaining to Subscriptions and its entitlements. This data is stored as Order records in CustomersDot - it is not granular enough to represent what the customer has purchased, and it is error prone as shown by the following issues:
  • Multiple order records for the same subscription
  • Multiple subscriptions active for the same namespace
  • Support Multiple Active Orders on a Namespace
• Continue to align with Zuora Billing being the SSoT for Subscription and Order data.
• Decrease dependency and reliance on Zuora Billing uptime.
• Improve CustomersDot performance by storing relevant Subscription data locally and keeping it in sync with Zuora Billing. This could be a key piece to making Seat Link more efficient and reliable.
• Eliminate confusion between CustomersDot Orders, which contain data more closely resembling a Subscription, and Zuora Orders, which represent a transaction between a customer and merchant and can apply to multiple Subscriptions.
  • The CustomersDot orders table contains a mixture of Zuora Subscription and trials, along with GitLab-specific metadata like sync timestamps with GitLab.com. GitLab does not store trial subscriptions in Zuora at this time.

Proposal

As the list of goals above shows, there are a good number of desired outcomes we would like to see at the end of implementation. To reach these goals, we will break this work up into smaller iterations.

1. Phase one: Zuora Subscription Cache

   The first iteration focuses on adding a local cache for Zuora Subscription objects, including Rate Plans, Rate Plan Charges, and Rate Plan Charge Tiers, in CustomersDot.

2. Phase two: Utilize Zuora Cache Models

   The second phase involves using the Zuora cache models introduced in phase one. Any code in CustomersDot that makes a read request to Zuora for Subscription data should be replaced with an ActiveRecord query. This should result in a big performance improvement.

3. Phase three: Transition from Order to Subscription

   The next iteration focuses on transitioning away from the CustomersDot Order model to a new model for Subscription.

Design and implementation details

Phase one: Zuora Subscription Cache

The first phase for this blueprint focuses on adding new models for caching Zuora Subscription data locally in CustomersDot. These local data models will allow CustomersDot to query the local database for Zuora Subscriptions. Currently, this requires querying directly to Zuora which can be problematic if Zuora is experiencing downtime. Zuora also has rate limits for API usage which we want to avoid as CustomersDot continues to scale.

This phase will consist of creating the new data models, building the mechanisms to keep the local data in sync with Zuora, and backfilling the existing data. It will be important that the local cache models are read-only for most of the application to ensure the data is always in sync. Only the syncing mechanism should have the ability to write to these models.

Proposed DB schema

Notes

• The namespace Zuora is already taken by the classes used to extend IronBank resource classes. It was decided to move these to the namespace Zuora::Remote to indicate these are intended to reach out to Zuora. This frees up the Zuora namespace to be used to group the models related to Zuora cached data.
• All versions of Zuora Subscriptions will be stored in this table to be able to support display of current as well as future purchases when Zuora is down. One of the guiding principles from the Architecture Review meeting on 2023-08-06 was “Customers should be able to view and access what they purchased even if Zuora is down”. Given that customers can make future-dated purchases, CustomersDot needs to store current and future versions of Subscriptions.
• zuora_id would be the primary key given we want to avoid the field name id which is magical in ActiveRecord.
• The timezone for Zuora Billing is configured as Pacific Time. Let’s account for this timezone as we sync data from Zuora into CDot’s cached models to allow for more accurate comparisons.

Keeping data in sync with Zuora

CDot currently receives and processes Order Processed Zuora callouts for Order actions like Update Product (full list). These callouts help to keep CustomersDot in sync with Zuora and trigger provisioning events. These callouts will be important to keeping Zuora::Subscription and related cached models in sync with changes in Zuora.

This existing callout would not be sufficient to cover all changes to a Zuora Subscription though. In particular, changes to custom fields may not be captured by these existing callouts. We will need to create custom events and callouts for any custom field cached in CustomersDot for any of these resources to ensure CDot is in sync with Zuora. This should only affect Zuora::Subscription though as no custom fields are used by CustomersDot on any of the other proposed cached resources at this time.

Read only models

Given the data stored in these new models are a copy of Zuora data, it will important to ensure these models are modified within the appropriate context, not throughout the application. We want a clear separation when a cached model can be in “write” mode versus “read-only” mode. This separation helps avoid writing to a cached model inappropriately or mistakenly. We considered different options as part of this Spike issue.

We aligned on creating a concern, ReadOnlyRecord, that will prevent a save when included in an ActiveRecord model.

module ReadOnlyRecord
  extend ActiveSupport::Concern

  included do
    after_initialize :readonly!
  end
end

• Attempting to save (e.g. create, update, or destroy) one of these models would raise an error (e.g. ActiveRecord::ReadOnlyRecord: Subscription is marked as readonly)
• Even with this code, a record could still be deleted with record.delete. We could write a rubocop cop for avoiding using delete (possibly even for just these ReadOnlyModels). We could also overwrite this method to raise an error as well.
• Within certain namespaces like the Zuora cache sync service, we want access to a model that have write privileges.

Rollout of Zuora Cache models

With the first iteration of introducing the cached Zuora data models, we will take an iterative approach to the rollout. There should be no impact to existing functionality as we build out the models, start populating the data through callouts, and backfill these models. Once this is in place, we will iteratively update existing features to use these cached data models instead of querying Zuora directly.

We will make this transition using many small scoped feature flags, rather than one large feature flag to gate all of the new logic using these cache models. This will help us deliver more quickly and reduce the length with which feature flag logic is maintained and test cases are retained.

Testing can be performed before the cached models are used in the codebase to ensure data integrity of the cached models.

Phase two: Utilize Zuora Cache Models

This phase covers the second phase of work of the Orders re-architecture. In this phase, the focus will be utilizing the new Zuora cache data models introduced in phase one. Querying Zuora for Subscription data is fundamental to Customers so there are plenty of places that will need to be updated. In the places where CDot is reading from Zuora, it can be replaced by querying the local cache data models instead. This should result in a big performance boost by avoiding third party requests, particularly in components like the Seat Link Service.

This transition will be completed using many small scoped feature flags, rather than one large feature flag to gate all of the new logic using these cache models. This will help to deliver more quickly and reduce the length with which feature flag logic is maintained and test cases are retained.

Phase three: Transition from Order to Subscription

The second phase for this blueprint focuses on transitioning away from the CustomersDot Order model to a new model for Subscription. This phase will consist of creating a new model for Subscription, supporting both models during the transition period, updating existing code to use Subscription and finally removing the Order model once it is no longer needed.

Replacing the Order model with a Subscription model should address the goal of eliminating confusion around the Order model. The data stored in the CustomersDot Order model does not correspond to a Zuora Order. It more closely resembles a Zuora Subscription with some additional metadata about syncing with GitLab.com. The transition to a Subscription model, along with the local cache layer in phase one, should address the goal of better data accuracy and building trust in CustomersDot data.

Proposed DB schema

Notes

• The name for this model is up for debate given a Subscription model already exists. The existing model could be renamed with the hope of eventually replacing it with the new model.
• This model serves as a record of the Subscription that is modifiable by the CDot application, whereas the Zuora::Subscription table below should be read-only.
• zuora_account_id could be added as a convenience but could also be fetched via the billing_account.
• There will be one Subscription record per actual subscription instead of a Subscription version.
  • This has the advantage of avoiding duplication of fields like gitlab_namespace_id or last_extra_ci_minutes_sync_at.
  • The zuora_subscription_id column could be removed or kept as a reference to the latest Zuora Subscription version.

Keeping data in sync with Zuora

The Subscription model should stay in sync with Zuora as subscriptions are created or updated. This model will be synced when we sync Zuora::Subscription records, similar to how the cached models are synced when processing Zuora callouts as described in phase one. When saving a new version of a Zuora::Subscription, an update could be made to the Subscription record with the matching zuora_subscription_name, or create a Subscription if one does not exist. The zuora_subscription_id would be set to the latest version on typical updates. Most of the data on Subscription is GitLab metadata (e.g. last_extra_ci_minutes_sync_at) so it wouldn’t need to be updated.

The exception to this update rule are the zuora_account_id and billing_account_id attributes. Let’s consider the current behavior when processing an Order Processed callout in CDot if the zuora_account_id changes for a Zuora Subscription:

1. The Billing Account Membership is updated to the new Billing Account for the CDot Customer matching the Sold To email address.
2. CDot attempts to find the CDot Order with the new billing_account_id and subscription_name.
3. If an Order isn’t found matching this criteria, a new Order is created. This leads to two Order records for the same Zuora Subscription.

This scenario should be avoided for the new Subscription model. One Subscription should exist for a unique Zuora::Subscription name. If the Zuora Subscription transfers Accounts, the Subscription should as well.

Unknowns

Several unknowns are outlined below. As we get further into implementation, these unknown should become clearer.

Trial data in Subscription?

The CDot Order model contains paid subscription data as well as trials. For Subscription, we could choose to continue to have paid subscription and trial data together in the same table, or break them into their own models.

The orders table has fields for customer_id and trial which only really concern trials. Should these fields be added to the Subscription table? Should Subscription contain trial information if it doesn’t exist in Zuora?

If trial orders were broken out into their own table, these are the columns likely needed for a (SaaS) trials table:

• customer_id
• product_rate_plan_id (or rename to plan_id or use plan_code)
• quantity
• start_date
• end_date
• gl_namespace_id
• gl_namespace_name

Resources

• FY24Q3 OKR - Create plan to align CustomersDot Orders to Zuora Orders
• Epic &9748 - Align CustomersDot Orders to Zuora objects