Experimentation Data Architecture

We define a canonical experiment model that is stable across tools and products. At minimum this includes: experiment identifier and version, variant identifiers, allocation and targeting rules, start/stop timestamps, and status transitions (draft, running, ramping, paused, concluded). We also capture metadata needed for interpretation later, such as primary metric, guardrail metrics, segmentation dimensions, and the source system that executed the assignment. In the warehouse, we typically separate reference data (experiment definitions) from behavioral data (assignment/exposure and outcomes). Reference data can come from an experimentation platform API export, a configuration repository, or a controlled table maintained by the experimentation program. Behavioral data is represented as immutable events with strong keys, so historical analyses remain reproducible even if naming or targeting rules evolve. We also design for versioning and deprecation. When an experiment is re-run or modified, the model supports explicit versions rather than overwriting prior definitions. This prevents ambiguous joins and enables longitudinal learning across experiments.

Assignment is the moment a user is allocated to a variant; exposure is the moment the user actually experiences the treatment (for example, the UI renders, an API response includes the change, or a feature flag is evaluated in a way that affects behavior). Many organizations only log one of these, or log them inconsistently, which can bias results. From an architecture perspective, we treat assignment and exposure as separate events with explicit semantics. Assignment is useful for intent-to-treat analysis and for debugging allocation logic. Exposure is essential for estimating treatment effects when not all assigned users actually see the change (due to caching, eligibility, client errors, or navigation paths). We define rules for when to log each, how to deduplicate, and how to handle edge cases such as multiple exposures, cross-device sessions, and server-side rendering. This clarity supports statistically valid analysis and makes it easier to compare results across tools and products.

We implement data quality controls at multiple layers: instrumentation validation, pipeline validation, and analytical sanity checks. On the instrumentation side, we validate schema conformance (required fields present, correct types, allowed values) and detect drift when payloads change. In pipelines, we check for late-arriving events, duplicate keys, and join integrity between exposure and outcome tables. For experimentation-specific monitoring, we add checks such as expected exposure volume, variant balance against allocation, and sudden shifts in eligibility rates. We also monitor identity metrics (anonymous-to-known stitching rates, cross-device duplication) because identity issues can silently invalidate experiment populations. Operationally, we recommend dashboards and alert thresholds that are owned jointly by analytics engineering and the experimentation program. The goal is to detect regressions quickly, isolate the affected release or product area, and provide a clear runbook for remediation and backfills when needed.

Exposure data is particularly sensitive to duplication because client retries, network failures, and page reloads can inflate counts and bias metrics. We design a deduplication strategy based on stable keys and clear event semantics. Common approaches include generating an exposure_id at the time of logging, or deriving a deterministic key from user/session, experiment, variant, and a bounded time window. For late events, we define acceptable lateness and implement incremental processing that can update recent partitions without rewriting the entire dataset. This often includes watermarking, partitioning by event time, and maintaining a small reprocessing window to capture delayed mobile events or offline queues. We also distinguish between “multiple exposures” that are valid (a user sees the treatment across sessions) and duplicates that are not. The curated tables typically include both raw exposure counts and a canonical first-exposure record per user per experiment version, so analysts can choose the appropriate methodology.

We align experimentation events with the same collection and governance mechanisms used for product analytics in the CDP. That means defining experiment context as first-class fields in the event schema (experiment_id, variant_id, exposure_type, eligibility, and optionally allocation metadata), and ensuring those fields are propagated consistently across web, mobile, and backend sources. In practice, we design where experiment context is attached: as dedicated assignment/exposure events, as context fields on downstream behavioral events, or both. The choice depends on your analysis needs and the capabilities of the experimentation platform. We also ensure identity resolution rules are consistent with the CDP’s identity graph so users are not counted in multiple variants due to stitching gaps. Downstream, we build transformations that normalize CDP raw events into curated experiment datasets. This preserves lineage from CDP ingestion through warehouse tables and supports consistent reporting in BI and experimentation readouts.

Yes, but it requires an explicit abstraction layer. Different tools represent experiments, variants, and exposures differently, and their exports often embed tool-specific assumptions. We design a canonical model and mapping rules so each tool’s raw data is transformed into the same curated structure. For example, feature flag evaluations may need to be translated into exposure semantics, and server-side experiments may require different identifiers and timing rules than client-side A/B tests. We define normalization logic for identifiers, variant naming, and lifecycle states, and we standardize how exposure is determined. This approach allows product teams to use the right tool for a given context while keeping analysis consistent. It also reduces migration risk because historical results remain interpretable even if one tool is replaced or consolidated later.

We treat metrics as governed assets with explicit definitions, owners, and change control. A metric definition typically includes the event sources, filters, identity rules, time windows, attribution logic, and the exact aggregation method. We then implement these definitions in a shared semantic layer or a controlled set of warehouse models. Governance includes a review workflow for changes, versioning for breaking updates, and documentation that is accessible to product and analytics teams. For experimentation, we also define which metrics are eligible as primary metrics, which are guardrails, and which require special handling (for example, revenue metrics with refunds, or metrics sensitive to seasonality). This reduces the common failure mode where each experiment analysis re-implements metrics slightly differently. It also improves comparability across experiments and makes it easier to audit how a reported lift was calculated months later.

We recommend documentation that is both implementation-ready and enforceable. At a minimum: an event and parameter catalog for assignment/exposure, a canonical experiment identifier policy, examples for each client type (web, mobile, backend), and a runbook for validating new experiments before launch. We also document how experiment context is joined to outcomes and which curated tables are the source of truth. Ownership is typically split: the experimentation program (or product analytics) owns the conceptual model and metric catalog, while analytics engineering owns the pipeline implementation and data quality controls. Product teams own correct instrumentation in their codebases, but they should not be responsible for redefining schemas or metrics. We also recommend a lightweight change control process: schema changes require review, metrics changes require versioning, and new experiment types (for example, server-side) require an explicit design update to avoid fragmentation.

The most common invalidation risks are: incorrect exposure logging, identity duplication, variant contamination, and inconsistent metric computation. Exposure issues include logging assignment as exposure, missing exposures due to client errors, or double-counting due to retries. Identity issues include users appearing in multiple variants because anonymous and authenticated identities are not stitched consistently. Variant contamination happens when users can switch variants (for example, due to inconsistent bucketing, caching layers, or server/client disagreement). Metric inconsistency occurs when analysts apply different filters, time windows, or attribution rules across tools. We mitigate these through explicit tracking contracts (assignment vs exposure), deterministic identity and eligibility rules, deduplication keys, and automated validation checks such as variant balance and exposure rate monitoring. We also standardize metric definitions in a governed layer so experiment readouts are computed consistently and can be reproduced later.

We design experimentation tracking to align with your privacy model rather than treating it as a special case. That includes ensuring exposure events respect consent signals, minimizing collection of unnecessary identifiers, and defining retention policies for raw and curated datasets. Where required, we support pseudonymization and separation of identifiers from behavioral data. We also ensure that experiment metadata does not leak sensitive targeting logic into broadly accessible datasets. For example, eligibility criteria may be represented as high-level flags rather than detailed attributes. Access control is handled through dataset permissions and, where applicable, row-level security for sensitive segments. Operationally, we document how consent affects experiment populations and analysis (for example, when consented users differ systematically). This prevents misinterpretation of results and supports compliance reviews by making data flows and retention explicit and auditable.

Typical deliverables include a canonical experiment data model, event schema specifications for assignment and exposure, and implementation guidance for each client type (web, mobile, backend). On the data side, we deliver curated warehouse tables that join exposure context to outcomes, plus documented transformation logic and lineage fields to support reproducibility. We also provide a governed metric catalog aligned to your semantic layer, including definitions for primary and guardrail metrics commonly used in experiments. Data quality checks and monitoring are included, with thresholds and runbooks for investigating anomalies such as variant imbalance or sudden exposure drops. Finally, we deliver governance artifacts: ownership model, change control workflow, and documentation that enables teams to launch new experiments without reintroducing inconsistent tracking. The exact scope is tailored to your current maturity and the experimentation tools in use.

Collaboration usually starts with a short audit focused on one or two representative products and a small set of recent experiments. We review: how assignment and exposure are implemented, what data is emitted into the CDP, how identity is stitched, how metrics are defined, and how results are currently produced in the experimentation tool and in the warehouse/BI layer. From that audit, we produce a target architecture and a prioritized implementation plan. The plan identifies quick wins (for example, standardizing exposure events and keys), foundational work (canonical model, metric contracts), and rollout sequencing across teams. We also define acceptance criteria such as reproducibility checks and data quality thresholds. Engagements then proceed in phases: implement the core model and pipelines, roll out instrumentation contracts to product teams, validate with test experiments, and establish governance and monitoring. This approach reduces disruption while creating a stable foundation for scaling experimentation.