Meta

What the interviewer is probing

This question probes architecture thinking across the kernel-userspace boundary: does the candidate understand why a general-purpose scheduler is a poor fit for latency-sensitive fan-out workloads, and can they design a partitioning policy with explicit trade-offs around pool sizing, cache affinity, and failure modes? The deployment model — policy as a reloadable user-space artifact rather than a kernel patch — tests whether the candidate sees iteration speed as an architectural constraint worth designing around. The follow-up on application-scheduler interfaces probes systems thinking around trust boundaries and interface design: who gets to signal priority, and what prevents that interface from becoming a source of instability.

Strong answer outline

• Diagnose first, design second. Before designing a custom policy, characterize the regression: use scheduler tracing (e.g., perf sched, BPF tracepoints on context switches, runqueue wait times) to identify whether the regression is excess preemption, poor cache affinity, priority inversion, or NUMA thrashing. The fix is different for each.
• Model the workload structure. A request-serving fan-out has a clear thread taxonomy: request-handling threads on the critical path (latency-sensitive, short-burst), background threads (compaction, logging, async I/O — throughput-oriented). A general-purpose scheduler cannot distinguish these; a custom policy can encode this knowledge directly.
• CPU pool partitioning. Soft-partition CPUs into a latency pool and a background pool. 'Soft' is important — hard partitioning wastes capacity during imbalance; soft partitioning allows the pools to borrow from each other under load, with a priority bias rather than a hard boundary. Pool sizing starts from observed utilization ratios and is adjusted dynamically via load-based heuristics.
• Cache locality as a first-class design goal. Scheduling related threads to the same physical cores over time improves L3 hit rates and reduces DRAM accesses — critical when the fan-out brings together many threads working on shared data structures. NUMA-aware placement (keep threads and their memory on the same socket) is a follow-on optimization.
• Deployment model determines iteration speed. Encoding the policy as a reloadable user-space program (BPF program loaded by a user-space daemon) decouples the scheduling improvement cycle from the kernel release cycle. Rolling out a change is a daemon restart, not a kernel patch with months of validation. This is an architectural choice, not an implementation detail — it is what makes continuous optimization tractable.
• Experiment framework for live scheduling changes. A/B testing two scheduling policies on a live fleet requires: (a) host-level assignment (not request-level, to avoid mixing policies on the same host), (b) a grace period on policy switchover to let in-flight requests drain, (c) a rollback trigger based on p99 latency and timeout error rate guardrails, (d) metrics stratified by traffic mix to catch load-dependent effects.
• Common wrong turns: designing for average latency rather than tail latency (the business impact lives at p99+); using hard CPU partitioning and starving one pool; building the policy into the kernel binary rather than user space (kills iteration speed); skipping the diagnosis step and tuning a policy against the wrong bottleneck.
• Application-scheduler interface (the hardest part). Allowing the application to signal thread importance (e.g., 'this thread is serving a high-value request') is powerful but risky. Trust model must be explicit: the scheduler must treat these signals as hints, not hard constraints, with a cap on the boost any thread can claim. Abuse vectors — all threads claiming high priority — must be rate-limited or quota-bounded. The interface should be instrumented so gaming is detectable.

The underlying concept

General-purpose CPU schedulers like CFS and EEVDF optimize for fairness and throughput across arbitrary workloads — they are deliberately ignorant of application semantics. For most workloads this is fine, but for latency-sensitive request-serving systems the cost of that ignorance is measurable: threads on the critical path compete on equal footing with background work, cache affinity is not preserved across scheduling decisions, and small changes in the scheduler's virtual-time accounting can shift tail latency by tens of milliseconds. The key insight behind workload-aware scheduling is that the application already has the information the scheduler lacks — which threads are on the critical path, which are background, which requests are high-value — and a custom scheduler can use that information to make better decisions than any general-purpose heuristic. The deeper architectural point is that where the policy lives determines how fast you can improve it: a policy encoded in a kernel module requires a full kernel release cycle to ship; a policy encoded in a reloadable BPF program ships in days. At the scale where a single percentage point of latency improvement translates to megawatts of power savings and measurable revenue lift, that iteration speed difference is itself a strategic asset.

Source

Derived from Modernizing the Meta Ads Service With an Open-Source Kernel Scheduler

What the interviewer is probing

This question tests whether the candidate can translate an OS-level abstraction — hierarchical memory with demand paging and prefetching — into a distributed systems design, and whether they can reason about the real optimization target (researcher iteration speed) rather than the obvious one (data-loading throughput). Strong candidates will recognize that the write-once, read-many pattern of training datasets makes aggressive caching safe, will design a prefetch API that hides cross-region latency behind compute work, and will explicitly frame the trade-off between performance predictability (full pre-copy) and iteration speed (on-demand hydration) rather than claiming the new design is strictly better.

Strong answer outline

• Frame the actual problem: For large training runs, pre-copy is fine — it amortizes over weeks. The problem is small iterative experiments where researchers spend more time waiting on ingestion than on the model. The goal is to cut the setup time from hours to minutes for this workload, accepting occasional I/O stalls that wouldn't occur with a full pre-copy.
• Core insight — treat storage as a tiered cache:
• L1: GPU-host memory (fastest, smallest — the current training batch)
• L2: Local flash on the GPU host (warm data across a few batches)
• L3: Regional disaggregated flash storage, co-located with GPUs (data for the current job's duration)
• Source of truth: global HDD-backed object store (remote, durable, cheap)
• This mirrors OS demand paging: data is pulled toward compute on first access and cached for reuse.
• Demand hydration: On a cache miss at L3, the client transparently fetches from the remote source of truth. First-access latency will be higher (cross-region), so the design must hide this.
• Prefetching to hide latency:
• Dataloader prefetch: the dataloader pre-loads the next batch into memory while processing the current one — standard overlap of I/O and compute.
• Explicit deep prefetch API: the dataloader triggers background hydration of the next several minutes of data into L3 before it's needed, prewarms metadata cache. This is what allows the first real training step to start before all data is local.
• Multi-epoch reuse: Training iterates over the same data multiple times. The L3 cache should hold data for the duration of the training cycle (TTL tied to job lifetime or a configured window), not evict after first use. LRU within that window handles capacity pressure when multiple jobs share the cache.
• Consistency: Training datasets are write-once — once a researcher publishes a dataset version, it doesn't change during the training run. Cache invalidation is therefore not a hot path; it happens on explicit dataset version update. Version identifiers in the cache key prevent stale reads across experiments.
• Failure mode: If the remote source of truth is unreachable and L3 has a miss, the training step stalls. For iterative small jobs this is acceptable; for production runs you'd want the full pre-copy. The system should expose a metric for cross-region fetch rate so researchers can identify when they've crossed the threshold where pre-copying would have been cheaper.
• Trade-offs to name explicitly: First-epoch performance will be lower than pre-copy (cold L3). Subsequent epochs are fast (warm L3). Power and flash capacity are consumed in the GPU region. The design is strictly worse than pre-copy on throughput predictability, and strictly better on time-to-first-step.
• Common wrong turns: Candidates who propose caching without a prefetch strategy will have a cold-start problem on the first epoch that stalls GPUs. Candidates who don't discuss eviction policies will overrun flash capacity. Candidates who claim this design is better in all cases have not thought carefully about the large-run use case.

The underlying concept

Operating systems have solved the problem of 'data is far away but compute needs it now' through hierarchical caching with demand paging and prefetching: bring data close to compute on first use, keep it there for reuse, and hide the fetch latency behind ongoing computation. Distributed training storage can borrow this model directly — the key enabling property is that training datasets are write-once and read-many, which makes aggressive caching safe (no cache coherence problem during a run) and makes prefetching tractable (access patterns are sequential and predictable). The architectural insight is that 'ingestion' and 'training' don't have to be sequential phases; demand hydration with deep prefetching collapses them into a single overlapped operation, converting an hours-long blocking step into a background process invisible to the researcher.

Source

Derived from Meta’s AI Storage Blueprint at Scale

What the interviewer is probing

This question tests metric judgment and causal reasoning in a multi-stage ML system: candidates must reason about offline-online gaps, latency distributions (not just means), the difficulty of attributing quality to a stage when downstream components compensate, and second-order effects like content fairness and recency that engagement metrics don't directly capture. Strong candidates treat latency as a budget with a distribution, not a single number, and raise the isolation problem unprompted.

Strong answer outline

• Decision framing: The question isn't just "does NDCG go up" — it's whether the improvement is real (not an artifact of offline evaluation), large enough to justify latency cost, and durable enough to survive production conditions. Frame the decision across three dimensions: quality impact, latency risk, and second-order effects.
• Offline-online gap diagnosis: NDCG on a held-out set measures whether the reranker correctly orders items that were eventually shown and interacted with — but it doesn't capture exploration (items never retrieved under the old system that would have been great), position bias in the training labels, or distribution shift between the offline eval set and live traffic. Common causes of a flat online experiment: training labels reflect the old retrieval distribution, so the reranker learned to score items the old system already selected well; or the downstream ranking model compensates for changes in retrieval order, absorbing the signal.
• Latency as a distribution, not a point: p50 improvement is not the right gate. Measure p95 and p99 under peak traffic, not just average load — social media traffic has sharp intraday spikes. Budget the 100ms across stages explicitly (e.g., 15ms user tower, 40ms ANN, 15ms filter, 20ms reranking, 10ms margin) and test whether the new layer fits in the reranking slot under the worst observed traffic conditions, not just median. Consider adaptive degradation: if the reranking layer times out, fall back to dot-product scoring rather than failing the request.
• Causal isolation of retrieval quality: The challenge is that ranking partially compensates for worse inputs — if retrieval degrades, the ranking model rescores from a worse pool but still finds the best available items. Measure retrieval quality directly: recall@K (what fraction of items the ranking model would have chosen are in the retrieved set), and use a forced-exposure experiment where you hold the ranking model fixed and vary only retrieval, rather than measuring end-to-end engagement which conflates the two.
• Guardrails for the experiment: Engagement lift (primary); latency p50/p95/p99 per stage (hard guardrail — automatic rollback if p99 exceeds budget under production traffic); same-day content rate (recency guardrail — a reranker trained on historical engagement may penalize fresh items with sparse signals); creator/content diversity metrics (a learned reranker may amplify popularity bias if not explicitly penalized).
• Recency tension: Engagement-trained models learn that items with engagement history are safer bets — new items have sparse or zero training signal and may get systematically downscored. Monitor same-day content share separately. Mitigate by: adding a freshness feature into the reranker, adding an exploration budget (hold back N% of retrieval slots for recency-boosted items), or using a separate retrieval path for fresh content that bypasses the reranker.
• Common wrong turns: Treating NDCG improvement as sufficient evidence to ship; ignoring latency distributions; not naming the isolation problem; missing that the reranker could systematically harm new content.

The underlying concept

Multi-stage recommendation systems create an attribution problem: the observable outcome (did the user engage?) is produced by a chain of stages, each partially compensating for the previous. Improving retrieval may not show up in engagement if ranking absorbs the change, and degrading retrieval may not hurt engagement if ranking compensates from a smaller pool. Isolating a stage's causal contribution requires either holding all downstream stages constant (forced-exposure design) or measuring a stage-specific metric (recall@K for retrieval) that doesn't depend on downstream behavior. Latency in these systems is also a distribution problem, not an average problem: the tail of the latency distribution during traffic peaks is what determines whether a feature ships, and optimizing for median latency while ignoring p99 is a common failure mode that surfaces only in production.

Source

Derived from SilverTorch: Index as Model — A New Retrieval Paradigm for Recommendation Systems

What the interviewer is probing

This question probes whether candidates understand that bootstrapping is a fundamentally different operational mode than steady-state — the failure modes are distinct and the standard tooling may not apply. It surfaces architecture instincts around dependency management (detecting statically vs. breaking dynamically), the hazard of a system being its own signal consumer (the boomerang pattern), and the maturity to reason about tolerable vs. intolerable failure modes rather than seeking a zero-risk design. The validation-staging section probes risk management and blast-radius thinking.

Strong answer outline

Problem framing - Distinguish two failure modes clearly: deadlock-at-boot from circular deps vs. the boomerang where the signal system takes down its own dispatcher. - Recognize that scale changes the problem: at 50-60x a single fault domain, replica placement means dependencies that 'usually resolve' can all block simultaneously.

Circular dependency design - Static detection first: encode the dependency graph of control plane services and enforce it in CI/CD (analogous to Belljar tests) — fail the build when a new dependency would introduce a cycle. This catches the problem cheaply and early. - But static detection can't catch everything in a fast-moving codebase, so you also need a dynamic break: a minimal 'jumpstart' toolkit — ideally a stripped-down, dependency-free binary or set of primitives — that can bring up the most foundational services without relying on the control plane itself. Think of it as a BIOS: it has no runtime dependencies. - The jumpstart kit must itself be tested in isolation, kept minimal, and owned with high discipline — it is the last resort and cannot fail. - Wrong turn: trying to enumerate a 'safe startup order' manually and encoding it as a config. This is fragile at scale and requires constant maintenance.

Boomerang problem design - Root cause: power-related unavailability events are broadcast globally and consumed by the orchestrator to initiate shutdown — but the orchestrator is also in the blast radius of its own broadcast. - Solution: control plane services exempt themselves from power-related shutdown signals (they treat them as informational, not actionable). They remain up to shepherd the recovery of other services. - Tradeoff: this means control plane services won't self-terminate if there's a real power-related reason they should (e.g., their own rack is unsafe). Mitigate by having a separate, narrower signal for 'this specific node must stop' that the control plane does respect. - Wrong turn: maintaining a static exclusion list of services to skip in the broadcast — operationally fragile, prone to staleness.

Defining tolerable vs. intolerable impact - Intolerable: data loss in storage/DB systems, DC hardware damage, failures that span beyond a single region, or failures that cannot be remediated within a defined MTTR. - Tolerable: transient service errors during boot convergence, bounded staleness in routing tables while the control plane reconverges, a small number of rack failures within a predefined threshold. - The principle: only accept impacts that post-incident remediation can fully reverse within a reasonable time window.

Validation strategy - Incremental blast radius: validate sub-problems (dependency graphs) on new/pre-production regions; then shadow (production-replica) regions; then smallest live production regions; then large production regions. - At each stage, confirm the specific invariant for that stage before proceeding: pre-prod confirms bootstrapping logic; shadow confirms production-fidelity behavior; small prod confirms blast-radius is bounded; large prod confirms storage/DB recovery. - Avoid preemptive actions before the test — draining traffic or pre-warming caches artificially — to represent a true zero-notice event. MTTR in the test mirrors real incident MTTR. - Continuous validation: don't treat this as a one-time milestone. Re-run as the system evolves; embed dependency checks in CI/CD so each deploy re-validates the invariant.

The underlying concept

Bootstrapping a distributed system after a total restart is categorically harder than operating it in steady state because the mechanisms that handle failures during normal operation — service discovery, health signals, orchestration broadcasts — are themselves services that must boot, yet may depend on each other to do so. This creates the circular dependency or 'ouroboros' problem: a true deadlock where no service can start because the one it needs hasn't started yet. The classical solution is a two-tier design: a minimal, dependency-free bootstrap layer (analogous to a computer's BIOS or bootloader) that brings up just enough foundation for the full control plane to self-assemble, combined with static dependency enforcement to prevent cycles from forming in the first place. The 'boomerang' failure mode is a subtler variant of the same class: a system that is both the producer and consumer of a critical signal must explicitly exempt itself from that signal's effects, or it will disable itself at the moment it is most needed. Both problems are examples of a general principle in resilience engineering — the recovery mechanism must be more primitive and more robust than the system it recovers.

Source

Derived from Lights Out, Systems On: Validating Instant Power Loss Readiness

What the interviewer is probing

This question tests whether the candidate can reason from first principles about why a system built for one latency regime fails in another — not just enumerate solutions, but articulate why each architectural layer (proxy, multi-hop metadata, global replication by default) was originally rational and what changed. Strong candidates will surface the tail-latency problem explicitly — that pMax latency, not average latency, is the binding constraint when a slow tail stalls a synchronized cluster — and will reason about trade-offs (fat client vs. proxy, flat schema vs. layered metadata, regional vs. global deployment) rather than treating redesign as obviously correct.

Strong answer outline

• Frame the constraint precisely: The binding constraint is p-max latency, not p50. A single slow metadata hop or a single slow storage node stalls the entire synchronized GPU cluster. This changes the optimization target fundamentally from the original system.
• Diagnose the legacy bottlenecks in order:
• Multi-hop sequential metadata lookups (namelayer → volumeslayer → containerlayer) add latency that compounds; any single slow hop dominates.
• A central data proxy adds a hop on the critical data path and consumes power/compute that competes with GPUs.
• Global-by-default replication means cross-region metadata fetches are on the hot path.
• Metadata redesign: Collapse the layered metadata into a single flat schema in a fast KV store, turning path resolution into O(1) per chunk. Trade-off acknowledged: the layered design offered richer indirection and cross-layer consistency guarantees — losing that requires tighter schema discipline and more careful migration paths.
• Eliminate the proxy / fat client: The API server returns a read plan (chunk → block address mapping) to the client SDK; the SDK streams directly from storage nodes. Gains: lower latency, lower power footprint, higher throughput ceiling. Costs: the client SDK is now complex and must be versioned carefully; failure handling that the proxy absorbed is now client responsibility.
• Regional deployment: Deploy a BLOB-storage stack colocated with GPUs in each AI region so metadata lookups and data reads are intra-region. Trade-off: you give up global-by-default durability and must be explicit about when cross-region replication is needed.
• Hot-spot and spike absorption:
• Cache read plans (path → block address) in a distributed memory tier — cheap to store, dramatically reduces metadata load during concurrent access.
• Use spare GPU-host memory as a distributed data cache for hot data (model weights, frequently reused checkpoints). With an 80%+ hit rate this absorbs most spike I/O before it reaches storage.
• Hedged reads: if a storage node is slow, fire a speculative second request to a replica after a short timeout; use whichever responds first. This directly attacks the tail-latency / single-slow-node problem.
• Dynamic concurrency control on the client SDK: auto-tune parallelism based on application-level congestion signals to prevent egress spikes from cascading into timeouts and retries.
• Common wrong turns: Candidates who treat average latency as the optimization target miss the point entirely. Candidates who add caching without addressing the metadata multi-hop problem fix the hot path but not the cold path. Candidates who eliminate the proxy without acknowledging the fat-client complexity trade-off are hand-waving.

The underlying concept

When a distributed system gates computation on storage, the relevant latency metric shifts from average to maximum — specifically the maximum across all parallel waiters, because a synchronized barrier (like a GPU all-reduce step) moves at the speed of the slowest participant. This means a storage system that is 'fast on average' can still stall every GPU in a large cluster on every training step. The architectural response is to attack tail latency at every layer: flatten multi-hop metadata into a single lookup, eliminate proxies from the data path, absorb hot-spot spikes in in-memory caches before they reach durable storage, and use hedged reads to neutralize slow nodes. Each of these interventions involves a real trade-off — simplicity of schema, client complexity, regional vs. global durability — and the design is only correct if those trade-offs are made consciously against the specific workload's requirements, not inherited from a system built for different physics.

Source

Derived from Meta’s AI Storage Blueprint at Scale

What the interviewer is probing

Whether the candidate recognizes that the celebrity case breaks any uniform design: pure fan-out-on-write melts on a 100M-follower post, pure fan-out-on-read makes every ordinary feed load expensive. The hybrid answer is table stakes; differentiation comes from the details — hot-key mitigation, cache strategy, pagination stability, and treating deletion propagation as a correctness requirement, not an afterthought.

Strong answer outline

• Hybrid fan-out by author class: normal users fan out on write (precomputed timeline lists per follower, cheap because follower counts are small); high-follower accounts fan out on read (their recent posts merged into follower feeds at request time from a heavily cached per-author store).
• Read path: timeline = precomputed list + merge of followed celebrities' recent posts + ranking layer; cache aggressively at the edge with short TTLs for celebrity content since one entry serves millions of readers — that cache is the hot-key absorber.
• Write path: async fan-out via queues with backpressure; idempotent timeline inserts (retries are constant); celebrity posts skip fan-out and instead invalidate/warm the per-author cache.
• Deletions and privacy: tombstones filtered at read time give fast effective deletion even before async cleanup of fanned-out copies completes; privacy changes flip a read-time ACL check rather than rewriting timelines.
• Pagination and dedup: cursor-based pagination over stable IDs (offsets break as items insert), dedup at merge since an item can arrive via multiple paths.
• Multi-region: writes home-regioned with async replication; feeds tolerate seconds of staleness (availability over freshness), but deletions get priority replication; degraded mode serves cached/stale timelines and queues writes.
• Targets stated: p99 read latency, fan-out completion SLO for normal posts, deletion propagation SLO, and cache hit-rate as the capacity lever.

The underlying concept

Fan-out placement is the classic read-amplification vs. write-amplification trade, and skewed follower distributions mean no single placement works — power-law workloads force hybrid designs where the strategy is chosen per key by its skew. The companion principle is that popularity concentrates load onto single keys, and the only real defenses are replication of that key's data (caching) and coalescing of its requests. Feeds also illustrate consistency budgeting: staleness is tolerable for inserts but not for removals, so the design spends its consistency effort asymmetrically.

Source

Distilled Prep canon — curated from Meta's public work on feed and caching infrastructure.

Source: Distilled Prep canon (curated)

What the interviewer is probing

Whether the candidate can put product gains and infrastructure costs in one decision framework instead of treating "engagement up" as self-justifying. Strong candidates convert both sides to comparable units (incremental revenue or retention value vs. serving cost), recognize that latency regressions can silently eat the engagement gain, and generate the middle-path options — distillation, cascades, selective inference — that experienced ML engineers reach for before accepting a 35% cost increase.

Strong answer outline

• Frame the decision as ROI with uncertainty: value of the engagement lift (via revenue-per-engagement estimates or long-term holdouts) vs. annualized serving cost delta, including capacity displacement — what else can't run if this model takes the compute.
• Audit the experiment for hidden costs: per-request latency and tail latency by device/region (slow feeds depress engagement — the model's gain may be net of a latency penalty already, or the penalty may bind only at peak), power/memory on constrained devices, and whether short-term engagement predicts long-term retention here at all.
• Insist on a long-term readout: novelty effects and engagement-quality concerns (is the lift from more low-value impressions?) argue for a weeks-long holdout and guardrails on quality proxies.
• Generate the middle paths: distill the model into a cheaper student; cascade (cheap model for most traffic, expensive one where it changes the decision); selective inference by user value or uncertainty; quantization/hardware placement; ship to segments where ROI clears.
• State a decision rule and its sensitivity: launch fully only if value exceeds cost with margin; otherwise ship a cascade and re-measure.
• Scale-awareness: at extreme scale, 35% inference cost is enormous in absolute dollars — engineering effort to shrink it almost always pays; at small scale, engineer time is the scarcer resource and shipping as-is may be right.

The underlying concept

At scale, ML models are products with unit economics: every ranking request has a marginal cost, and a model change shifts both the revenue and the cost curve. The discipline is marginal analysis — value of the lift minus cost to serve it — plus the recognition that serving budgets couple models to each other through shared capacity. Cascading and distillation exist precisely because prediction value is unevenly distributed across requests: most traffic doesn't need the expensive model, so spending compute only where it changes decisions dominates uniform spending almost everywhere.

Source

Distilled Prep canon — curated from Meta's public work on ranking systems and ML efficiency.

Source: Distilled Prep canon (curated)

What the interviewer is probing

Whether the candidate sees this as two systems with one source of truth — a fast approximate path and a slow correct path — and understands the reconciliation contract between them. Also probing governance instincts: at thousands of experiments, the hard problems shift from throughput to semantics — metric definitions, lineage, and reproducibility — and strong candidates treat those as architecture, not process.

Strong answer outline

• Ingestion: durable event log as the spine; producers write with event IDs and event-time timestamps; schema registry with compatibility-checked evolution so old readers never break.
• Two paths, one truth: streaming path computes windowed aggregates with watermarks for late data and effectively-once semantics via idempotent, keyed upserts — labeled provisional; batch path recomputes from the log with full dedup and late-data inclusion — labeled final and overwriting provisional numbers on a fixed lag (e.g., T+1). Dashboards show provisional; decisions and experiment readouts use final.
• Honest semantics: exactly-once end-to-end is a contract you engineer per sink via idempotency, not a property you inherit; state it that way.
• Metric governance as code: a central metric-definition layer (versioned, code-reviewed) from which both paths compile their computations — one definition of DAU, by construction; lineage tracked from metric to source events.
• Reproducibility: immutable raw events with retention policy, versioned metric definitions, versioned experiment assignment logs, and snapshotted dimension tables — reproducing an analysis means re-running pinned versions against immutable data.
• Experiment scale: precompute per-experiment-arm aggregates rather than per-query scans; cardinality control on dimensions; access controls and query cost budgets so one team's exploration can't starve the platform.

The underlying concept

This is the lambda-architecture idea matured: speed and correctness have different physics (streaming must decide with incomplete information; batch can wait for completeness), so mature platforms run both and define which one owns the truth and when. Watermarks formalize "how long we wait for stragglers," turning late data from a bug into a parameter. And reproducibility is an immutability discipline — recomputing f(data) requires that both f and the data be versioned and frozen — which is why metric governance belongs in the architecture, not in a wiki.

Source

Distilled Prep canon — curated from Meta's public work on data infrastructure and experimentation platforms.

Source: Distilled Prep canon (curated)

What the interviewer is probing

Comfort with missing-data reasoning under operational pressure: can the candidate construct estimates of the unobservable using redundancy (client vs. server logs, sequence numbers, invariant checks), and do they immediately grasp that correlated loss is the dangerous case — peak-time drops disproportionately delete data from the most active users and busiest markets, biasing every metric that matters. Also probing decision hygiene: which conclusions drawn from this data must now be revisited.

Strong answer outline

• Quantify via redundancy: join client-side counters against server-received events; exploit per-session sequence numbers to count gaps; compare against independent systems that observe the same behavior (billing, server-side actions); inject synthetic heartbeat events at known rates as a canary.
• Characterize the bias, not just the rate: loss concentrated at peaks means it correlates with heavy users, dense regions, evening hours, possibly device types with slower upload paths — quantify loss rate by segment and by event type (are purchases dropped as often as scrolls?).
• Triage downstream: rank metrics and recent experiment decisions by exposure — anything comparing across time-of-day, geography, or engagement intensity is suspect; flat-rate loss mostly cancels in ratio metrics but correlated loss does not.
• Communicate with error bars: publish a known-issues banner on affected dashboards, an estimated-impact range per metric, and a list of decisions under re-review — trust survives acknowledged uncertainty better than discovered silence.
• Remediate in two tracks: fix the pipeline (backpressure, client-side buffering and retry, sampling that is explicit rather than accidental) and backfill where client buffers allow.
• Prevent: data-quality SLIs (delivery rate vs. client counters), loss-rate alerts by segment, and end-to-end synthetic event monitoring as a permanent canary.

The underlying concept

Data missing "at random" merely adds noise; data missing in a way correlated with the outcome or the segment (missing-not-at-random) adds bias that no amount of volume fixes. Peak-load loss is almost guaranteed to be the second kind. The estimation strategy is always the same: find a second observation channel with different failure modes and use the disagreement to measure the first channel's loss. And the organizational lesson mirrors the statistical one — metrics infrastructure needs its own observability, because the failure mode of measurement systems is silence, not alarms.

Source

Distilled Prep canon — curated from Meta's public work on data infrastructure and measurement integrity.

Source: Distilled Prep canon (curated)

What the interviewer is probing

Whether the candidate understands the funnel economics that force the retrieval → ranking → re-ranking architecture: compute per candidate rises steeply while candidate count falls, and the design skill is allocating latency and compute across stages. Also probing operational ML maturity — feature freshness, offline/online parity, and safe rollout are where these systems actually fail, and strong candidates raise them unprompted.

Strong answer outline

• Funnel structure with budgets: candidate retrieval (millions → thousands; ANN over embeddings, plus heuristic and graph sources) in tens of ms; ranking (thousands → hundreds; mid-size model, hydrated features) taking the largest share; re-ranking/policy (hundreds → the slate; diversity, integrity filters, business rules) in single-digit ms.
• Feature infrastructure as the real bottleneck: a feature store with an online serving tier, batched hydration, caching of user features per session, and strict timeout-plus-default behavior so one slow feature never breaks the page. Offline/online parity — same feature definitions in training and serving — prevents the classic silent quality bug.
• Freshness tiers: user long-term embeddings refresh daily; item embeddings on ingest; short-term interest via a real-time session feature stream rather than model retraining.
• Cold start: fall back to popularity-by-cohort and content-based retrieval; explicitly widen exploration for new items to gather signal (and accept the small engagement tax it costs).
• Feedback loops and integrity: the system trains on data it created — log propensities, hold out random exploration traffic, and put content integrity gates in the re-rank layer, not post-hoc.
• Rollout: offline metrics gate → shadow scoring for parity checking → small-percentage online experiment with engagement and latency guardrails → staged ramp with automatic rollback triggers.

The underlying concept

Multi-stage ranking is the standard answer to a scaling identity: you cannot afford heavy computation on every candidate, so you spend cheap computation to decide where to spend expensive computation. Each stage trades recall for precision at rising cost per item — the architecture is a compute-allocation curve, and latency budgets are how it's enforced. The second load-bearing idea is that at this scale the model is a minority of the system: feature pipelines, freshness, parity, and rollout safety determine realized quality more than architecture choices, which is why interviewers push hardest there.

Source

Distilled Prep canon — curated from Meta's public work on recommendation infrastructure.

Source: Distilled Prep canon (curated)