Meta

What the interviewer is probing

This question probes staff-level systems thinking: can the candidate frame empirical model scaling as an engineering and planning instrument rather than a research curiosity, reason about where the LLM scaling analogy holds and where it breaks for recommendation systems, and translate a measured curve into actionable infrastructure decisions with explicit uncertainty.

What a strong answer covers

Must establish

• A scaling law is empirically established by training multiple model variants at different compute budgets and fitting a functional form to the quality-versus-compute relationship, so the candidate must describe what experiments are needed and what metric captures quality in a way that is continuous and comparable across scales.
• Balanced scaling across depth, width, and sequence length is necessary because bottlenecking on a single axis produces diminishing returns while the others are underutilized, so the candidate must explain how to diagnose which axis is the current bottleneck.
• The scaling law applies to a specific model family under specific training conditions, and changes to data distribution, feature vocabulary, or architecture can shift or break the curve, so the candidate must name the conditions that would invalidate an extrapolation.

A strong answer adds

• Normalized entropy or an equivalent continuous loss metric is more suitable for establishing a scaling curve than a business metric like conversion rate, because business metrics are noisy, delayed, and not comparable across model sizes; the candidate should explain how to bridge from the scaling-law metric to the business metric.
• Sequence diversity in the training data, including a mix of action types and temporal ranges, affects the quality ceiling the model can reach at any given compute level, so a strong answer treats data composition as a variable in the scaling experiment, not just model size.
• The offline and online stages of a two-stage system have different cost structures and different saturation points, so a mature answer identifies that the scaling law for each stage should be characterized separately and investment allocated to whichever stage has the steeper slope relative to its cost.
• Extrapolating a scaling curve beyond the measured range carries risk, and a strong candidate names the conditions under which extrapolation is reasonable versus when it requires additional ablations.

Exceptional depth

• Connecting the scaling synergy principle, that depth, width, and sequence length must grow in balance, to the practical problem of Pareto-optimal resource allocation: given a fixed compute budget, the optimal allocation across axes can be solved empirically by fitting the joint scaling surface rather than optimizing each axis independently.

Common misses

• Conflates a scaling law with a benchmark result: reports that a larger model outperforms a smaller one without establishing the functional form that enables extrapolation.
• Uses online business metrics directly to fit the scaling curve, ignoring the noise and delay that make them unsuitable for this purpose.
• Assumes the LLM scaling literature transfers directly without questioning the structural differences introduced by sparse ID features and the absence of a dense token vocabulary.
• Treats sequence length as the primary scaling lever in isolation, missing that returns saturate if depth and width are not grown proportionally.
• Does not distinguish between the scaling properties of the offline and online stages, applying a single curve to a system with two very different cost and latency profiles.

The underlying concept

Scaling laws, as studied in the LLM literature, describe a power-law or log-linear relationship between training compute and model quality measured by a continuous loss. They are useful engineering tools because they enable extrapolation: if the relationship is stable, you can estimate the quality of a model you have not yet trained and therefore make principled compute-budget decisions. Establishing such a law requires sweeping multiple model configurations systematically and fitting the observed relationship, not just comparing two points. In recommendation systems the analogy is imperfect because inputs are sparse and heterogeneous rather than dense token sequences, but if the architecture is well-matched to the task, a similar predictable relationship can emerge. The practical value is that scaling decisions become data-driven rather than speculative, and infrastructure investments can be justified with explicit quality-per-compute estimates and confidence intervals on extrapolations.

Source

Derived from From User Sequences to Scaling Laws: A Multi-Stage Architecture for Meta’s Ads Ranking

What the interviewer is probing

This question tests whether the candidate can identify the core latency-versus-model-capacity tension and resolve it through a principled decomposition of the ranking pipeline. Strong candidates demonstrate architecture judgment by separating compute that is user-specific and amortizable from compute that is query-specific and must be online, and they reason honestly about the freshness and context-dependence trade-offs that decomposition introduces.

What a strong answer covers

Must establish

• User-side sequence computation is expensive but largely independent of any specific ad candidate, so it can be precomputed asynchronously and cached, separating it from the latency-critical online path.
• The online ranking stage must still incorporate query-specific signals, meaning the current ad candidate and real-time context, so the cached user representation is an input to, not a replacement for, online scoring.
• Freshness is a genuine trade-off: a cached embedding is always somewhat stale, and the candidate must articulate what staleness costs (missing recent intent signals) and how to bound it (TTL, event-triggered recomputation, or lightweight online updates).

A strong answer adds

• Recognizing that a fixed user embedding is context-blind raises the question of how the online model fuses it with candidate-specific signals, and the candidate should reason about what the online model must learn versus what the offline embedding provides.
• A genuine second path exists beyond pure precompute-and-cache: compress the long user history offline into a compact intermediate representation, then run a lightweight online attention or fusion step over that compression plus real-time signals. This trades some latency budget for materially fresher, more context-aware user representations. Pure caching is favored when user intent is stable and online latency headroom is tight; the hybrid is favored when intent shifts quickly or headroom allows a small additional online pass.
• Scaling the offline model is cheaper than scaling the online model because it is not on the critical serving path, so a strong answer explains how capacity investments should be allocated asymmetrically across the two stages.
• Sequence diversity in the user history matters for representation quality, including a mix of action types rather than only high-signal events, and a strong candidate connects this to what the offline model should be trained on.
• Semantic or content features from pretrained foundation models can substitute for behavioral history in cold-start cases, and a strong answer names this as a complementary signal source with its own freshness and retrieval properties.

Exceptional depth

• Identifying that the online model's job of fusing a cached user vector with ad candidate signals creates a target-awareness problem: the offline embedding was computed without knowledge of the candidate, so the online model must learn to condition the user representation on the target, effectively doing lightweight re-weighting at serving time rather than full recomputation.
• Discussing how a predictable compute-vs-quality scaling curve for the offline model allows infrastructure planning to be data-driven rather than heuristic, analogous to how LLM scaling laws guide training budget allocation.

Common misses

• Proposes caching without addressing staleness, treating the cached embedding as always fresh even though user intent changes between cache writes.
• Makes the online stage re-run the full sequence model over the cached embedding rather than treating the embedding as a fixed input, negating the latency benefit of the two-stage split.
• Treats the two stages as fully independent models trained separately on different objectives, missing that joint or distillation-style training is needed to prevent lossy knowledge transfer between stages.
• Ignores the cold-start user entirely, leaving a gap in coverage that matters disproportionately for growth surfaces.
• Scales both stages equally when budget is constrained, rather than recognizing that offline capacity is cheaper per quality point than online capacity.

The underlying concept

Transformer-based sequence models scale in quality with depth and context length, but their compute cost is superlinear in sequence length, making inline inference infeasible under tight latency budgets. The resolution is to recognize that user-side computation and query-side computation have different amortization properties: a user's behavioral embedding changes slowly relative to the rate at which the system scores ad candidates, so the expensive user-side work can be done asynchronously and cached. The online path then handles only the fast, candidate-conditioned fusion. This is a specific instance of the general principle of separating read-heavy from write-heavy computation paths in large-scale serving systems, applied to the ML model itself rather than to storage. The residual design challenge is that a context-free user embedding loses the ability to express intent conditional on what is being scored, which is why the online stage must retain some expressive capacity for candidate-aware re-weighting.

Source

Derived from From User Sequences to Scaling Laws: A Multi-Stage Architecture for Meta’s Ads Ranking

What the interviewer is probing

The question tests whether the candidate can decompose a compound efficiency problem into independent levers, specifically per-GPU compute efficiency versus scaling efficiency across GPUs, before proposing solutions. It surfaces causal reasoning (which properties of the workload cause which inefficiencies), architecture judgment (parallelism strategy matched to topology and parameter type), and decision quality (trade-offs between precision, memory, and communication). Strong candidates do not immediately jump to adding hardware or switching frameworks; they frame the problem first.

What a strong answer covers

Must establish

• MFU must be decomposed into two separable factors before any fix is proposed: per-GPU compute efficiency (how close each GPU runs to its hardware roofline) and scaling efficiency (how much of that single-GPU performance survives when distributing across thousands of GPUs). Fixes for one factor do not automatically fix the other, and conflating them leads to optimizing the wrong thing.
• Jagged input sequences break the tiling and parallelization assumptions of standard attention kernels, which expect uniform sequence lengths. The candidate must identify that naive padding to max length wastes substantial compute, and must propose and defend a strategy, such as length-aware batching and packing, or kernels that operate directly on variable-length data, that reduces that waste, evaluated against the trade-offs it introduces.
• The dense and sparse components of the model have fundamentally different communication and compute profiles, and they require distinct parallelism strategies. Applying a single uniform parallelism scheme across both components will leave one or both under-served.

A strong answer adds

• Frames the scaling efficiency problem using the E2E latency equation: total step time is determined by the slowest rank, which is the max of compute time and exposed communication time. Near-linear scaling requires compute to dominate communication, communication to be hidden behind compute without resource contention, minimal activation recomputation from memory pressure, and balanced load across ranks. The candidate identifies which of these this workload violates and why.
• Recognizes that the asymmetric attention patterns in this workload (long queries with short key-value, short queries with long key-value, and self-attention over long sequences) each have different kernel efficiency characteristics, and that a single general-purpose attention kernel handles none of them well.
• Matches parallelism dimensions to network topology tiers: highest-bandwidth links (intra-node) should carry the most communication-heavy operations, while lower-bandwidth cross-zone links should carry only small, gradient-fraction messages. A flat parallelism scheme that ignores topology will saturate the slow links first.
• For low-precision training, identifies quantization overhead as a distinct problem from numerical stability. Overhead comes from extra casting and data movement that can offset Tensor Core speedup; stability comes from outliers, rounding bias, and gradient sensitivity. The candidate proposes mitigations for both independently.
• Proposes a mixed-precision strategy where ultra-low precision is applied where the GEMM size makes the speedup large and the numerical sensitivity is low, and falls back to higher precision for layers or operations that are more sensitive, rather than applying a single precision level uniformly.

Exceptional depth

• Identifies that quantization overhead for sharded parameters can be partially amortized by quantizing each rank's local shard before the all-gather collective rather than after, so the all-gather itself communicates lower-precision payloads, reducing communication volume and latency simultaneously.
• Recognizes that jagged load imbalance across ranks is data-driven and varies per step, making static load balancing insufficient. Sequence-length-aware batch construction or dynamic assignment is needed to prevent straggler ranks from setting the step latency.

Common misses

• Jumps to proposing a specific parallelism scheme or kernel optimization without first decomposing MFU into per-GPU compute efficiency and scaling efficiency, leaving the diagnosis incomplete.
• Treats padding as a minor inefficiency rather than a structural problem. For jagged recommendation workloads, padding to max sequence length can waste a significant fraction of compute, and the fix requires rethinking how the kernel tiles and parallelizes work.
• Applies a single parallelism strategy uniformly to both sparse and dense parameters, ignoring that sparse embeddings and dense layers have different all-to-all versus all-gather communication patterns and different sensitivity to group size.
• Proposes low-precision training as a straightforward throughput multiplier without addressing quantization overhead or numerical stability, missing that naively applying low precision to this model's ranking predictions causes quality regression because the scores are sensitive to small numeric shifts.
• Ignores memory pressure as a scaling efficiency lever. Long sequences generate large activations, and when memory is exhausted, activation recomputation is forced, which adds compute that erodes the scaling ratio independently of communication overhead.

The underlying concept

Model FLOPs Utilization measures how close a training workload runs to the theoretical peak throughput of the hardware. For recommendation models, two distinct forces suppress it. First, per-GPU compute efficiency is degraded by workload properties that break kernel assumptions: jagged sequences waste compute through padding, asymmetric attention shapes leave pipeline stages underutilized, and memory-bound operations stall Tensor Cores. Second, scaling efficiency is degraded by the gap between single-GPU throughput and multi-GPU throughput, driven by communication overhead, load imbalance across ranks, and activation recomputation forced by memory pressure. These two factors multiply to produce end-to-end MFU, so optimizing only one leaves the other as the binding constraint. Recommendation models are particularly hard because their hybrid sparse-dense architecture combines the communication patterns of embedding tables with the compute patterns of dense transformers, and standard LLM infrastructure assumes neither.

Source

Derived from GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model

What the interviewer is probing

This question probes architecture thinking across the kernel-userspace boundary: does the candidate understand why a general-purpose scheduler is a poor fit for latency-sensitive fan-out workloads, and can they design a partitioning policy with explicit trade-offs around pool sizing, cache affinity, and failure modes? The deployment model — policy as a reloadable user-space artifact rather than a kernel patch — tests whether the candidate sees iteration speed as an architectural constraint worth designing around. The follow-up on application-scheduler interfaces probes systems thinking around trust boundaries and interface design: who gets to signal priority, and what prevents that interface from becoming a source of instability.

What a strong answer covers

• Diagnose first, design second. Before designing a custom policy, characterize the regression: use scheduler tracing (e.g., perf sched, BPF tracepoints on context switches, runqueue wait times) to identify whether the regression is excess preemption, poor cache affinity, priority inversion, or NUMA thrashing. The fix is different for each.
• Model the workload structure. A request-serving fan-out has a clear thread taxonomy: request-handling threads on the critical path (latency-sensitive, short-burst), background threads (compaction, logging, async I/O — throughput-oriented). A general-purpose scheduler cannot distinguish these; a custom policy can encode this knowledge directly.
• CPU pool partitioning. Soft-partition CPUs into a latency pool and a background pool. 'Soft' is important — hard partitioning wastes capacity during imbalance; soft partitioning allows the pools to borrow from each other under load, with a priority bias rather than a hard boundary. Pool sizing starts from observed utilization ratios and is adjusted dynamically via load-based heuristics.
• Cache locality as a first-class design goal. Scheduling related threads to the same physical cores over time improves L3 hit rates and reduces DRAM accesses — critical when the fan-out brings together many threads working on shared data structures. NUMA-aware placement (keep threads and their memory on the same socket) is a follow-on optimization.
• Deployment model determines iteration speed. Encoding the policy as a reloadable user-space program (BPF program loaded by a user-space daemon) decouples the scheduling improvement cycle from the kernel release cycle. Rolling out a change is a daemon restart, not a kernel patch with months of validation. This is an architectural choice, not an implementation detail — it is what makes continuous optimization tractable.
• Experiment framework for live scheduling changes. A/B testing two scheduling policies on a live fleet requires: (a) host-level assignment (not request-level, to avoid mixing policies on the same host), (b) a grace period on policy switchover to let in-flight requests drain, (c) a rollback trigger based on p99 latency and timeout error rate guardrails, (d) metrics stratified by traffic mix to catch load-dependent effects.
• Common wrong turns: designing for average latency rather than tail latency (the business impact lives at p99+); using hard CPU partitioning and starving one pool; building the policy into the kernel binary rather than user space (kills iteration speed); skipping the diagnosis step and tuning a policy against the wrong bottleneck.
• Application-scheduler interface (the hardest part). Allowing the application to signal thread importance (e.g., 'this thread is serving a high-value request') is powerful but risky. Trust model must be explicit: the scheduler must treat these signals as hints, not hard constraints, with a cap on the boost any thread can claim. Abuse vectors — all threads claiming high priority — must be rate-limited or quota-bounded. The interface should be instrumented so gaming is detectable.

The underlying concept

General-purpose CPU schedulers like CFS and EEVDF optimize for fairness and throughput across arbitrary workloads — they are deliberately ignorant of application semantics. For most workloads this is fine, but for latency-sensitive request-serving systems the cost of that ignorance is measurable: threads on the critical path compete on equal footing with background work, cache affinity is not preserved across scheduling decisions, and small changes in the scheduler's virtual-time accounting can shift tail latency by tens of milliseconds. The key insight behind workload-aware scheduling is that the application already has the information the scheduler lacks — which threads are on the critical path, which are background, which requests are high-value — and a custom scheduler can use that information to make better decisions than any general-purpose heuristic. The deeper architectural point is that where the policy lives determines how fast you can improve it: a policy encoded in a kernel module requires a full kernel release cycle to ship; a policy encoded in a reloadable BPF program ships in days. At the scale where a single percentage point of latency improvement translates to megawatts of power savings and measurable revenue lift, that iteration speed difference is itself a strategic asset.

Source

Derived from Modernizing the Meta Ads Service With an Open-Source Kernel Scheduler

What the interviewer is probing

This question tests whether the candidate can translate an OS-level abstraction — hierarchical memory with demand paging and prefetching — into a distributed systems design, and whether they can reason about the real optimization target (researcher iteration speed) rather than the obvious one (data-loading throughput). Strong candidates will recognize that the write-once, read-many pattern of training datasets makes aggressive caching safe, will design a prefetch API that hides cross-region latency behind compute work, and will explicitly frame the trade-off between performance predictability (full pre-copy) and iteration speed (on-demand hydration) rather than claiming the new design is strictly better.

What a strong answer covers

• Frame the actual problem: For large training runs, pre-copy is fine — it amortizes over weeks. The problem is small iterative experiments where researchers spend more time waiting on ingestion than on the model. The goal is to cut the setup time from hours to minutes for this workload, accepting occasional I/O stalls that wouldn't occur with a full pre-copy.
• Core insight — treat storage as a tiered cache:
• L1: GPU-host memory (fastest, smallest — the current training batch)
• L2: Local flash on the GPU host (warm data across a few batches)
• L3: Regional disaggregated flash storage, co-located with GPUs (data for the current job's duration)
• Source of truth: global HDD-backed object store (remote, durable, cheap)
• This mirrors OS demand paging: data is pulled toward compute on first access and cached for reuse.
• Demand hydration: On a cache miss at L3, the client transparently fetches from the remote source of truth. First-access latency will be higher (cross-region), so the design must hide this.
• Prefetching to hide latency:
• Dataloader prefetch: the dataloader pre-loads the next batch into memory while processing the current one — standard overlap of I/O and compute.
• Explicit deep prefetch API: the dataloader triggers background hydration of the next several minutes of data into L3 before it's needed, prewarms metadata cache. This is what allows the first real training step to start before all data is local.
• Multi-epoch reuse: Training iterates over the same data multiple times. The L3 cache should hold data for the duration of the training cycle (TTL tied to job lifetime or a configured window), not evict after first use. LRU within that window handles capacity pressure when multiple jobs share the cache.
• Consistency: Training datasets are write-once — once a researcher publishes a dataset version, it doesn't change during the training run. Cache invalidation is therefore not a hot path; it happens on explicit dataset version update. Version identifiers in the cache key prevent stale reads across experiments.
• Failure mode: If the remote source of truth is unreachable and L3 has a miss, the training step stalls. For iterative small jobs this is acceptable; for production runs you'd want the full pre-copy. The system should expose a metric for cross-region fetch rate so researchers can identify when they've crossed the threshold where pre-copying would have been cheaper.
• Trade-offs to name explicitly: First-epoch performance will be lower than pre-copy (cold L3). Subsequent epochs are fast (warm L3). Power and flash capacity are consumed in the GPU region. The design is strictly worse than pre-copy on throughput predictability, and strictly better on time-to-first-step.
• Common wrong turns: Candidates who propose caching without a prefetch strategy will have a cold-start problem on the first epoch that stalls GPUs. Candidates who don't discuss eviction policies will overrun flash capacity. Candidates who claim this design is better in all cases have not thought carefully about the large-run use case.

The underlying concept

Operating systems have solved the problem of 'data is far away but compute needs it now' through hierarchical caching with demand paging and prefetching: bring data close to compute on first use, keep it there for reuse, and hide the fetch latency behind ongoing computation. Distributed training storage can borrow this model directly — the key enabling property is that training datasets are write-once and read-many, which makes aggressive caching safe (no cache coherence problem during a run) and makes prefetching tractable (access patterns are sequential and predictable). The architectural insight is that 'ingestion' and 'training' don't have to be sequential phases; demand hydration with deep prefetching collapses them into a single overlapped operation, converting an hours-long blocking step into a background process invisible to the researcher.

Source

Derived from Meta’s AI Storage Blueprint at Scale

What the interviewer is probing

This question tests whether the candidate can reason from first principles about why a system built for one latency regime fails in another — not just enumerate solutions, but articulate why each architectural layer (proxy, multi-hop metadata, global replication by default) was originally rational and what changed. Strong candidates will surface the tail-latency problem explicitly — that pMax latency, not average latency, is the binding constraint when a slow tail stalls a synchronized cluster — and will reason about trade-offs (fat client vs. proxy, flat schema vs. layered metadata, regional vs. global deployment) rather than treating redesign as obviously correct.

What a strong answer covers

• Frame the constraint precisely: The binding constraint is p-max latency, not p50. A single slow metadata hop or a single slow storage node stalls the entire synchronized GPU cluster. This changes the optimization target fundamentally from the original system.
• Diagnose the legacy bottlenecks in order:
• Multi-hop sequential metadata lookups (namelayer → volumeslayer → containerlayer) add latency that compounds; any single slow hop dominates.
• A central data proxy adds a hop on the critical data path and consumes power/compute that competes with GPUs.
• Global-by-default replication means cross-region metadata fetches are on the hot path.
• Metadata redesign: Collapse the layered metadata into a single flat schema in a fast KV store, turning path resolution into O(1) per chunk. Trade-off acknowledged: the layered design offered richer indirection and cross-layer consistency guarantees — losing that requires tighter schema discipline and more careful migration paths.
• Eliminate the proxy / fat client: The API server returns a read plan (chunk → block address mapping) to the client SDK; the SDK streams directly from storage nodes. Gains: lower latency, lower power footprint, higher throughput ceiling. Costs: the client SDK is now complex and must be versioned carefully; failure handling that the proxy absorbed is now client responsibility.
• Regional deployment: Deploy a BLOB-storage stack colocated with GPUs in each AI region so metadata lookups and data reads are intra-region. Trade-off: you give up global-by-default durability and must be explicit about when cross-region replication is needed.
• Hot-spot and spike absorption:
• Cache read plans (path → block address) in a distributed memory tier — cheap to store, dramatically reduces metadata load during concurrent access.
• Use spare GPU-host memory as a distributed data cache for hot data (model weights, frequently reused checkpoints). With an 80%+ hit rate this absorbs most spike I/O before it reaches storage.
• Hedged reads: if a storage node is slow, fire a speculative second request to a replica after a short timeout; use whichever responds first. This directly attacks the tail-latency / single-slow-node problem.
• Dynamic concurrency control on the client SDK: auto-tune parallelism based on application-level congestion signals to prevent egress spikes from cascading into timeouts and retries.
• Common wrong turns: Candidates who treat average latency as the optimization target miss the point entirely. Candidates who add caching without addressing the metadata multi-hop problem fix the hot path but not the cold path. Candidates who eliminate the proxy without acknowledging the fat-client complexity trade-off are hand-waving.

The underlying concept

When a distributed system gates computation on storage, the relevant latency metric shifts from average to maximum — specifically the maximum across all parallel waiters, because a synchronized barrier (like a GPU all-reduce step) moves at the speed of the slowest participant. This means a storage system that is 'fast on average' can still stall every GPU in a large cluster on every training step. The architectural response is to attack tail latency at every layer: flatten multi-hop metadata into a single lookup, eliminate proxies from the data path, absorb hot-spot spikes in in-memory caches before they reach durable storage, and use hedged reads to neutralize slow nodes. Each of these interventions involves a real trade-off — simplicity of schema, client complexity, regional vs. global durability — and the design is only correct if those trade-offs are made consciously against the specific workload's requirements, not inherited from a system built for different physics.

Source

Derived from Meta’s AI Storage Blueprint at Scale

What the interviewer is probing

This question probes whether candidates understand that bootstrapping is a fundamentally different operational mode than steady-state — the failure modes are distinct and the standard tooling may not apply. It surfaces architecture instincts around dependency management (detecting statically vs. breaking dynamically), the hazard of a system being its own signal consumer (the boomerang pattern), and the maturity to reason about tolerable vs. intolerable failure modes rather than seeking a zero-risk design. The validation-staging section probes risk management and blast-radius thinking.

What a strong answer covers

Problem framing - Distinguish two failure modes clearly: deadlock-at-boot from circular deps vs. the boomerang where the signal system takes down its own dispatcher. - Recognize that scale changes the problem: at 50-60x a single fault domain, replica placement means dependencies that 'usually resolve' can all block simultaneously.

Circular dependency design - Static detection first: encode the dependency graph of control plane services and enforce it in CI/CD (analogous to Belljar tests) — fail the build when a new dependency would introduce a cycle. This catches the problem cheaply and early. - But static detection can't catch everything in a fast-moving codebase, so you also need a dynamic break: a minimal 'jumpstart' toolkit — ideally a stripped-down, dependency-free binary or set of primitives — that can bring up the most foundational services without relying on the control plane itself. Think of it as a BIOS: it has no runtime dependencies. - The jumpstart kit must itself be tested in isolation, kept minimal, and owned with high discipline — it is the last resort and cannot fail. - Wrong turn: trying to enumerate a 'safe startup order' manually and encoding it as a config. This is fragile at scale and requires constant maintenance.

Boomerang problem design - Root cause: power-related unavailability events are broadcast globally and consumed by the orchestrator to initiate shutdown — but the orchestrator is also in the blast radius of its own broadcast. - Solution: control plane services exempt themselves from power-related shutdown signals (they treat them as informational, not actionable). They remain up to shepherd the recovery of other services. - Tradeoff: this means control plane services won't self-terminate if there's a real power-related reason they should (e.g., their own rack is unsafe). Mitigate by having a separate, narrower signal for 'this specific node must stop' that the control plane does respect. - Wrong turn: maintaining a static exclusion list of services to skip in the broadcast — operationally fragile, prone to staleness.

Defining tolerable vs. intolerable impact - Intolerable: data loss in storage/DB systems, DC hardware damage, failures that span beyond a single region, or failures that cannot be remediated within a defined MTTR. - Tolerable: transient service errors during boot convergence, bounded staleness in routing tables while the control plane reconverges, a small number of rack failures within a predefined threshold. - The principle: only accept impacts that post-incident remediation can fully reverse within a reasonable time window.

Validation strategy - Incremental blast radius: validate sub-problems (dependency graphs) on new/pre-production regions; then shadow (production-replica) regions; then smallest live production regions; then large production regions. - At each stage, confirm the specific invariant for that stage before proceeding: pre-prod confirms bootstrapping logic; shadow confirms production-fidelity behavior; small prod confirms blast-radius is bounded; large prod confirms storage/DB recovery. - Avoid preemptive actions before the test — draining traffic or pre-warming caches artificially — to represent a true zero-notice event. MTTR in the test mirrors real incident MTTR. - Continuous validation: don't treat this as a one-time milestone. Re-run as the system evolves; embed dependency checks in CI/CD so each deploy re-validates the invariant.

The underlying concept

Bootstrapping a distributed system after a total restart is categorically harder than operating it in steady state because the mechanisms that handle failures during normal operation — service discovery, health signals, orchestration broadcasts — are themselves services that must boot, yet may depend on each other to do so. This creates the circular dependency or 'ouroboros' problem: a true deadlock where no service can start because the one it needs hasn't started yet. The classical solution is a two-tier design: a minimal, dependency-free bootstrap layer (analogous to a computer's BIOS or bootloader) that brings up just enough foundation for the full control plane to self-assemble, combined with static dependency enforcement to prevent cycles from forming in the first place. The 'boomerang' failure mode is a subtler variant of the same class: a system that is both the producer and consumer of a critical signal must explicitly exempt itself from that signal's effects, or it will disable itself at the moment it is most needed. Both problems are examples of a general principle in resilience engineering — the recovery mechanism must be more primitive and more robust than the system it recovers.

Source

Derived from Lights Out, Systems On: Validating Instant Power Loss Readiness

What the interviewer is probing

This question tests metric judgment and causal reasoning in a multi-stage ML system: candidates must reason about offline-online gaps, latency distributions (not just means), the difficulty of attributing quality to a stage when downstream components compensate, and second-order effects like content fairness and recency that engagement metrics don't directly capture. Strong candidates treat latency as a budget with a distribution, not a single number, and raise the isolation problem unprompted.

What a strong answer covers

• Decision framing: The question isn't just "does NDCG go up" — it's whether the improvement is real (not an artifact of offline evaluation), large enough to justify latency cost, and durable enough to survive production conditions. Frame the decision across three dimensions: quality impact, latency risk, and second-order effects.
• Offline-online gap diagnosis: NDCG on a held-out set measures whether the reranker correctly orders items that were eventually shown and interacted with — but it doesn't capture exploration (items never retrieved under the old system that would have been great), position bias in the training labels, or distribution shift between the offline eval set and live traffic. Common causes of a flat online experiment: training labels reflect the old retrieval distribution, so the reranker learned to score items the old system already selected well; or the downstream ranking model compensates for changes in retrieval order, absorbing the signal.
• Latency as a distribution, not a point: p50 improvement is not the right gate. Measure p95 and p99 under peak traffic, not just average load — social media traffic has sharp intraday spikes. Budget the 100ms across stages explicitly (e.g., 15ms user tower, 40ms ANN, 15ms filter, 20ms reranking, 10ms margin) and test whether the new layer fits in the reranking slot under the worst observed traffic conditions, not just median. Consider adaptive degradation: if the reranking layer times out, fall back to dot-product scoring rather than failing the request.
• Causal isolation of retrieval quality: The challenge is that ranking partially compensates for worse inputs — if retrieval degrades, the ranking model rescores from a worse pool but still finds the best available items. Measure retrieval quality directly: recall@K (what fraction of items the ranking model would have chosen are in the retrieved set), and use a forced-exposure experiment where you hold the ranking model fixed and vary only retrieval, rather than measuring end-to-end engagement which conflates the two.
• Guardrails for the experiment: Engagement lift (primary); latency p50/p95/p99 per stage (hard guardrail — automatic rollback if p99 exceeds budget under production traffic); same-day content rate (recency guardrail — a reranker trained on historical engagement may penalize fresh items with sparse signals); creator/content diversity metrics (a learned reranker may amplify popularity bias if not explicitly penalized).
• Recency tension: Engagement-trained models learn that items with engagement history are safer bets — new items have sparse or zero training signal and may get systematically downscored. Monitor same-day content share separately. Mitigate by: adding a freshness feature into the reranker, adding an exploration budget (hold back N% of retrieval slots for recency-boosted items), or using a separate retrieval path for fresh content that bypasses the reranker.
• Common wrong turns: Treating NDCG improvement as sufficient evidence to ship; ignoring latency distributions; not naming the isolation problem; missing that the reranker could systematically harm new content.

The underlying concept

Multi-stage recommendation systems create an attribution problem: the observable outcome (did the user engage?) is produced by a chain of stages, each partially compensating for the previous. Improving retrieval may not show up in engagement if ranking absorbs the change, and degrading retrieval may not hurt engagement if ranking compensates from a smaller pool. Isolating a stage's causal contribution requires either holding all downstream stages constant (forced-exposure design) or measuring a stage-specific metric (recall@K for retrieval) that doesn't depend on downstream behavior. Latency in these systems is also a distribution problem, not an average problem: the tail of the latency distribution during traffic peaks is what determines whether a feature ships, and optimizing for median latency while ignoring p99 is a common failure mode that surfaces only in production.

Source

Derived from SilverTorch: Index as Model — A New Retrieval Paradigm for Recommendation Systems

What the interviewer is probing

Whether the candidate recognizes that the celebrity case breaks any uniform design: pure fan-out-on-write melts on a 100M-follower post, pure fan-out-on-read makes every ordinary feed load expensive. The hybrid answer is table stakes; differentiation comes from the details — hot-key mitigation, cache strategy, pagination stability, and treating deletion propagation as a correctness requirement, not an afterthought.

What a strong answer covers

Must establish

• Hybrid fan-out by author class: normal users fan out on write (precomputed timeline lists per follower, cheap because follower counts are small); high-follower accounts fan out on read (their recent posts merged into follower feeds at request time from a heavily cached per-author store).
• Read path constructs timeline as precomputed list plus merge of followed celebrities' recent posts plus ranking layer; celebrity content is cached aggressively at the edge with short TTLs because one cache entry serves millions of readers, making that cache the primary hot-key absorber.
• Deletions use tombstones filtered at read time, giving fast effective deletion before async cleanup of fanned-out copies completes; privacy changes flip a read-time ACL check rather than rewriting timelines.

A strong answer adds

• Write path uses async fan-out via queues with backpressure, with idempotent timeline inserts so retries are safe.
• Celebrity posts skip fan-out entirely and instead invalidate or warm the per-author cache.
• Cursor-based pagination over stable IDs, because offset-based pagination breaks as items insert.
• Deduplication at merge time, because an item can arrive via multiple paths.
• Multi-region writes are home-regioned with async replication; feeds tolerate seconds of staleness, trading availability over freshness.
• Deletions receive priority replication across regions as a correctness requirement, not a best-effort concern.
• Degraded mode serves cached or stale timelines and queues writes during a regional outage.
• Targets stated explicitly: p99 read latency, fan-out completion SLO for normal posts, deletion propagation SLO, and cache hit-rate as the primary capacity lever.

Common misses

• Proposes a uniform fan-out strategy (pure fan-out-on-write or pure fan-out-on-read) without recognizing that skewed follower distributions break either extreme.
• Does not identify the per-author celebrity cache as the hot-key absorber, leaving no concrete mechanism for the 100M-follower post case.
• Treats deletion propagation as an afterthought rather than a correctness requirement with its own priority replication path.
• Uses offset-based rather than cursor-based pagination, missing that offsets break as new items insert.
• Does not address deduplication at merge time, leaving the feed undefined when an item arrives via multiple paths.

The underlying concept

Fan-out placement is the classic read-amplification vs. write-amplification trade, and skewed follower distributions mean no single placement works — power-law workloads force hybrid designs where the strategy is chosen per key by its skew. The companion principle is that popularity concentrates load onto single keys, and the only real defenses are replication of that key's data (caching) and coalescing of its requests. Feeds also illustrate consistency budgeting: staleness is tolerable for inserts but not for removals, so the design spends its consistency effort asymmetrically.

Source

Distilled Prep canon — curated from Meta's public work on feed and caching infrastructure.

Source: Meta engineering blog

What the interviewer is probing

Whether the candidate can put product gains and infrastructure costs in one decision framework instead of treating "engagement up" as self-justifying. Strong candidates convert both sides to comparable units (incremental revenue or retention value vs. serving cost), recognize that latency regressions can silently eat the engagement gain, and generate the middle-path options — distillation, cascades, selective inference — that experienced ML engineers reach for before accepting a 35% cost increase.

What a strong answer covers

Must establish

• Frame the decision as ROI with uncertainty: value of the engagement lift, estimated via revenue-per-engagement proxies or long-term holdouts, versus the annualized serving cost delta, including capacity displacement (what else cannot run if this model takes the compute).
• Audit the experiment for hidden latency costs: per-request and tail latency by device and region, because slow feeds depress engagement and the model's measured gain may already be net of a latency penalty, or that penalty may bind only at peak load.
• State a decision rule and its sensitivity: launch fully only if value exceeds cost with margin; otherwise pursue a middle path and re-measure.

A strong answer adds

• Insist on a long-term readout using a weeks-long holdout, because novelty effects can inflate short-term engagement numbers and the lift may come from more low-value impressions rather than genuine retention gains.
• Add guardrails on quality proxies alongside raw engagement, so the experiment can distinguish engagement-quality improvement from volume inflation.
• Generate the middle paths before accepting the 35% cost increase: distill the model into a cheaper student; cascade by routing most traffic to the cheap model and the expensive model only where it would change the decision; apply selective inference by user value or prediction uncertainty; explore quantization and hardware placement; or ship only to segments where ROI clearly clears.
• Apply scale-awareness explicitly: at extreme scale a 35% inference cost increase is enormous in absolute dollars and engineering effort to shrink it almost always pays; at small scale engineer time is the scarcer resource and shipping as-is may be the right call.
• Account for power and memory on constrained devices as part of the cost audit, not only server-side budget.

Common misses

• Treats 'engagement up' as self-justifying without converting both the engagement gain and the serving cost to comparable units.
• Ignores capacity displacement, missing that the compute taken by this model prevents other workloads from running.
• Does not check whether the latency increase silently eats part or all of the engagement gain, especially at peak or on constrained devices.
• Accepts only the binary of launch-as-is or do-not-launch, never reaching for distillation, cascades, or selective inference as intermediate options.
• Relies solely on the short-term experiment readout without arguing for a weeks-long holdout to separate novelty effects from durable retention value.

The underlying concept

At scale, ML models are products with unit economics: every ranking request has a marginal cost, and a model change shifts both the revenue and the cost curve. The discipline is marginal analysis — value of the lift minus cost to serve it — plus the recognition that serving budgets couple models to each other through shared capacity. Cascading and distillation exist precisely because prediction value is unevenly distributed across requests: most traffic doesn't need the expensive model, so spending compute only where it changes decisions dominates uniform spending almost everywhere.

Source

Distilled Prep canon — curated from Meta's public work on ranking systems and ML efficiency.

Source: Meta engineering blog

What the interviewer is probing

Whether the candidate sees this as two systems with one source of truth — a fast approximate path and a slow correct path — and understands the reconciliation contract between them. Also probing governance instincts: at thousands of experiments, the hard problems shift from throughput to semantics — metric definitions, lineage, and reproducibility — and strong candidates treat those as architecture, not process.

What a strong answer covers

Must establish

• Two paths, one truth: a streaming path producing provisional windowed aggregates and a batch path recomputing final numbers from the raw log, with batch overwriting provisional on a fixed lag (e.g., T+1); dashboards consume provisional, experiment decisions consume final.
• Durable event log as the spine, with producers writing event IDs and event-time timestamps, so both paths derive from a single authoritative source.
• Metric governance as code: a central, versioned, code-reviewed metric-definition layer from which both paths compile their computations, giving one definition of any metric (such as DAU) by construction.

A strong answer adds

• Streaming path delivers effectively-once semantics via idempotent, keyed upserts, and uses watermarks to formalize how long the system waits for late-arriving events, turning late data from a bug into a parameter.
• Exactly-once end-to-end is a contract engineered per sink through idempotency, not a property inherited from the framework; the answer must state it that way.
• Batch path performs full dedup and includes late-arriving data that the streaming window already closed, which is why batch numbers are authoritative and provisional numbers are not.
• Lineage is tracked from each metric back to its source events, so the provenance of any number is queryable.
• Reproducibility rests on four pillars held together: immutable raw events with a defined retention policy, versioned metric definitions, versioned experiment assignment logs, and snapshotted dimension tables; reproducing an analysis means re-running pinned versions against immutable data.
• Schema registry with compatibility-checked evolution so that old readers never break as schemas evolve.
• Experiment scale is managed by precomputing per-experiment-arm aggregates rather than serving per-query scans, and by applying cardinality controls on dimensions.
• Access controls and query cost budgets prevent one team's exploration from starving the platform across thousands of concurrent experiments.

Common misses

• Treats the streaming and batch paths as alternatives rather than complementary layers with a defined reconciliation contract, missing that provisional and final labels are architectural commitments.
• Does not frame late data, duplication, and out-of-order arrival as parameters with explicit semantics (watermarks, idempotent upserts, full dedup in batch) and instead treats them as edge cases to be handled ad hoc.
• Omits metric governance entirely, leaving metric definitions in documentation or team convention rather than in a versioned, code-reviewed layer that both paths compile from, which is how the platform prevents conflicting DAU numbers.
• Treats reproducibility as a query feature rather than an immutability discipline, failing to name all four required pillars: immutable raw events, versioned metric definitions, versioned assignment logs, and snapshotted dimension tables.
• Claims exactly-once semantics as an inherited property of the streaming framework rather than a guarantee engineered per sink through idempotency.

The underlying concept

This is the lambda-architecture idea matured: speed and correctness have different physics (streaming must decide with incomplete information; batch can wait for completeness), so mature platforms run both and define which one owns the truth and when. Watermarks formalize "how long we wait for stragglers," turning late data from a bug into a parameter. And reproducibility is an immutability discipline — recomputing f(data) requires that both f and the data be versioned and frozen — which is why metric governance belongs in the architecture, not in a wiki.

Source

Distilled Prep canon — curated from Meta's public work on data infrastructure and experimentation platforms.

Source: Meta engineering blog

What the interviewer is probing

Comfort with missing-data reasoning under operational pressure: can the candidate construct estimates of the unobservable using redundancy (client vs. server logs, sequence numbers, invariant checks), and do they immediately grasp that correlated loss is the dangerous case — peak-time drops disproportionately delete data from the most active users and busiest markets, biasing every metric that matters. Also probing decision hygiene: which conclusions drawn from this data must now be revisited.

What a strong answer covers

Must establish

• Quantify the missing fraction using redundancy: join client-side counters against server-received events, exploit per-session sequence numbers to count gaps, and compare against independent systems that observe the same behavior such as billing or server-side actions.
• Characterize the bias, not just the rate: loss concentrated at peaks correlates with heavy users, dense regions, evening hours, and possibly device types with slower upload paths, so correlated loss does not cancel in ratio metrics the way flat-rate loss does.
• Triage downstream metrics and recent experiment decisions by exposure: anything comparing across time-of-day, geography, or engagement intensity is suspect and must be flagged for re-review.

A strong answer adds

• Inject synthetic heartbeat events at known rates as a canary to provide a continuous, independent estimate of delivery loss.
• Quantify loss by event type, not just overall rate: high-value events (purchases) and low-value events (scrolls) losing at different rates biases different metrics differently.
• Communicate with error bars while the investigation is running: publish a known-issues banner on affected dashboards, an estimated-impact range per metric, and a list of decisions under re-review, because trust survives acknowledged uncertainty better than discovered silence.
• Remediate in two parallel tracks: fix the pipeline through backpressure, client-side buffering and retry, and explicit sampling rather than accidental sampling; and backfill where client buffers allow.
• Prevent recurrence with permanent instrumentation: data-quality SLIs measuring delivery rate against client counters, loss-rate alerts broken out by segment, and end-to-end synthetic event monitoring as a standing canary.

Common misses

• Estimates only a single aggregate loss rate without breaking it down by segment, missing that correlated loss produces bias no volume correction fixes.
• Treats the loss as missing at random and therefore assumes it merely adds noise to metrics, without recognizing that peak-load loss is almost guaranteed to be missing-not-at-random.
• Fails to identify which downstream metrics and experiment decisions are most exposed, rather than treating all dashboards as equally affected.
• Does not propose a second observation channel with different failure modes to measure the first channel's loss.
• Stays silent toward stakeholders while the investigation runs, rather than publishing acknowledged uncertainty with error bars.

The underlying concept

Data missing "at random" merely adds noise; data missing in a way correlated with the outcome or the segment (missing-not-at-random) adds bias that no amount of volume fixes. Peak-load loss is almost guaranteed to be the second kind. The estimation strategy is always the same: find a second observation channel with different failure modes and use the disagreement to measure the first channel's loss. And the organizational lesson mirrors the statistical one — metrics infrastructure needs its own observability, because the failure mode of measurement systems is silence, not alarms.

Source

Distilled Prep canon — curated from Meta's public work on data infrastructure and measurement integrity.

Source: Meta engineering blog

What the interviewer is probing

Whether the candidate understands the funnel economics that force the retrieval → ranking → re-ranking architecture: compute per candidate rises steeply while candidate count falls, and the design skill is allocating latency and compute across stages. Also probing operational ML maturity — feature freshness, offline/online parity, and safe rollout are where these systems actually fail, and strong candidates raise them unprompted.

What a strong answer covers

Must establish

• Funnel structure with explicit stage budgets: candidate retrieval (millions to thousands via ANN over embeddings plus heuristic and graph sources) in tens of ms; ranking (thousands to hundreds via a mid-size model with hydrated features) taking the largest share; re-ranking and policy (hundreds to the final slate via diversity filters, integrity filters, and business rules) in single-digit ms.
• Feature infrastructure as the real bottleneck: a feature store with an online serving tier, batched hydration, per-session caching of user features, and strict timeout-plus-default behavior so one slow feature never breaks the page.
• Offline/online parity: using the same feature definitions in training and serving prevents the classic silent quality bug.

A strong answer adds

• Freshness tiers: user long-term embeddings refresh daily, item embeddings refresh on ingest, and short-term interest is served via a real-time session feature stream rather than model retraining.
• Cold start for users: fall back to popularity-by-cohort and content-based retrieval.
• Cold start for items: explicitly widen exploration for new items to gather signal, and accept the small engagement tax that costs.
• Feedback loop integrity: the system trains on data it created, so log propensities and hold out random exploration traffic to avoid bias compounding.
• Content integrity gates belong in the re-rank layer, not applied post-hoc.
• Rollout sequence: offline metrics gate, then shadow scoring for parity checking, then a small-percentage online experiment with engagement and latency guardrails, then staged ramp with automatic rollback triggers.

Common misses

• Does not justify why multiple stages are needed and cannot explain what breaks if one heavy model ranks all candidates.
• Treats feature hydration as a background detail rather than identifying it as the primary latency and reliability bottleneck.
• Omits timeout-plus-default behavior, leaving no answer for what happens when a single slow feature blows the latency budget.
• Ignores offline/online feature parity, missing the most common source of silent quality regressions at this scale.
• Has no answer for cold-start users or new items.
• Does not address the feedback loop problem: training on system-generated data without logging propensities or holding out exploration traffic.
• Describes rollout only as an A/B test without mentioning shadow scoring, latency guardrails, or automatic rollback triggers.

The underlying concept

Multi-stage ranking is the standard answer to a scaling identity: you cannot afford heavy computation on every candidate, so you spend cheap computation to decide where to spend expensive computation. Each stage trades recall for precision at rising cost per item — the architecture is a compute-allocation curve, and latency budgets are how it's enforced. The second load-bearing idea is that at this scale the model is a minority of the system: feature pipelines, freshness, parity, and rollout safety determine realized quality more than architecture choices, which is why interviewers push hardest there.

Source

Distilled Prep canon — curated from Meta's public work on recommendation infrastructure.

Source: Meta engineering blog