Google

What the interviewer is probing

This question tests whether the candidate immediately recognizes the network interference problem — the defining reason standard A/B testing fails here — and whether they can reason about the trade-offs of the available alternatives (geographic clustering, switchback/crossover, synthetic control). It also probes metric judgment: the distinction between local segment effects and network-wide effects is not cosmetic, and a strong candidate will articulate why only one of them represents the policy-relevant estimand. Finally, the follow-ups reveal causal discipline — can they enumerate the confounders a switchback design is exposed to and propose concrete countermeasures?

What a strong answer covers

• Why standard A/B fails: routing changes are not independent across users — diverting user A changes congestion for user B, so individual-level randomization produces SUTVA violations everywhere. Treatment and control units interfere, making estimated effects meaningless.
• Randomization unit options and their costs:
• Geographic clustering (randomize by city or district): eliminates within-city interference but requires many cities for power; spillover at cluster boundaries remains.
• Switchback / crossover design (alternate treatment and control over time across the whole city): removes geographic spillover by giving the entire network the same treatment state at any moment. Cost: assumes the city reaches a new equilibrium quickly, and that consecutive-day effects don't bleed over (carryover bias).
• Synthetic control (match treated cities to untreated ones): useful for city-level rollouts but requires rich pre-treatment data and assumes no network ties between cities.
• Segment selection is a design choice, not a tuning knob: selecting ~100 historically congested segments creates a well-defined estimand (effect on those bottlenecks) but limits external validity to similar bottleneck types. Pre-registration matters here to prevent post-hoc fishing.
• Carryover bias in switchback: traffic patterns have memory (rush hours, weekly seasonality). Mitigation: enforce washout periods between arms, balance arm assignment by day-of-week, and model residual autocorrelation explicitly in the analysis.
• Analysis with hierarchical Bayesian model: pools information across cities and hours while allowing city-level heterogeneity; appropriate when per-city power is limited. Key output is a posterior distribution over the effect, not just a point estimate — which forces honest uncertainty communication.
• Interpreting the two effect sizes: the ~2% targeted-segment effect measures local decongestion; the ~0.35% network-wide effect is the policy-relevant number because it captures both the gains on cleared segments and the costs imposed on absorbing segments. Reporting only the larger local number would overstate societal benefit.
• Common wrong turns: claiming individual-level causal effects from a city-level design; ignoring that non-app users benefit (which is actually an argument for the intervention, not against measurement); conflating statistical significance with practical significance at 0.35%.

The underlying concept

Standard A/B testing rests on the Stable Unit Treatment Value Assumption (SUTVA): one unit's treatment does not affect another's outcome. Traffic networks violate this structurally — a rerouted vehicle changes travel times for every other vehicle on its new and old path, a phenomenon called interference or spillover. When interference is global (the whole network is one connected system), the only valid randomization units are ones that contain the interference: entire cities, or time periods in a switchback design. Switchback designs trade spatial isolation for temporal isolation, measuring the city's response to switching the entire system between states, but they introduce carryover risk whenever the system's equilibrium lag exceeds the switching interval. The causal estimand must be defined at the level of the randomization unit — network-wide average speed, not individual trip time — or the estimate is not causally identified.

Source

Derived from The power of collaboration: How we can reduce traffic congestion

What the interviewer is probing

This question probes whether the candidate can translate an economic constraint (pay-per-byte-hour) into an algorithmic design, rather than treating resource allocation as a static engineering parameter. It tests architecture judgment — separating the TTL assignment problem from the capacity management problem — and forces the candidate to reason about the feedback loops between a prediction model and the system it controls. The validation portion surfaces metric judgment: distinguishing between cost savings and latency degradation, and knowing which cache miss increases are acceptable.

What a strong answer covers

• Frame the cost model first. Total cost = (memory cost × bytes cached × time held) + (miss penalty × miss rate). The insight is that holding a page whose next access is far away costs more in memory rental than the miss penalty you'd pay to re-fetch it. This reframes eviction as: "when should I stop renting this page's slot?"
• The ski-rental analogy. Each cached page faces the same rent-vs-buy decision under uncertainty about future access. The break-even rule: keep the page as long as its accrued memory cost is below the miss penalty; evict when it crosses. A randomized variant reduces expected cost versus the adversarial worst case.
• TTL assignment via a lightweight model. Features: data size (miss penalty scales with it), operation type (read vs. scan), historical inter-arrival time for this page, relative miss cost vs. memory cost ratio. Candidate should reach for a shallow decision tree or small lookup table — not a neural net — because inference runs on every cache hit in a hot path. Wrong turn: proposing a complex model without noting the latency budget.
• Capacity eviction as a backstop. When the cache fills, fall back to a generalized LRU (or GDSF to handle variable-size pages). TTL-based eviction shrinks the working set first; LRU handles physical overflow. These are complementary, not competing.
• Validation strategy. Offline: replay cache traces with the new policy vs. fixed-size baseline; measure integrated memory-time cost and miss rate. Distinguish miss rate by miss cost — a high miss rate on cheap-to-fetch rows is acceptable; misses on expensive cross-shard reads are not. Online: shadow the new policy against production traffic before cutover; track p99 latency, actual I/O cost per request, and memory footprint over time. Alert on tail-latency degradation, not just aggregate miss rate. The critical wrong turn: using miss rate alone as the health signal — a cost-aware system deliberately increases misses on cheap data.
• Adaptation. TTL model should be retrained on rolling windows of production access logs. Feature drift (e.g., a new query pattern) shows up as systematic TTL misprediction; detect this by monitoring model-predicted TTL vs. actual next-access gap for sampled pages.

The underlying concept

Static cache sizing is a peak-provisioning problem: you allocate for the worst hour and pay for all the idle hours in between. When memory is metered, this becomes a literal dollar cost rather than an amortized hardware cost, which makes the economics of dynamic sizing much more attractive. The ski rental problem is the canonical model for this class of decision: you must commit to a recurring cost (renting memory) or a one-time cost (paying the miss penalty) without knowing how long you'll need the resource. The break-even algorithm — rent until your total rental equals the buy price, then buy — is worst-case optimal within a factor of 2. In practice, access patterns are predictable enough that a learned model can beat the adversarial bound significantly. The deeper structural insight is that TTL-based eviction and capacity-based eviction solve different subproblems — TTL controls the holding duration under normal load, and LRU/GDSF controls behavior when the cache fills — and keeping them cleanly separated makes both easier to reason about and tune.

Source

Derived from Optimizing cloud economics with linear elastic caching

What the interviewer is probing

The question tests whether the candidate can reason about transfer learning not as a recipe but as a set of assumptions — about distribution shift, label density, and feature alignment — that must each be checked empirically. Metric judgment is probed through the carbon-accounting use case, which makes false-negative costs concrete. Causal reasoning about geographic clustering in the label set is the hardest dimension; strong candidates will recognize that spatially autocorrelated data requires spatial cross-validation, not random splits.

What a strong answer covers

• Why a pre-trained backbone helps: satellite imagery shares low-level texture features (vegetation spectral signatures, edge patterns, shadow geometry) across geographies; a backbone trained on hundreds of millions of global images will have learned these representations far better than a model trained from scratch on a few hundred km² of annotations. The fine-tuning hypothesis: the head and upper layers need to specialize to local landscape character; the lower layers may transfer almost directly.
• Validation set design — the critical issue: spatial autocorrelation means that if you randomly split pixels or even patches, train and validation sets will be geographically adjacent and share context. Use spatial cross-validation: hold out geographically contiguous blocks (e.g., entire counties), not random patches. This gives honest generalization signal.
• Checking that pre-training actually helps: ablation — train the same head from random init vs. from the pre-trained backbone, on the same labeled data, evaluated on the spatial hold-out. If the backbone doesn't improve sample efficiency (fewer labeled km² needed for the same performance), the distribution shift is too large and the representations don't transfer.
• How much to freeze: start with the backbone frozen, establish a baseline, then progressively unfreeze upper layers and measure validation performance. The inflection point where unfreezing hurts (overfitting to the small labeled set) tells you where to stop. Monitor training vs. validation loss gap carefully.
• Distribution shift failure modes in deployment:
• Geographic bias in labels: if annotators covered accessible or visually distinct areas, the model learns those contexts. Field-interior hedgerows, less-managed boundaries, or features in remote uplands may be systematically missed.
• Seasonal/illumination shift: fine-tuning imagery may be from one season; deployment imagery from another — vegetation appearance changes significantly.
• Resolution or sensor shift: if inference imagery comes from a different satellite or processing pipeline, spectral statistics shift.
• Rare but high-value features: stone walls may be underrepresented in training; the model's recall on them may be near zero without explicit oversampling or a weighted loss.
• Asymmetric cost — false negatives: use a recall-weighted loss (e.g., focal loss or class-weighted cross-entropy biased toward the positive class), tune the classification threshold explicitly on the spatial validation set against a precision-recall curve, and report recall and false-negative rate per feature class as primary metrics, not aggregate pixel accuracy.
• Wrong turns: using random pixel splits for validation (falsely optimistic generalization); treating aggregate IoU as the primary metric when the use case is carbon accounting (hides per-class recall differences); freezing the entire backbone without testing whether fine-tuning upper layers helps on this specific landscape character.

The underlying concept

Transfer learning from large foundation models works when two conditions hold: the source pre-training distribution shares low-level structure with the target task (so early-layer features transfer), and the target dataset, while small, is representative enough of the deployment distribution that fine-tuning the upper layers doesn't overfit to annotation artifacts. Geographic data violates the standard i.i.d. sampling assumption — nearby locations are correlated — so a random train/val split produces optimistically biased validation metrics. Spatial cross-validation, which holds out contiguous geographic blocks, is the correct analog to time-series cross-validation in temporal data: it simulates the actual generalization challenge, predicting on regions the model has never seen. Asymmetric misclassification costs should be encoded in both the loss function and the evaluation protocol from the start, not corrected post-hoc.

Source

Derived from From pixels to planning: Earth AI for nature restoration

What the interviewer is probing

Metric literacy in search: CTR measures the promise of a result (title/snippet attractiveness), not its delivery. Rising reformulations and pogo-sticking are classic signals that clicks are failing to satisfy. Strong candidates assemble a success-oriented metric story (abandonment done right, long clicks, session-level success) and slice by query intent before deciding.

What a strong answer covers

Must establish

• The reconciling story: the change likely surfaces clickable results with compelling titles/snippets that under-deliver; users click, bounce back (pogo-stick), and re-query, meaning CTR up plus reformulation up equals attractiveness up but relevance flat or down.
• The evidence pattern points against launch despite the CTR gain; the correct recommendation is either to iterate on snippet honesty and relevance, or to launch only for the query classes where session success genuinely improved.
• Prevention requires making session-success and pogo-stick rate guardrail metrics so attractiveness-only wins cannot clear the launch bar again.

A strong answer adds

• Long clicks (dwell above threshold) versus short clicks are a better signal than raw CTR for distinguishing attractive results from useful ones.
• Pogo-stick rate directly measures the failure mode this experiment exhibits and must be part of the metric set.
• Session-level success (did the session end in a satisfied state?) is the unit of value closest to the user's true goal.
• Time-to-result belongs in the metric set alongside long clicks and pogo-stick rate.
• Good abandonment (query answered on the results page itself) must be separated from bad abandonment before interpreting session-end states.
• Diagnose by intent: navigational, informational, and transactional queries can respond differently to the same change, and the aggregate hides that; the launch decision should reflect which classes actually improved.
• Also slice by position: determine whether the added CTR is coming from top positions or deep exploration.

Exceptional depth

• Dwell time is ambiguous: long dwell can mean satisfaction or struggle, so time signals must be calibrated per query class against explicit-feedback panels or human relevance ratings before being used as success proxies.

Common misses

• Treating rising CTR as sufficient evidence of improvement without interrogating what the rising reformulations and back-button returns reveal.
• Failing to separate good abandonment from bad abandonment, which leaves session-end signals uninterpretable.
• Analyzing only aggregate results without slicing by query intent, missing that the change may help one query class while harming another.

The underlying concept

Every proxy metric encodes an assumption about what it proxies for; CTR assumes clicking predicts satisfaction, and ranking changes can break exactly that link by optimizing the promise rather than the payoff. Search measurement matured by moving from action counts to session-level success models — the unit of value is the user's task, not the click. The generalizable habit: when two proxies disagree (clicks up, re-queries up), the disagreement itself is the finding, and the tiebreaker is the metric closest to the user's true goal.

Source

Distilled Prep canon — built on Google's published work on search quality measurement.

Source: Google engineering blog

What the interviewer is probing

Whether the candidate understands that at extreme power the binary significance question dissolves and the real questions become effect size, practical value, and portfolio-level error control. Also probing knowledge of overlapping-experiment infrastructure — layered traffic allocation — which Google pioneered and which any candidate claiming experimentation depth should be able to reconstruct from first principles.

What a strong answer covers

Must establish

• Replace 'significant?' with 'big enough to matter?': use pre-registered practical-significance thresholds per metric (minimum launch-worthy effect) and report confidence intervals against those thresholds rather than against zero.
• Convert core metrics to a common currency (long-term value per user) with explicit exchange rates set by leadership, so mixed-sign results are decidable rather than debated ad hoc.
• Solve concurrency with layered/orthogonal experiment infrastructure: divide the system into layers (e.g., UI, ranking, ads) where experiments within a layer split traffic exclusively but layers overlap freely via independent hashing.

A strong answer adds

• Run explicit interaction tests only for suspected-conflicting pairs rather than for all concurrent experiments.
• Control the expected number of false launches across thousands of tests by setting higher evidence bars for launch-gating metrics.
• Require replication runs for surprising wins and holdback verification post-launch as part of portfolio error control.
• Address long-term validity: tiny short-term gains can be novelty or cannibalization, so use long-running holdbacks to measure cumulative and lagged effects.
• Validate proxy metrics against long-running holdbacks to confirm they predict what they claim to predict.
• Pre-register any segment analyses or use hierarchical shrinkage estimates; treat exploratory segment findings as hypotheses requiring confirmation in a fresh experiment, never launched from discovery data.

Common misses

• Continues to use statistical significance as the primary launch bar rather than replacing it with practical-significance thresholds.
• Cannot resolve mixed-sign results across metrics because no explicit exchange rates or common currency have been established.
• Treats concurrent experiments as a scheduling problem (serialization) rather than solving it with layered/orthogonal traffic allocation.
• Does not address portfolio-level false-launch rate, focusing on per-experiment error control instead.
• Launches from exploratory segment findings without requiring confirmation in a fresh experiment.

The underlying concept

Statistical significance answers "is the effect exactly zero?" — a question nobody asked. With enormous samples, standard errors collapse and every real-but-trivial effect clears p-thresholds, so decision theory must take over: minimum effects of interest, explicit trade-off prices between metrics, and error control at the portfolio level (false launches, not false positives). Overlapping-layer designs solve the concurrency problem by making experiment assignments statistically independent across layers, spending scarce exclusive traffic only where interference is real.

Source

Distilled Prep canon — built on Google's published experimentation methodology (overlapping experiments, long-term holdbacks).

Source: Google engineering blog

What the interviewer is probing

Production-ML maturity: the understanding that most real-world model failures are data failures — schema drift, distribution shift, broken upstream joins, feature pipeline bugs — and that reliability comes from treating data like code: schemas, validation, versioning, and gated promotion. The TFX worldview, reconstructable from first principles.

What a strong answer covers

Must establish

• Data failure taxonomy covering schema changes (renamed or dropped fields, type changes), distribution shift (gradual drift vs. sudden breakage), missing partitions, upstream logic changes silently redefining a feature, and label problems (delay, leakage).
• Automated data validation stage: maintain a versioned schema plus expected statistics per feature (ranges, missingness, distributions); hard-fail on schema violations, alert on statistical anomalies against a reference window, and quarantine bad batches rather than train on them.
• Staged promotion gates: offline eval thresholds, then canary serving on small traffic with online guardrails, then gradual ramp with automatic rollback, with the previous model kept warm for instant reversion.

A strong answer adds

• Skew prevention by construction: a single feature-definition source compiled to both training and serving (shared transforms or feature store), so the structural guarantee replaces vigilance.
• Continuously compare serving-time feature distributions against training distributions, treating skew alarms as among the highest-value monitors.
• Training pipeline reproducibility: versioned data snapshots plus versioned code plus config produce reproducible artifacts.
• Evaluate trained models on holdout and on sliced metrics across segments where regressions hide, using the current production model as the explicit baseline.
• Versioned lineage (which data, which code, which model) as the diagnosis mechanism: when production quality dips, lineage localizes whether the cause is data, model, or world; without it, every incident is archaeology.

Common misses

• Treats validation as a single schema check and misses statistical anomaly detection against a reference window (distribution drift, missingness rates).
• Does not quarantine bad batches: trains through detected anomalies rather than blocking them.
• Addresses training/serving skew through monitoring alone rather than preventing it structurally with a single shared feature definition.
• Evaluates the new model only on holdout metrics, missing sliced evaluation against the incumbent production model.
• Omits versioned lineage, leaving the pipeline without a fast path to localize the cause of a production degradation.

The underlying concept

Continuous training turns ML into a data-processing system whose correctness depends on inputs no one reviews — so the engineering answer is to give data the same discipline code has: declared schemas, automated validation, version control, and staged rollout. The two structural insights: skew is prevented by construction (one definition, two consumers), not by vigilance; and models should be promoted like binaries — evaluated against the incumbent, canaried, and reversible — because "newer" is not "better" when the data decides.

Source

Distilled Prep canon — built on the themes of Google's TFX and ML production-systems publications.

Source: Google engineering blog

What the interviewer is probing

Understanding of the core datacenter economics — services reserve for peak and idle most capacity, so reclaiming that idle capacity for preemptible batch work is worth billions — plus the mechanisms that make sharing safe: priorities/tiers, preemption, resource isolation, and overcommit informed by actual usage vs. reservations. This is Borg territory; Kubernetes-only intuition covers maybe half of it.

What a strong answer covers

Must establish

• Why mixing matters: dedicated pools idle at 20-40% utilization because services size for peak, and co-locating preemptible batch in the trough reclaims that capacity; the danger is interference where batch steals CPU cache, memory bandwidth, or IO from latency-critical work.
• Job spec must carry resource requests (CPU, RAM, disk, accelerators), a priority tier (production vs. best-effort), placement constraints (region, hardware, anti-affinity), and disruption tolerance; quota is enforced at admission and priority at runtime.
• Priority mechanics: a top-tier service preempts best-effort tasks by having the scheduler evict enough batch (respecting disruption budgets), reclaim resources, and place the service; batch tasks are checkpointed or simply rescheduled under a contract of 'will finish eventually.'

A strong answer adds

• Scheduling loop has two stages: feasibility filtering to find machines that can host the job, then scoring on bin-packing for utilization, spreading for failure domains, and data locality.
• At fleet scale, use equivalence classes and cached scores rather than rescoring every machine, combined with randomized sampling of candidate machines instead of evaluating the whole fleet.
• Shard or replicate schedulers with optimistic concurrency on placement commits to keep scheduling latency low.
• Overcommit safely by scheduling against predicted actual usage rather than reservations for lower tiers, enforcing node-level isolation (cgroups-style), and running a killer that reclaims from best-effort first when a node runs hot.
• Reliability design: scheduler state is rebuilt from node agents (nodes are the source of truth), so running workloads are unaffected by scheduler outages; scheduler masters are consensus-replicated per cell; failure domains and maintenance are handled via eviction with notice.

Common misses

• Treats the scheduling objective as pure optimality rather than good-enough placement found fast via sampling, caching, and sharding.
• Does not articulate the two-stage feasibility-then-scoring structure, collapsing them into a single vague 'find a machine' step.
• Ignores overcommit entirely, scheduling only against reservations and leaving the economic reclamation argument incomplete.
• Couples the data plane to the control plane, so the scheduler's own failure would take down scheduled workloads.
• Omits disruption budgets when describing preemption, implying unconstrained eviction of batch tasks.

The underlying concept

Cluster scheduling is economically a statistical multiplexing problem: peak-reserved capacity is mostly air, and tiered priorities plus preemption convert that air into throughput while preserving the latency tier's guarantees. Technically it's constrained optimization under a latency budget — solved not by optimality but by good-enough placement found fast (sampling, caching, sharding). The deepest design rule: keep the data plane independent of the control plane, so the scheduler's own failures never take down what it scheduled.

Source

Distilled Prep canon — built on the themes of Google's Borg and cluster-management publications.

Source: Google engineering blog

What the interviewer is probing

Whether the candidate can localize the strong-consistency requirement (budget commitment) and engineer around paying cross-continent latency on the hot path — the budget-reservation/quota-lease pattern. Also probing real distributed-systems literacy: consensus per shard, read-path options (leases, bounded staleness), and why globally ordered transactions need bounded clock uncertainty.

What a strong answer covers

Must establish

• Split the problem into two distinct paths: the invariant (total spend never exceeds budget) requires serialized commitment, while serving decisions (serve this ad?) require microsecond reads. These two paths must never be co-located.
• Budget ledger sharded by account or campaign, with each shard running as a consensus-replicated state machine (Paxos or Raft) across regions. Writes commit with cross-region quorum, paying the ~100 ms latency once per budget mutation, which is rare traffic.
• Regions lease budget slices (tranches) from the ledger so ad servers decrement local tranches at memory speed. Overspend is bounded by outstanding tranche size, and tranche sizes are chosen to trade slight budget under-utilization against overspend risk.

A strong answer adds

• Serving reads hit local replicas or leases and are bounded-stale by construction. Reporting reads can request linearizable reads through the shard leader when exactness matters.
• External consistency (transactions ordered as observed globally) requires bounded clock uncertainty. The TrueTime approach exposes uncertainty intervals and waits them out on commit. Plain NTP either forces waiting too long or violates ordering under skew.
• Partition behavior: a cut-off region can spend only its already-leased tranches, then fails closed and stops spending. Leases expire and return to the ledger, bounding damage without violating the global invariant.
• State the CAP position explicitly: the ledger chooses consistency. Engineered availability comes from leases, not from weakening the invariant.

Common misses

• Putting serving decisions on the strong-consistency path, forcing ad-serving reads to wait on cross-region consensus.
• Failing to bound overspend during a partition: allowing a region to continue spending without any cap when cut off from the ledger.
• Not addressing clock uncertainty: assuming NTP is sufficient for globally ordered transactions without accounting for skew or uncertainty intervals.

The underlying concept

Global strong consistency is affordable when you stop buying it wholesale: identify the single invariant that needs serialization, run consensus for exactly that, and convert the hot path into locally served, pre-authorized capacity — the quota-lease/escrow pattern that turns a global constraint into bounded local ones. The Spanner lesson completes it: global transaction ordering is fundamentally a clock problem, solved not by perfect clocks but by known-bounded uncertainty plus commit waits.

Source

Distilled Prep canon — built on the themes of Google's Spanner and planet-scale infrastructure work.

Source: Google engineering blog