As data infrastructure evolves to support autonomous AI, the engineering requirements for reliability have shifted. Mallika Rao, who has led infrastructure and platform teams across Twitter, Walmart, and most recently as engineering Manager at Netflix shares her perspective on moving beyond traditional ETL toward self-healing systems that can survive the scale and complexity of modern search and recommendation engines.
The transition from human-read dashboards to automated AI decision-making has fundamentally altered the risk profile of modern data infrastructure. When pipelines feed search, ranking, and recommendation engines instead of static reports, the “operational blast radius” of a silent failure or a delayed signal becomes a critical product risk.
Addressing this challenge of decision quality at scale has been a career-long focus for Mallika Rao. Drawing on her tenure leading engineering teams at Twitter, Walmart, and Netflix, Rao specializes in the high-stakes intersection of platform foundations and real-time AI infrastructure. Her experience has been instrumental in moving recommendation engines beyond “black box” behavior toward transparent, resilient systems that drive member engagement.
In this conversation with Hyperight, Rao explores the shift toward self-healing systems-moving beyond backend plumbing to treat data platforms as a resilient product surface. She breaks down why modern data engineering must prioritize idempotent processing, adaptive orchestration, and robust data contracts to meet the unforgiving requirements of autonomous AI systems.
She notes that the perspectives shared here are her own and do not necessarily reflect those of her current or former employers.
Why are traditional data pipelines no longer sufficient in the AI era? 
Mallika Rao: If you look at how most data pipelines were originally designed, they assumed two things. First, that data moved relatively slowly. Second, that humans were the primary consumers of that data.
Most pipelines were built to power dashboards, reports, or offline analysis. When something went wrong, someone eventually noticed and investigated.
That assumption breaks down completely in the AI era. Today, pipelines increasingly feed automated systems; search, ranking, recommendations, fraud detection. These systems make decisions continuously and at massive scale, often without any human in the loop.
When you move into that world, the bar changes dramatically. Freshness matters. Correctness matters. Consistency matters.
And the biggest challenge is that small issues don’t stay small anymore. A delayed signal, a skewed feature distribution, or a quiet data drop can propagate through models and affect millions of automated decisions before anyone notices.
Traditional pipelines are very good at moving data from point A to point B. But they’re not designed to reason about whether that data is still safe to use.
In AI systems, that gap becomes a real risk surface.
What do you mean by “self-healing pipelines”?
Mallika Rao: The term can sound a little magical, so I like to clarify what it actually means. Self-healing pipelines are not systems that never fail. At scale, failure is inevitable.
What they are is systems designed with failure as a first-class assumption.
The goal is to ensure that when known failure modes occur, the system can recover safely without waking up a human every time.
There are a few foundational principles behind that.
One is idempotent processing, so retries are safe and don’t corrupt downstream state.
Another is intelligent retry strategies that distinguish between transient failures; like temporary infrastructure issues; and systemic problems where retrying would just amplify the issue.
Self-healing systems also rely heavily on adaptive orchestration. Instead of running pipelines on rigid schedules, the system should be able to slow down, shed load, or reroute work based on real conditions.
And finally, good fallback strategies matter. If an upstream signal is late or partially unavailable, the system should degrade gracefully rather than failing catastrophically.
The goal is not perfection. The goal is containing failures and shrinking the operational blast radius.
What production engineering principles matter most for data pipelines today?
Mallika Rao: There are three principles that I believe matter most.
The first is explicit failure modeling. Too many pipelines are designed around the happy path; how data flows when everything works perfectly. But real incidents rarely look like total outages. More often, they show up as partial failures, backpressure, queue buildup, or subtle tail-latency problems.You might see p99 processing time slowly drift, which eventually translates into stale features or degraded recommendations. If you’re not designing for those scenarios, you will eventually get surprised by them.
The second principle is observability that reflects reality.
Averages are often misleading. What really matters is freshness, completeness, and distributional behavior.You want to know not just that data arrived, but whether it arrived on time, whether it was complete, and whether it’s safe to consume downstream.
One example I’ve seen repeatedly is shadow canaries. Teams often deploy them to validate new pipelines or transformations. But if those canaries aren’t wired into alerting or freshness monitoring, they silently fail. The system looks healthy on dashboards, while correctness is already drifting.
The third principle is prevention.Every incident should leave the system stronger than it was before.That means turning operational pain into automation: stronger data contracts, new validation checks, better backpressure handling, or architectural improvements.
A runbook alone isn’t enough. Production engineering is about systematically converting incidents into resilience.
How do search and recommendation systems change pipeline requirements?
Mallika Rao: Search and recommendation systems are uniquely unforgiving.
They’re extremely sensitive to both latency variance and data quality issues. A small delay in a real-time signal, or a subtle correctness bug in a feature pipeline, can directly degrade relevance and user experience.The tricky part is that many of these failures are silent.
The system continues to return results, but those results are quietly worse.
Because of that, pipelines feeding these systems need stronger guarantees.
Freshness becomes a hard requirement rather than a nice-to-have. Provenance becomes critical so you can trace where signals came from and how they were transformed. And consistency across pipelines matters when multiple features converge inside models.
At that point, throughput alone isn’t the goal anymore.
The real goal is predictable, explainable behavior under load.
Once pipelines influence user-facing decisions, they stop being backend plumbing. They effectively become part of the product surface.
What should data engineering leaders invest in now?
Mallika Rao: The biggest advice I give leaders is to invest in fundamentals.
Start with clear ownership. Every pipeline should have an accountable owner who understands both the data and its downstream impact.
Then invest in observability that reflects user-facing risk, not just system health. If a pipeline breaks, the first question shouldn’t be “is the job running?” It should be “is the data still safe to use?”
Data contracts are another major area. Producers and consumers should share explicit expectations about schema, freshness, and quality.
And finally, automation, but the right kind of automation.The goal isn’t just more tooling. The goal is reducing cognitive load on engineers. Fewer fragile manual processes. Fewer alerts that don’t matter. Fewer situations where someone has to reverse-engineer a pipeline at 2 a.m.
These investments are rarely flashy.
But they are the difference between teams that constantly firefight operational issues and teams that can confidently build the next generation of AI-driven systems.
Join Mallika Rao at the upcoming Data Innovation Summit, as she shares a practical blueprint for implementing idempotent processing, automated remediation, and metadata contracts at scale. Don’t miss the chance to learn how to eliminate manual toil and secure the “operational blast radius” of your AI-driven search and recommendation platforms.
