Building real-time observability into Expedia’s air booking funnel

From the moment a traveler hits “Purchase,” Expedia’s systems orchestrate a series of calls to Global Distribution Systems (GDSs), airline connectors, payment gateways, and other services. Most of the time, this works perfectly and the booking goes through successfully. But sometimes, a fare expires, a supplier times out, or a schedule changes mid-purchase, and the booking fails.

The metric we use to protect this experience is Purchase Success Rate (PSR), the fraction of booking attempts that actually complete. Because our supplier chain is so long, a dip of even 1–2% in PSR can mean thousands of failed bookings and a massive loss in revenue.

The visibility gap

For a long time, investigating a drop in PSR was a manual, “needle in a haystack” effort. Failures were buried in the noise of millions of rows of service logs. To find a root cause, an on-call engineer often had to run queries in Querybook and manually investigate in Datadog, a process that could take one to four hours.

Beyond the time sink, many signals were simply too quiet to catch. When a single carrier or a specific route started degrading, the signal was often drowned out by the overall volume of the flight funnel. To solve this, we built the Air Booking Insights platform, a system designed to turn monitoring from a passive chart into an automated triage engine.

Building the ETL and Data Layer

The first challenge was processing millions of raw events into something queryable on our lakehouse: Trino as a distributed SQL engine over Hive-cataloged datasets in S3. We implemented an ETL (Extract, Transform, Load) pipeline on top of that stack.

Extract: raw service-call events are materialized into lossless intermediate storage.

Transform: this is where messy logs become analysis-ready. The key step is signature normalization: multi-stage regex strips volatile tokens (UUIDs, ARLs, timestamps, fare versions) from free-text errors and collapses them into stable signatures. Those signatures are mapped into high-level failure categories, then joined with other tables to enrich failure data with booking-side dimensions like cabin class and route.

Load: the result lands in an aggregated Iceberg table at an hourly grain (Hour × GDS × Carrier × Error). For the same dashboard breakdowns, Trino now reads that pre-aggregated table instead of re-scanning raw service-call logs row by row. Comparing bytes scanned on those query paths is where the roughly 70% reduction comes from. On top of that, we added a three-layer cache so the dashboard stays fast under real usage:

• In-memory TTL cache for instant repeated lookups
• Single-flight de-duplication so identical concurrent queries do not hammer the database
• On-disk SQLite cache for immutable historical data, so date navigation stays near-instant

Detecting dips in PSR

Before we can explain a drop, we have to catch it. Anomaly detection is the layer that watches PSR continuously and flags when the funnel is behaving differently from what we expect, so a quiet dip does not sit unnoticed until someone happens to open a chart.

Defining “Normal” with the Safety Band

Detecting an anomaly is only useful if the system knows what “normal” looks like. We can’t use a flat threshold because booking volume is seasonal: a quiet Tuesday at 2:00 AM looks very different from a Friday evening surge.

We built a Seasonal Safety Band using eight weeks of history for every specific hour-of-the-week slot. By using Median and Median Absolute Deviation (MAD) instead of standard averages, our “normal” range isn’t corrupted by past outages. A dip only flags as an incident if it breaks this statistical band and crosses a volume-scaled impact floor, which separates minor blips from high-severity outages.

Automating the Failure Analysis

Once a dip clears that bar, the core of the platform is explaining why it is happening. The system performs a root-cause breakdown by comparing the current “bad” window against the same seasonal baseline, specifically the same day-of-week and clock-window over the prior eight weeks.

This allows us to see exactly what changed by looking for attribute concentration:

• Supplier Concentration: If failures spike for one GDS across all airlines, it’s a vendor-level outage.
• Carrier Concentration: If failures spike for one airline across multiple GDSs, it’s a carrier-side issue.
• Pattern Recognition: By grouping those normalized signatures into error clusters, we can instantly see if a drop is driven by a specific failure pattern.

AIRA: Empowering Non-Technical Teams

To ensure these insights weren’t locked behind a SQL console, we integrated AIRA, our AI assistant. AIRA lets non-technical teams triage failures in natural language, using the Trino MCP for accurate data access.

Instead of waiting for an engineer, a product manager can ask AIRA: “What’s driving the drop in PSR for the last week?” AIRA then writes the validated SQL, executes it, and narrates the cause with inline charts. This removes the context switching between tools and democratizes incident response across the organization.

Real Impacts

The impact of shifting from manual logs to baseline-aware triage has been immediate:

• Time Compression: Investigations that once took hours are now completed in minutes. Measuring the median manual investigation time against the automated triage flow works out to a ~95% reduction, and translating those reclaimed on-call engineer hours across a year's worth of incidents into loaded labor cost resulted in $220,000+ in annual operational savings.
• Proven Accuracy: In backtesting, the system caught documented major incidents and also surfaced smaller outages that still moved PSR, ones that had gone previously untracked and undocumented.
• Recoverable Value: The system identified millions of dollars in monthly Gross Booking Value in recoverable bookings that were previously lost in the noise.

By automating the heavy lifting with ETL and AIRA, we can isolate, resolve, and analyze outages before they hit thousands of travelers. I’m proud to have helped kick off the Air Booking Insights platform, and I’m excited to see where the team takes it next!