Consolidator Pipeline

The Manual Process

A consolidator is a potential partner company — a wholesaler that sells airline fares through its own API. Before we work with one, someone has to answer a deceptively simple question: are their prices actually competitive? Until recently, answering it meant people searching flights by hand on websites — route by route, date by date — and copying prices into spreadsheets. A single evaluation tied up people for days, the data was inconsistent, and by the time it was assembled the fares had already moved. I built the automation that replaced that process end to end: data gathering, data processing and data analysis, with a human only at the start and at the end.

One Batch, End to End

To the analyst, the whole thing is a form: pick the partner, pick a routes file, type the departure dates, press run. From there the pipeline takes over. It merges the route list with the partner's request defaults — one entry per route per date — fires the API requests as a concurrent batch, and streams every log line live to the browser, so progress is something you watch rather than wait for.

Every raw response is parsed into normalised rows — price, taxes, carrier, segments, baggage, response time — using that partner's mapping config. Prices are converted to a single currency and matched against our own production fares for the same routes, so every partner offer lands next to the number it has to beat. The output is a single self-contained HTML file with sortable tables and charts: open it in any browser, no installs, no exports. Several partners can also run against the same routes in one go, and the pipeline generates a combined comparison report at the end.

Every API Speaks Its Own Dialect

The expensive part of this system was never sending the requests — it was onboarding. Every partner's API returns a differently shaped JSON document. Some nest flight segments inside each offer; others keep offers and flights in separate lists joined by reference IDs. Prices are sometimes bare fares and sometimes tax-inclusive totals. Timestamps arrive in a different format every time. So I made the parser config-driven: one generic parser, plus a per-partner response mapping that tells it where everything lives. That reduced onboarding to producing one config file — and then I built an AI agent pipeline that produces it.

The Response-Mapping Agent

Onboarding a new partner today means uploading two files — their API request collection and a single sample response — and letting a pipeline of four small LLM agents do the rest. Each agent has a narrow job and at most two tools; the heavy JSON never enters a prompt.

The design principle is: never trust the model, verify by execution. The Mapper's draft is never accepted on its own merits — the Validator runs the actual production parser against the sample response and reads the result. If zero rows come back, or prices and timestamps are malformed, it writes a specific list of what failed and the loop sends the Mapper back to fix exactly those issues. Only when real rows parse cleanly does the loop exit. The prompts also encode the domain traps I kept hitting by hand: bare fare versus tax-inclusive total, path syntax the parser actually supports, timestamp format strings confirmed against real values rather than guessed.

How It Worked Out

Nobody searches flights by hand for these evaluations anymore. A multi-date batch across hundreds of routes completes in minutes, benchmark included, and produces a report anyone can open and explore. Evaluations became repeatable — same routes, same dates, re-run next week and compare. And because onboarding collapsed from a developer-day of JSON archaeology to a few minutes of agent compute, we can evaluate far more potential partners than before — the bottleneck is now the decision, not the data.