In the fast-paced world of global travel, customer support is more than just answering emails. It’s about precision, speed and cultural nuance. At Sixt, a mobility giant operating in over 100 countries, the challenge of managing a massive fleet and a diverse global clientele means that traditional flat classification systems simply don’t cut it.
If you’ve ever wondered how a massive corporation handles thousands of multilingual inquiries without losing a customer in a “routing loop”, you won’t want to miss our latest featured talk. This session dives deep into the architecture of Hierarchical Classification and how Large Language Models (LLMs) are replacing outdated manual workflows. It will also include practical insights drawn from SIXT’s experience applying hierarchical classification in the context of customer support.
The Cost of a Misclassification
What happens when a system misclassifies an email? In a flat system, a ticket about a billing error might land in the queue for mechanical issues. When a customer sends an email the first step is classification, and the stakes are high:
“If there is a misclassification the email stays there for a while, a human agent picks it up, realizes it’s the wrong queue, and routes it again. The response is delayed, and the customer experience suffers”.
To combat this, Sixt moves away from simple labels to a three-plus level hierarchy. This approach ensures that a query isn’t just tagged as “account”, but it is funneled through the specific parent nodes, down to granular child nodes.
The Modern ML Dilemma: Classic vs. LLM
One of the most compelling segments of this talk covers the change from classic Machine Learning (ML) to LLM-based classification. While classic ML is cost-effective, it struggles with the constant changes in market trends, new feature releases or even with language. This is known as Data Drift.
Polezhaev explains how LLMs allow them to:
- Skip Massive Training Sets: Transitioning from thousands of hand-labeled samples (which are often prone to human error) to high-quality, expert-driven category descriptions.
- Handle Complex Prompts: Utilizing “Other” categories as a strategic tool for early stopping, ensuring the model doesn’t “force-fit” an inquiry into the wrong category when the context is missing.
- Automate Quality Assurance: Using AI to audit the logic of the hierarchy itself, checking for “discriminability” which is the assurance that two categories are distinct enough for a machine to tell them apart.
Navigating the “Path” to Accuracy
How to measure success when classification has multiple levels? The talk provides a masterclass in modern evaluation metrics. The session explores:
Over-classification vs. Under-classification: How to measure success when a classification isn’t just right or wrong, but potentially “partially correct” across different levels? The talk provides a masterclass in modern evaluation metrics, exploring the nuances of Over-classification (going too deep into the tree) versus Under-classification (stopping too early).
Hierarchical Precision and Recall: Petr introduces the concept of Path-based Metrics, where success is measured by the intersection of the predicted path and the “ground truth” path. This ensures the system isn’t just accurate on paper, but truly effective in the real world.
From a Data Scientist, a Product Manager or a Customer Experience Lead, this video offers a pragmatic, according to Petr Polezhaev – case study on connecting cutting-edge AI and real-world business logistics. There are insights about internal booking systems integration with AI to provide context-aware classification and how to manage the transition from single-label to multi-label sub-trees.
This preview is part of our Premium Videos.
Sign up to see the whole presentation about: Exploring Metrics and Challenges in Hierarchical Multi-Class Classification with Large Language Models
Presented by: Petr Polezhaev, SIXT
