The situation
In a subscription business, churn rarely gives warning. By the time a customer clicks “cancel,” they probably decided weeks earlier, so the list the team needs isn't last month's cancellations but next month's. But there's a difficulty that's easy to miss: only 5% of customers actually cancel. With a problem that rare, a model that answered “nobody leaves” would be right almost every time. And it would be useless.
What was built
We prepared the data and built usage features: logins, features used, open tickets and days since the last interaction. We compared three models with time-based validation, so none of them learned from the future. Each prediction is explained customer by customer, because whoever makes the call needs to know why they're calling. On top sits a web app: you look up a customer and it gives you their churn probability, their risk level and a suggested action.
What came out
We chose the model by F2, not by precision, because losing a customer who leaves costs more than calling one who was going to stay. A smoke alarm works the same way: one that goes off too often costs you a minute; one that stays quiet costs you the house. On that criterion, the model catches 84 out of every 100 cancellations. But the sensitivity analysis showed something less obvious: past a certain volume, the cost of making contact eats the benefit. So the gain isn't in calling more people, it's in choosing better who to call.
Choosing the metric is probably the most important decision in a project, which is why on every implementation we agree on it in writing before touching anything. And if the numbers say it doesn't pay off, we tell you.
Our own project, built on a public telecoms dataset adapted to the scenario. The usage features are synthetic and the financial impact is a model, not an invoice. The code is on GitHub (opens in a new tab).