- 84 %of cancellations are caught in time
- 0,93AUC of the chosen model
- 5 %of real cancellations: the problem is rare, and that is what makes it hard
The situation
In a subscription business, churn gives no warning. By the time the customer clicks “cancel”, the decision was made weeks earlier, and by then there is no conversation left to have. The retention team has last month's list of cancellations; what it needs is next month's.
What was built
An end-to-end early warning system: data preparation, feature engineering on usage —logins, features used, open tickets, days since the last interaction—, three models compared with time-based validation so as not to cheat with the future, and, for each customer, an explanation of why they are at risk. On top of that, a web application to look up one customer and get back a probability, a risk level and a suggested action.
What came out
The metric chosen was not precision but F2: letting a leaving customer slip away costs far more than calling one who was going to stay anyway. On that criterion, the model catches 84 out of every 100 cancellations. The sensitivity analysis also showed the opposite of what you would expect: past a certain volume, the cost of making contact eats the benefit, so the lever is not calling more people but being sharper about which ones.
What this has to do with you
It is exactly the way of working we sell: a business problem translated into a metric, a model that can be explained customer by customer, and a back-of-the-envelope sum that says when it stops being worth it. No mystery and no black box.
All case studies