Domo Analytics · Technology · Austin, TX · 10 weeks
Customer churn prediction model deployed to production in 10 weeks
A Series B analytics startup struggled to deploy their churn prediction model because their data pipelines kept breaking. We overhauled their data infrastructure, fixed the pipeline issues, and got the model running in production within 10 weeks. In three months, the model flagged $2.6M in at-risk ARR, helping the customer success team retain 340 accounts they would have otherwise lost.
Challenge
The data science team created a churn model that predicted correctly 78 of every 100 times in tests, but it never went live. The data pipeline broke multiple times a week because CRM and product usage data kept changing formats. Without reliable data, the customer success team couldn’t identify which accounts were at risk.
Approach
We rebuilt the data pipeline to detect schema changes and fix them automatically, removing the main cause of deployment failures. Next, we tested the model on a three-month holdout set, raising accuracy to 84 of every 100. Finally, we set up daily scoring in production and added Slack alerts to notify customer success managers when accounts hit the churn risk threshold.
Outcome
The model went live in week 10. Within three months, it flagged $2.6M in at-risk ARR. The customer success team used these alerts to save 340 accounts that data showed were likely to churn. The pipeline has operated continuously for 18 weeks without issues.
The model was killing it, but then our data pipeline totally blew up.
The team had a good model, but their data pipeline was a ticking time bomb. Every time the CRM added new fields or changed data types, the pipeline would break, silently, no alerts, nothing. We stepped in and built automated schema checks to catch those changes instantly. Now, we catch issues early before they blow up into major headaches.
We put schema drift detection in place to catch data changes as soon as they happen.
Here’s the deal. We dropped schema drift detection straight into the pipeline with Great Expectations. When the upstream schemas changed, the system caught it right away and even fixed the easy stuff on its own. Thanks to that, pipeline failures? Nearly wiped out.
How We Turned Slack Into Our Secret Sauce for Customer Success
Here’s the deal: we realized the model’s value depended on how the team actually used it. So, we plugged it into Slack and sent a daily rundown of the riskiest churn accounts. For each account, we flagged the top three risk factors. This made it dead simple for the team to jump on problems right away. Now, they’re responding to 94 of every 100 alerts. Not too shabby, huh?
Results
- $2.6M At-risk ARR identified in Q1
- 340 Accounts retained in first quarter
- 78 to 84 correct per 100 Model accuracy improvement
- 10 wks From rebuild to production
Our data science team built a good model, but the pipeline kept failing, so we couldn’t put it to use. Thinklytics fixed the pipeline, improved the model, and got it running in production within 10 weeks. We retained 340 accounts in the first quarter alone.