Thinklytics

Domo Analytics · Technology & SaaS · Austin, TX · 10 weeks

Customer churn prediction model deployed to production in 10 weeks

A Series B analytics startup struggled to deploy their churn prediction model because their data pipelines kept breaking. We overhauled their data infrastructure, fixed the pipeline issues, and got the model running in production within 10 weeks. In three months, the model flagged $2.6M in at-risk ARR, helping the customer success team retain 340 accounts they would have otherwise lost.

Challenge

The data science team created a churn model that predicted correctly 78 of every 100 times in tests, but it never went live. The data pipeline broke multiple times a week because CRM and product usage data kept changing formats. Without reliable data, the customer success team couldn’t identify which accounts were at risk.

Approach

We rebuilt the data pipeline to detect schema changes and fix them automatically, removing the main cause of deployment failures. Next, we tested the model on a three-month holdout set, raising accuracy to 84 of every 100. Finally, we set up daily scoring in production and added Slack alerts to notify customer success managers when accounts hit the churn risk threshold.

Outcome

The model went live in week 10. Within three months, it flagged $2.6M in at-risk ARR. The customer success team used these alerts to save 340 accounts that data showed were likely to churn. The pipeline has operated continuously for 18 weeks without issues.

The model was killing it, but then our data pipeline totally blew up.

The team had a good model, but their data pipeline was a ticking time bomb. Every time the CRM added new fields or changed data types, the pipeline would break, silently, no alerts, nothing. We stepped in and built automated schema checks to catch those changes instantly. Now, we catch issues early before they blow up into major headaches.

We put schema drift detection in place to catch data changes as soon as they happen.

Here’s the deal. We dropped schema drift detection straight into the pipeline with Great Expectations. When the upstream schemas changed, the system caught it right away and even fixed the easy stuff on its own. Thanks to that, pipeline failures? Nearly wiped out.

How We Turned Slack Into Our Secret Sauce for Customer Success

Here’s the deal: we realized the model’s value depended on how the team actually used it. So, we plugged it into Slack and sent a daily rundown of the riskiest churn accounts. For each account, we flagged the top three risk factors. This made it dead simple for the team to jump on problems right away. Now, they’re responding to 94 of every 100 alerts. Not too shabby, huh?

Results

  • $2.6M At-risk ARR identified in Q1
  • 340 Accounts retained in first quarter
  • 78 to 84 correct per 100 Model accuracy improvement
  • 10 wks From rebuild to production

Our data science team built a good model, but the pipeline kept failing, so we couldn’t put it to use. Thinklytics fixed the pipeline, improved the model, and got it running in production within 10 weeks. We retained 340 accounts in the first quarter alone.

VP of Customer Success, Series B Analytics Startup

In the client's terms

The question

Which of this work should a person still be doing?

The situation

The data science team had a churn model that scored 78 correct in 100 in testing and had never reached production. The pipeline feeding it broke several times a week when CRM and product usage schemas changed. Customer success had no list of at-risk accounts.

What we found

The model was never the blocker. A pipeline that breaks weekly cannot carry a daily prediction, and the team had been asked to improve model accuracy when the problem was upstream fragility.

What it produced

Live in week 10. $2.6M in at-risk ARR flagged in the first quarter. Customer success retained 340 accounts the model had identified. The pipeline has run 18 weeks without an interruption.

The roadmap

The pipeline was rebuilt to detect schema changes and adapt rather than fail, which removed the cause of every previous deployment attempt. The model was then revalidated on a three-month holdout, which also lifted accuracy to 84 in 100. Daily scoring went into production with Slack alerts to the customer success manager who owns the account, not to a shared channel.

The work behind this result

Related case studies

Frequently asked questions

Why does a good machine learning model never reach production?

Often the pipeline rather than the model. At Domo Analytics the churn model scored 78 correct in 100 in testing but the pipeline feeding it broke several times a week whenever CRM or product usage schemas changed, silently. The team had been asked to improve model accuracy when the problem was upstream fragility.

How do you stop data pipelines breaking when upstream schemas change?

Detect the change and handle the routine cases automatically. Domo's pipeline was rebuilt with schema drift detection through Great Expectations, which took failures from several a week to close to none and has run 18 weeks without interruption.

What return does a churn model produce?

Domo's flagged $2.6M in at-risk ARR in its first quarter live, and customer success retained 340 accounts from those alerts. The alerts go to the customer success manager who owns the account rather than to a shared channel, which is why 94 of every 100 get a response.