Data Science Interview Challenge

Welcome to today's data science interview challenge! Let’s switch gear today to discuss about experiment design, inspired by my recent AB test work! The following question is open-ended:

How to ensure data quality when running experiments?

---

---

Here are some tips for readers' reference:

Question:

There are many ways to ensure data quality for your AB test. One example I encountered was the need to -

consider the longevity of experiment metrics.

You need to be clear upfront what metric(s) to test on, and how long you need to monitor the metrics in order to make decisions.

For example, when implementing a modification in your customer support strategy aimed at promoting the use of the in-app chat as opposed to traditional phone calls, it’ll be beneficial to measure the long-term impact on NPS (Net Promoter Score) and customer satisfaction.

The immediate way to evaluate the experiment's success is by tracking the drop in the number of support ticket and monitoring call volumes during the trial phase. However, it’s also important to measure NPS and customer satisfaction over a longer time period, so that we account for longer term implications such as the impact on happiness for the customers.

As shown in the above metrics table, we have our primary test metric and we plan to also report on our support metrics for a longer period of time as their trend don’t become obvious immediately.

This means that we have to put in the guardrails to make sure that these secondary metrics are tracked reliably over the entire study period. For the example of NPS score, we’ll need to ensure that the method of generating NPS scores should stay consistent throughout this period.

Another example related, is that you need to -

get your metrics straight before running your experiment!