Data Science Interview Challenge

Welcome to today's data science interview challenge! Continuing from last week’s challenge:

Question 1: In the setting of LLM applications, how do you measure whether your new prompt is better than the old one?

Question 2: Why does this matter?

Image credit: Source

---

Here are some tips for readers' reference:

Question 1:

Continuing from the previous week's interview challenge, I'm adding the aspect that was omitted last week.

Introducing new prompts often leads to varied outcomes across different scenarios. The conventional approach to evaluating a model's success, typical in traditional machine learning, doesn't directly align with the nature of generative models. Metrics like accuracy (or the ones we talked about last week) might not seamlessly apply, as determining correctness can be subjective and challenging to quantify.

At a broader level, there are two key focal points to consider:

1. Curate an Evaluation Dataset Incrementally: Develop a dataset tailored to your specific tasks. This dataset will aid in evaluating prompt performance during its development and this data should be built out incrementally.

2. Identify an Appropriate Metric or Framework for Evaluation: Select a suitable metric or framework to gauge performance. We covered this a little bit last week in Question 2:

This question as a whole is a complex topic and I’ll write a more detailed post about it next time!

Question 2:

Some key points about why this matters:

1. LLMs makes a lot of mistakes.

2. You can tweak the prompts that results in improvements in some cases, but not necessarily means improvements in general.

3. Building trust with your users is important, ultimately you need to maintain the performance of their tasks.

Let’s check out how this is explained by Josh!