Master the Classic Question: “What Does a Dip in the Metrics Mean?”

How I answer this data analytics and product execution interview question. With In-N-Out hamburgers 🍔

This graph started going up and to the right … and then it went down and to the left!? Source: Unsplash

It’s the classic data analytics question:

I was once asked it this way:

It’s fair game for data analysts, product managers, and any tech role where you look at graphs.

This question falls in the category of “product execution” interview questions. Product execution can includes a few other kinds of questions, but I’ll address them at the end of this article or direct you to search online.

For this article I’ll focus just on the “dip in the metric” question with the burger restaurants as the example.

My goal is to give you some tips for your next product execution interview. Also to prove to myself and future employers that I can answer this question, though admittedly with the luxury of time.

Here’s the outline:

  1. Ask clarifying questions
  2. Set a framework
  3. Narrow the pain
  4. Hypothesize a cause
  5. Test your hypothesis
  6. Propose a solution
Why In-N-Out’s metrics would ever dip is beyond me. Source: Unsplash

(1) Ask Clarifying Questions

No one expects you to 100% understand the prompt. Especially if you’re nervous and if the business model from the prompt is new to you.

In fact, the prompt is usually intentionally ambiguous. Interviewers expect you to ask clarifying questions. It shows thoughtfulness and humility on your part. For the burger chain example, you can ask:

The tactic here is to “choose the vague keyword from the prompt and narrow it to something more specific.”

The tactic here is to “clarify the business model” or “clarify what are my abilities and resources”. Generally, it can be “am I allowed to bring in prior information that I know beforehand” or whether the focus is on logical decomposition based on core principles.

Keep it to 1 or 2 clarifying questions for now. More than that and it sounds like you’re not smart enough to understand the prompt. More questions also suck time from the rest of the interview, as you’ll have many more questions later.

If the prompt is simple enough to not need questions, you can just recap it to show you understand. Clarifying questions or no, I always end with:

Let me clarify — would you like onions on your hamburger? Source: Unsplash

(2) Set a Framework

Again, these interviews are intentionally ambiguous. They aren’t meant to have one correct right answer.

What the interview is really testing is your ability to structure the problem, as well as your verbal communication abilities. So before you get into it, communicate your plan of attack.

Any structure is fine. A structure you memorize from business school or a business book or website is fine. A structure is a good starting point if you can’t think of anything. But you don’t have to stick to the structure if it’s dragging you down.

For the dip in the metric, I find this structure useful:

  • Narrow the pain
  • Hypothesize a cause
  • Test your hypothesis
  • Propose a solution

Four points is ideal. Three is fine but perhaps shallow. Five and you’re unlikely to hit everything in time.

Remember to watch your time. You want to cover everything you said you would and leave 5 minutes for your questions to the interviewer.

My structure hits all the food groups: carbs, proteins, vegetables, and coke. Source: Unsplash

(3) Narrow the Pain

Here’s where you ask a series of questions with the goal of changing the problem from a vague pain to a specific pain. A specific pain is easier to ascribe a cause and a suggest a solution. Narrowing can be separated into measurement and segmentation.

(3.1) Measurement

Usually the interviewer will reply “it’s sudden.”

Business metrics normally move up and down. Is this dip within the normal amount that the metric fluctuates? There’s no need to impress here with your knowledge of statistics, as the answer will usually be: “It dipped below the normal fluctuating level.”

Usually the answer is “It just happened this week [or this time period]. We don’t know if the problem is fixed yet.”

This fine because the goal is usually “catch, understand, and fix the dip ASAP.” But we have to be very careful about which timeframes we are measuring. It is bad to compare incomplete time periods (partial weeks) to complete time periods (full weeks).”

Sales are typically highly seasonal. For B2C (like burgers), sales are likely to rise on weekends/holidays and dip on weekdays. For B2B, it’s the opposite. Again, your interviewer’s answer is usually “no” or “it’s low even factoring in seasonality.”

Usually the answer is “no it’s totally unexpected”, but it’s worth a shot. If yes, you can short circuit the investigation and just verify it’s the same cause as last time. But then, that wouldn’t be a fun interview question.

So your interviewer will usually just say “no” to all of these questions. But I think these questions do serve an important purpose in the interview and in real life. The goal is to rule out:

  1. Noise
  2. Misinterpretation

That is, you’re checking that the metric (1) dipped below normal levels and (2) was a real dip and not a perceived dip. In other words, it’s a sanity check that we aren’t over-analyzing the data and worrying about nothing.

The logical follow-up is that it dipped because of an unexpected event. Then the next goal is to find that event.

So you’re saying the drive thru line is 3 blocks long, but it’s normally 5 blocks. Right? Source: Unsplash

(3.2) Segmentation

Here’s the fun part where you can slice and dice the data. I lead with a key insight that’s often true in both real life and these interviews:

Now the goal is to find the segment(s) with a large dip. As the target metric is usually sales or revenue, we can logically break down the sales process into 3 types.

  • Type of Customer
  • Type of Product/Service
  • Type of Transaction

Then brainstorm ways within those types to segment:

  • Type of Customer: Your typical demographics (age, gender, etc) for B2C or firmographics (company size, industry, etc) for B2B.
  • Type of Product/Service: Usually by categories of your products. Another good way is by the star product (the most popular) vs the non-star product.
  • Type of Transaction: For burgers, whether the order was made in-person, via an app, over the phone, drive thru vs sit down vs to go, etc. Physical location is common as well. Also how the transaction was paid (cash, credit card, gift card, etc). For B2C/B2B tech, it’s also by native app vs browser usage as well as iOS vs Android.

For the burger shop, say the burger chain had 3 stores across town. A location-based segmentation showed reasonably normal sales in locations A and B, but a big dip in location C.

A big dip is usually a smoking gun and a signal for you to move on, especially as you’re crunched for time in an interview.

I propose we segment customers by whether or not they wear a hat. Source: Unsplash

(4) Hypothesize a Cause

Once you’ve narrowed the pain, then it is easier to assign it a cause.

(4.1) Internal Factors

In general, “can we link the dip to something that was under our control?” For example, questions like:

  • Did we change our processes?
  • Did we change our people?
  • Is there a bug in the system?
  • Is the data pipeline healthy?
  • Did we have a specific campaign, event, or launch recently?

For the burger chain example, I would also:

  • Talk to the store manager. Visit the place in person. See if you notice anything weird or different.

The goal of the on-site visit is to discover if there’s something that wouldn’t be captured in a phone call with the store manager or from a dashboard. For example, a bad smell caused by too much trash or too much bleach used to clean it up.

Based on my own personal experience in real life, internal causes are easier to ascribe and fix. Usually, but not always, they are the cause of these sudden dips. In addition, a data pipeline failure is a common reason why numbers dip. That is, the transactions really did happen, but the database didn’t collect them properly.

In an product execution interview though, they usually make it more interesting by saying:

Ingredient quality is extremely important. It’s the key behind In-N-Out’s success. Source: Unsplash

(4.2) External Factors

In general, “can we link the dip to something outside of our control?” For example, questions like:

  • Seasonality: As mentioned before. Yearly, monthly, weekly, and even intra-day seasonality for some businesses.
  • Weather: Both physical weather and ‘mental’ weather like the news. Could include a bad social media story about your brand.
  • Slowdown: Any kind of industry wide slowdown or recession.
  • Indirect competition: A grocery store opened nearby. A national campaign to eat healthier.
  • Direct competition: A new restaurant opened across the street. A n nearby restaurant ran a promotion.

Direct competition is usually when the interviewer throws you a bone to the root cause. For example:

Or

The airport screwed us over when they changed the flight paths. Source: Unsplash

(5) Test Your Hypothesis

At this point, I’m usually running out of time in the interview. But what I would do in real life is to verify that my guess of the root cause is true.

For example, somehow monitor (if possible) foot and car traffic to the taco shop across the street. You can also approach it using user research, so something like:

  • Ask your current customers about your competitors
  • Ask someone impartial about the state of the market
  • Send a survey to your entire TAM (total addressable market)

In an interview, I think it’s enough to just point out that you would verify and offer one method. In both the interview and in real life, it’s okay (for now) to gloss over it because you really want to get to the solution.

We need to monitor our arch enemy, the taco shop. Source: Unsplash

(6) Propose a Solution

Here’s where you close the loop and propose a solution to bring the metric back to normal. If the root cause can be solved directly (an internal cause), then great. If not (an external cause), then you have to solve it another way.

(6.1) Zoom Out Then Zoom In

The logic here is:

  • Take the specific root cause
  • Generalize to a overall user pain point
  • Change your product-marketing to meet that pain point

For the burger chain example:

  • Taco shop ran a marketing campaign with coupon
  • Customers are price conscious
  • Run your own campaign with a coupon. Develop a “value menu” if it doesn’t exist. If it does exist, reinforce it in marketing.

Another potential example:

  • Taco shop ran marketing campaign (but no coupon)
  • Customers are aware of competitors and are open to switching
  • Run your own brand awareness campaigns to remind customers. Raise the cost of switching through loyalty campaigns, a loyalty program, or superior customer service.

Finally, another one:

  • Poke shop opened next door
  • Customers are looking for healthier options or more variety
  • Introduce new, exciting, healthy products. Introduce ethnic flavors. Remind users of customization options.
The menu is sacred. But you can product-innovate with the secret menu. Source: Unsplash

(6.2) A/B Test It

Ideally you would A/B test a solution: common and relatively easy to do for online software.

For the restaurant prompt though, it might be difficult. You can’t really tell a waiter or waitress to randomly choose a customer, treat them differently, then record that interaction and result somewhere. Testing is still possible, but you just have to deal with the constraints.

If taco shop ran a promo then you can run a promo too. For example, buy-one-get-one-free burger for October. Then to measure you would:

  • Compare to previous month (September): Most direct, but unable to control for seasonal variations. For example, less vacations in the fall.
  • Compare to same month last year: Seasonality can be controlled, but there may be long-term growth or decline in sales.

Both ways (or other comparison methods) have good and bad points. You just have to be careful and interpret it appropriately.

In addition it would be best to wait for the competitor’s promo to end, to reduce interference and be more confident that your intervention alone led to the change. But the boss may be impatient and unwilling to wait.

You know you’re in San Francisco when you see scooters. Source: Unsplash

I Don’t Like This Prompt

Overall, I’m not a big fan of the dip-in-the-metric question. First, it exploits a deep psychological fear: “the fear of losing something is a bigger motivator than the hope of gaining something.”

It also isn’t a common situation in real life, at least in my experience. While it can happen, I more commonly see a different situation:

  • You have a growth metric like transaction count.
  • You changed something in your product or marketing, with the hope that this metric would go up.
  • It didn’t go up. It stayed in the normal range.

The same for a defect metric you wanted to squash but stayed normal. This type of problem has similar analytical-decomposition skills, but tends more towards a “failure of adoption” and getting your product-sense right.

Whenever I give a product execution interview, I tend to ask something like “How would you measure the success of [innovative product X]?”. That question tends towards north star metrics and decomposing that. Similar analytical skills but different context.

Did you know that In-N-Out stays open until 1 am? Source: Unsplash

Wrap Up

In summary, remember this for your next product execution interview:

  1. Ask clarifying questions
  2. Set a framework
  3. Narrow the pain
  4. Hypothesize a cause
  5. Test your hypothesis
  6. Propose a solution

(1) and (2) are just the setup. The rest are actually solving the problem. Finally, good luck interviewing. Stay positive and keep your head up :D

Finally, it’s time to eat. Source: Unsplash

Did I miss anything? Was I totally off the mark somewhere? How would you answer this problem? How do you really feel about In-N-Out burgers? 🍔 Feel free to comment below 😃

If you liked this article, then please clap, follow, and share with others. Thanks for your support!

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store