How to answer “Tell me about a production incident”
a guide for data engineers.
It is one of the most common questions in a data engineering interview, and one of the easiest to answer badly. Here is what interviewers are really listening for, a structure you can use, and a worked example.
The question behind the question
Nobody wants a perfect record. They want to know how you behave when something breaks and someone else is depending on it.
Judgement under pressure
Did you work out what mattered first, or chase whatever was loudest? Interviewers want to hear how you triaged, not just what you fixed.
Real ownership
Were you accountable, or a bystander with a good view? Say what you decided and what you did, and give the team credit where it is due.
Whether it happens again
Fixing an incident is expected. Preventing the next one is what separates senior answers from junior ones.
Impact, detection, mitigation, root cause, prevention
Five parts, in this order, in about two to three minutes. Lead with impact so the listener knows why to care.
- 01
Impact
What to sayStart with who was affected and why it mattered. One or two sentences, with a number if you have one.
What they listen forThat you understand the business consequence, not only the technical fault.
- 02
Detection
What to sayHow you found out. An alert, a user, an analyst spotting a mismatch? Be honest if a person found it before your monitoring did.
What they listen forHonesty, and whether you later closed the monitoring gap.
- 03
Mitigation
What to sayWhat you did first to stop the damage: paused a refresh, rolled back, warned stakeholders, backfilled. Say what you chose not to do, too.
What they listen forCalm prioritisation, and communication while it was still burning.
- 04
Root cause
What to sayWhat actually went wrong, and how you knew. Show the trail: logs, lineage, the change that broke it.
What they listen forDepth. A root cause is a cause you can point to, not a guess.
- 05
Prevention
What to sayWhat you changed so it cannot recur silently: a check, a contract, an alert, a runbook, a design change. This is the part to spend time on.
What they listen forWhether you improve systems, not just repair them.
What it sounds like
An illustrative answer for a duplicate-data incident. It is not from a real candidate, and the numbers are made up.
The daily revenue dashboard showed numbers about 8% too high for 11 days before anyone noticed. Finance had already used them in a weekly forecast.
Not our alerts. An analyst noticed that orders on the dashboard did not match the payments report. We had row-count checks, but nothing that looked for duplicates.
I paused the downstream refresh and told the finance channel the same hour so nobody kept using the numbers. Then I re-ran the 11 days from raw data with duplicates removed.
An upstream service had changed its retry behaviour and started sending some events twice. Our job appended rows without a unique key, so every duplicate landed in the table.
I switched the load to an idempotent merge on the event id, added a duplicate check that fails the run, and set an alert on day-over-day drift. I wrote it up and shared it with the upstream team, who added the id to their contract.
Notice what it does: it opens with a consequence, admits that monitoring missed it, shows a clear order of actions, points to a specific cause and spends its last breath on what changed for good.
Pick an incident worth telling
Data incidents come in a few common shapes. Choose one where something was at stake and you had a real part.
- Missing or late data that broke a downstream report
- Duplicates or double counting, especially after a retry or a backfill
- A schema change upstream that your pipeline did not expect
- A backfill or migration that went wrong
- A cost spike from a runaway job or an unpartitioned scan
- Silently wrong numbers, the hardest kind, because nothing failed
Mistakes to avoid
The same few errors turn a good incident into a forgettable answer.
- Blaming another team. Describe the cause without assigning fault.
- Skipping the impact. If you never say why it mattered, the story is a debugging anecdote.
- Ending at the fix. Interviewers are waiting for the prevention.
- Playing the lone hero. Incidents are team efforts, and saying so reads as maturity.
- Choosing a trivial incident. Pick one with a real consequence, even a small one.
- Sharing confidential details. Anonymise the client and the data, but keep the scale and the reasoning.
Get the details before you forget them
The hard part is rarely the structure. It is remembering the numbers and the order of events a year later.
- Postmortem or incident documents, if your team wrote them
- The Slack or Teams thread from the day, which holds the timeline you will forget
- Your pager or alert history, and the ticket that tracked the fix
- The pull request that fixed it, and the one that prevented it happening again
This is also the gap in most resumes. In our analysis of 128 real Data Engineer resumes, only about 1 in 5 mentioned an incident, a root cause or an outage at all, even though nearly all described monitoring or data quality work. See the full analysis.
Common questions
- How long should the answer be?
- Two to three minutes. Spend about half of it on root cause and prevention, and keep impact and detection short. Stop and let the interviewer pull on whatever they find interesting.
- What if I have never had a serious incident?
- Use a smaller one that still had a consequence, such as a report that was wrong for a day, or a job that failed and needed a backfill. Interviewers care about how you reason, not how dramatic it was. A personal or open-source project can work if you owned it and someone relied on it.
- Should I prepare more than one?
- Yes, two or three of different types, such as a data quality problem, a performance or cost problem and a failure during a migration. The same story rarely fits every version of the question.
Write the story down the day it happens.
Capture what broke, what you did and what you changed while it is fresh. graph asks the follow-up questions and builds an interview prep sheet from your own stories.