Ana içeriğe geç

Rehberler / İzleme rehberleri: her kontrol türü, adım adım kurulum

Incident postmortem template, with a filled-in example

An incident postmortem is a blameless written record of an outage: what happened, who was affected, a timeline in UTC, the root cause, and action items with an owner and a due date. Copy the template below, then see it filled in for a fictional online shop.

What a postmortem is for, and when to write one

Google's Site Reliability Engineering book defines a postmortem as a written record of an incident, its impact, the actions taken to resolve it, the root causes, and the follow-up actions that prevent it from happening again. Its purpose is learning. A postmortem that looks for someone to blame teaches people to hide problems.

Decide when a postmortem is required before the next incident, so nobody has to argue about it afterward. Common triggers from the SRE book are:

  • user-visible downtime or degradation beyond a set threshold
  • data loss of any kind
  • an on-call engineer had to step in, for example by rolling back a release or rerouting traffic
  • resolution took longer than a set time
  • a monitoring failure, which usually means someone found the problem by hand

Anyone on the team can also ask for one. Write it while memory is fresh, usually within a few days.

The sections of a postmortem

SectionWhat to writeCommon mistake
SummaryTwo or three sentences: what broke, for how long, who was affected, whether it is fixed.Starting with technical detail before the impact.
ImpactDuration, number of users or requests affected, orders or money lost, SLA effect."Some users" with no number.
TimelineTimestamps in UTC: cause introduced, first symptom, detection, escalation, mitigation, recovery.Mixed time zones, or no entry for detection.
DetectionHow you found out and how many minutes passed.Skipping it when customers reported the problem first.
Root cause and triggerThe underlying flaw, and the event that exposed it.Stopping at "human error".
Contributing factorsWhat made it likelier, longer or worse: a missing alert, an unclear runbook, a slow rollback.A list of people instead of conditions.
ResolutionWhat stopped the impact, and what removes the cause for good.Treating a mitigation as the permanent fix.
What went well, what went badly, where we were luckyHonest notes on the response itself.Leaving out the luck. A near miss is a lesson too.
Action itemsSpecific tasks, each with an owner, a due date and a type: prevent, detect or mitigate.Vague tasks such as "improve monitoring".

Copy-paste template

INCIDENT POSTMORTEM: [SHORT TITLE]

Incident ID: [ID]
Date: [YYYY-MM-DD]
Authors: [NAMES OR ROLES]
Status: [Draft / In review / Complete]
Severity: [as defined by your team]

1. SUMMARY
[Two or three sentences: what broke, for how long, who was affected,
and whether it is fixed.]

2. IMPACT
- Duration: [START] to [END] UTC ([N] minutes)
- Users or customers affected: [NUMBER OR SHARE]
- Failed requests, orders or messages: [NUMBER]
- Other impact: [data loss, SLA breach, support tickets, revenue]

3. TIMELINE (all times UTC)
[HH:MM] - [Event: change deployed, alert fired, person paged, fix applied]
[HH:MM] - [Event]
[HH:MM] - [Event]

4. DETECTION
[How did we find out: alert, customer report, or a person noticing?
How many minutes passed between the start of the impact and the first alert?]

5. ROOT CAUSE AND TRIGGER
Root cause: [the underlying flaw or condition]
Trigger: [the event that exposed it]

6. CONTRIBUTING FACTORS
- [What made the incident more likely, longer or worse]

7. RESOLUTION
Mitigation: [what stopped the impact]
Permanent fix: [what removes the cause]

8. WHAT WENT WELL / WHAT WENT BADLY / WHERE WE GOT LUCKY
- [Note]

9. ACTION ITEMS
Action | Type (prevent, detect, mitigate) | Owner | Due date | Ticket
[Specific task] | [Type] | [Role or name] | [YYYY-MM-DD] | [Link]

10. LESSONS
[What the team learned that is not an action item.]

Filled-in example (fictional)

This incident is fictional. The shop, the numbers and the times are invented to show the format.

Title. Example Shop: slow product pages and failed checkout after release 2026.09.14

Summary. On Monday 14 September 2026, a release made product pages slow for 40 minutes, and checkout returned errors for 29 of those minutes. The release was rolled back and the service recovered. No data was lost.

Impact. Product pages: slow from 14:02 to 14:42 UTC (median 7 seconds, normally 0.8 seconds). Checkout: errors from 14:09 to 14:38 UTC, about 620 failed attempts. No payments were taken twice.

Time (UTC)Event
13:55Release 2026.09.14 is deployed. It adds a "related products" section to product pages.
14:02Product pages start to slow down. Database connections are in use for longer.
14:09The database connection pool is full. Checkout starts returning 503 errors.
14:12The external checkout check fails twice and sends an SMS to the on-call engineer.
14:14The on-call engineer acknowledges the alert.
14:18Status page: "Investigating errors at checkout."
14:24The database dashboard shows the pool at its limit since 14:09. The release is the main suspect.
14:26The decision is made to roll back.
14:33The rollback is complete. It needed several manual steps.
14:38Checkout works again.
14:42Product page response time is back to normal.
14:44Status page: resolved.

Detection. An external check on the checkout page alerted 3 minutes after the first checkout failure and 10 minutes after the slowdown began. No alert covered slow responses, because the monitors only alerted on failed checks. Customers had not yet reported anything.

Root cause. The new section ran one database query for each related product, about 12 per page view. In staging, with 200 products, this was fast. In production, with a much larger catalog and normal afternoon traffic, the queries used the shared connection pool faster than it could free connections.

Trigger. The release at 13:55 UTC, followed by the daily traffic peak.

Contributing factors. The load test did not cover product pages. Product pages and checkout shared one connection pool, so a slow page could block payments. There was no alert for slow responses. The rollback was manual and took 7 minutes.

Resolution. Mitigation: rollback to the previous release at 14:33. Permanent fix: the query was rewritten to fetch all related products in one call. It was released on 16 September after a load test.

What went well. The checkout alert fired within 3 minutes of the first failure. The status page was updated 6 minutes after the alert. The rollback caused no data loss.

What went badly. The slowdown ran unnoticed for 10 minutes. The rollback was slow and manual.

Where we were lucky. The release happened on a weekday afternoon, with the whole team online.

ActionTypeOwnerDue
Rewrite the related-products query as one query with a row limitPreventBackend lead2026-09-16
Alert when product page response time stays above 3 seconds for 5 minutesDetectOn-call lead2026-09-18
Give checkout its own connection poolMitigatePlatform engineer2026-10-02
Add product pages to the load test, with a production-size catalogPreventQA lead2026-09-25
Turn the rollback into one tested commandMitigateRelease manager2026-09-30

A slowdown that later becomes failures is the pattern described in the guide to service degradation.

Writing it without blame

A blameless postmortem assumes that everyone involved had good intentions and did what made sense with the information they had. The SRE book puts it this way: you cannot fix people, but you can fix systems and processes.

  • Describe the release, the alert or the process, not the person. Write "the release added a query that opens one connection per related product", not "the developer wrote a bad query".
  • Do not stop at "human error". Ask why the system let the mistake reach production, and why nothing caught it.
  • Use roles in the timeline ("the on-call engineer") unless names are needed to contact someone.
  • Prefer a few specific action items over a long list. Each needs an owner and a date, and someone has to check that it is done.

Filling the timeline with real data

Memory is poor evidence of when an incident started and ended. Take the timestamps from your records: when checks first failed, when they recovered, and when the first alert went out. HostTracker checks from 300+ checkpoints in 158 cities, so the monitoring history shows whether an outage was global or regional. A HostTracker status page publishes incidents with timestamped updates, which you can paste into the timeline and reuse as the short customer-facing summary.

The detection section is where alerting gaps appear. If customers told you before your tools did, check which contacts and channels were set up. HostTracker sends alerts by email, SMS, voice call, Slack, Telegram and more, and the alert channels guide explains how to choose between them and how to escalate to a second person after a delay. Before you mark the incident closed, run the website down checker, which tests the address live from many locations and shows what each one saw.

Frequently asked questions

What is the difference between a postmortem and a root cause analysis?

A root cause analysis explains why the incident happened. A postmortem is the whole document: impact, timeline, detection, root cause, resolution and action items. The root cause analysis is one section of it.

How long should a postmortem be?

Long enough to be complete, and short enough that people read it. For most incidents that is one to two pages. The example above is about that size.

Who writes the postmortem?

Someone who took part in the response, usually the incident owner. Others who were involved review it and correct the timeline, because they remember details the author does not.

Does every incident need a postmortem?

No. Set criteria in advance, for example downtime above a threshold, any data loss, or a rollback. Smaller events can get a short note in the incident log.

Should customers see the postmortem?

For serious incidents, many teams publish a shorter version: what happened, how long, what was affected and what will change. The internal document keeps the details. State only facts you have confirmed.

Sources and further reading

Şimdi kontrol edin

Ücretsiz kontrolü kendi sitenizde çalıştırın - hesap gerekmez.

Website down checker

Bunu sürekli izleyin

Bir sorun çıktığı anda haberdar olun: HostTracker 300’den fazla konumdan kontrol eder ve size e-posta, SMS, Slack, Telegram ve daha fazlasıyla bildirim gönderir.

HostTracker özellikleri

Bu bölümde daha fazlası: İzleme rehberleri: her kontrol türü, adım adım kurulum