Skip to content

Shipping a forward fix instead of rolling back

When a clean revert is not possible (a migration ran, the suspect PR sits on top of unrelated work or revert fails CI), the engineer ships a minimal forward fix instead. They get a thumbs-up from the secondary on-call posted on the PagerDuty status update and record the agreement to ship forward on the incident timeline.

Category
Tags
githubhotfixforward-fixmigrationminimal-diff
What and why
The observed behaviour and the reasoning behind it.
Behaviour
Reasoning
Cause and effect
What initiates this pattern and what it produces.
Trigger
Outcome
Standard operating procedure
Step-by-step instructions to reproduce this pattern.
1

GitHub

From the affected service's repository on the default branch, create a new branch named hotfix/<incident-id>.

Use the PagerDuty incident ID as the suffix, for example hotfix/PT4ZXKR. This makes the branch self-documenting. Anyone reading git log months later can trace the change to the original incident in PagerDuty without needing the post-mortem.

Expected: A new branch hotfix/<incident-id> is checked out locally or via the GitHub web editor.

2

GitHub

Make the minimum change needed to restore service.

Resist the temptation to fix related issues, refactor the touched function or add tests for the broader area. Every additional line is new risk on a hotfix that will skip the full test suite. If the smallest possible fix is two lines, ship two lines and open a follow up PR for the rest.

Expected: The branch contains a tightly scoped diff, ideally under 30 lines of production code.

3

PagerDuty

Post a status update on the incident timeline asking the secondary on-call to thumbs up the forward fix.

Example: 'Migration ran on api-gateway #4892, going forward fix. Two line patch on the retry policy default. Priya, ack to ship?' The secondary already has the situational read on the incident. Routing through PagerDuty rather than a GitHub review thread keeps the conversation in the audit trail and saves the secondary from a tab change.

Expected: The PagerDuty timeline carries the request and the secondary's acknowledgement in reply.

4

GitHub

Open the hotfix PR with a [hotfix] title prefix and the [hotfix] label applied.

The label is what the deploy pipeline reads to decide whether to use the abridged test suite. The prefix is the human signal for reviewers and dashboard filters. The body of the PR should link the original suspect PR, the PagerDuty incident and explain in one or two sentences why a revert was not viable.

Expected: The hotfix PR is open with the [hotfix] label applied and CI is running the abridged suite.

5

GitHub

Merge the hotfix PR once CI is green.

Use Squash for hotfixes rather than Merge. The branch typically has multiple small commits from rapid iteration during the incident and a single squashed commit on main is easier to read in the post-mortem.

Expected: The hotfix PR is merged to the default branch and the deploy pipeline has started.

6

GitHub

Add a comment to the original suspect PR explaining the forward fix.

Reference the hotfix PR by number, name the change and indicate whether the original PR is now considered fixed or still requires a follow up. Example: 'Forward fix shipped in #4910 (PD-PT4ZXKR), retry policy default reverted to 3. Will follow up to make this a config rather than a constant.'

Expected: The original PR has a comment linking the forward fix and indicating whether further work is expected.

Related patterns
How this pattern connects to other patterns in the library.
Supporting actions
Actions that provide evidence for this pattern.
Hotfix #4910 on api-gateway, migration meant clean revert wasn't viable
Forward fix on checkout-service hotfix/PT5MN8L, 2-line patch
Skipped revert on billing-worker, shipped hotfix #3820 instead
Asked Priya for thumbs up in PD before hotfix, 90 sec turnaround
Metadata
Timestamps and identifiers.
EvidenceObserved 8 times across 3 connections
ApplicationsGitHub, PagerDuty
First seen19 Feb 2026, 22:51
Last seen22 Apr 2026, 11:04
Questions

Frequently asked questions

Speak to the founder

Henry Denton, founder of FusedFrames

Get a demo. Watch a live capture, then an AI agent query the result.

Ask anything. Pricing, security or integrating with your stack.

No purchase obligation

Start capturing

Record in minutes. Install once and work as normal.

Plug AI agents in. One API call from any AI agent stack.

Refund on unused credits if you cancel