Scale post-implementation reviews: lightweight evidence packs, routing rules and conversion flows for portfolio learning

Most PMOs already run post-implementation reviews. That's not the problem. The problem is that the output goes to die in a SharePoint folder that gets opened maybe twice a year, usually during audit season. The lesson gets written down. The lesson never changes anything. Six months later a nearly identical project overruns for the exact same reason, and someone writes a nearly identical lesson.

If you're managing a handful of projects, you can compensate with memory. Someone who was there remembers the last time you underestimated integration testing on a vendor-hosted platform and flags it. But once you're past 30–40 projects a year, tribal memory stops working. The person who remembers is on a different program, or they left. Scaled PIR PMO lessons capture is the discipline that replaces memory with a routing system — so a lesson learned on one project actually reaches the estimator working the next one.

This piece is narrow on purpose. It's about the mechanics: how often to run PIRs, what evidence to collect without drowning teams, how to route each lesson to the person or model that can use it, and how to convert raw observations into adjustments to your prioritization scoring and estimation baselines. Worked examples included.

Why lessons don't convert into anything

The failure isn't in the review meeting. It's in the gap between "we noticed something" and "the next estimate reflects it." That gap usually comes down to three things.

The first is that PIRs happen too late and too heavy. A lot of teams schedule the review 60–90 days after go-live, block a two-hour workshop, and produce a 12-page report. By the time that report exists, the people who made the actual decisions have moved on, the details are fuzzy, and the document is too long for anyone downstream to extract anything useful. Nobody re-reads a 12-page retrospective while scoping a new project.

The second cause is that lessons have no address. "Communication with the vendor could have been better" — okay, who owns that? Which template changes? Which gate gets a new checklist item? A lesson without a destination is just commentary. This is the single biggest reason lessons pile up: they're written as observations, not as changes to a specific artifact.

The third is that estimation and prioritization models get treated as fixed. The scoring rubric was set two years ago. The estimation baseline for "cloud migration, medium complexity" hasn't moved even though the last four cloud migrations all ran 20–30% over. The model is stale, and there's no formal path to update it from what PIRs surface.

Fix those three and PIRs stop being a compliance ritual.

PIR cadence: match the review to the risk, not the calendar

Running every project through the same review process is where most PMOs quietly waste effort. A $40k internal tooling project and a $2M platform replacement do not deserve the same review depth. What tends to work is a tiered cadence tied to project size and risk profile.

Two things matter here. First, run light and standard PIRs sooner than most guidance suggests. Two to four weeks after go-live, decisions are still fresh and people can recall the actual sequence of events. Waiting 90 days trades accuracy for a false sense of "letting the dust settle." The dust settling mostly means the memory fading.

Second, the variance trigger. Any project that lands more than roughly 20% off its cost or schedule baseline automatically escalates to a deep review regardless of size — because that's exactly the project your estimation model got wrong, and the one most worth learning from. A project that came in on plan teaches you very little. The overrun is the free tuition.

The lightweight evidence pack

The evidence pack is where scaled PIR programs live or die. If collecting evidence takes a project team half a day, they'll skip it or fill it with fluff. The goal is a pack a PM can put together in 20–30 minutes because most of it already exists.

A workable pack has five components, no more:

1. 1
   Variance snapshot — planned vs actual on cost, schedule, and scope. Pulled from your tracking tool, not re-typed. One screen.
2. 2
   Top 3 deviations — the three things that most drove the variance, each in one or two sentences. Not ten. Three forces prioritization.
3. 3
   Decision points — 2–4 moments where a different call would have changed the outcome, with what was known at the time. This is the highest-value artifact and the one most people skip.
4. 4
   Estimate delta — for the biggest work packages, what was estimated vs what it actually took, and a one-line reason for the gap.
5. 5
   Reusable evidence — links to actual artifacts (the vendor contract clause that bit you, the test plan, the risk log entry). No re-writing, just pointers.

Notice what's not in there: a narrative summary, a list of thank-yous, a page of generic "communication could improve" filler. Those are the parts nobody downstream uses.

The estimate delta section deserves emphasis. A typical example: a data migration work package was estimated at 15 person-days and took 26. The one-line reason: "source data quality far worse than sampled — 40% of records needed manual cleanup." That single line, routed correctly, is worth more than the entire rest of the report, because it's directly reusable for the next migration estimate.

Routing rules: give every lesson an address

A lesson is useless until it lands somewhere that can act on it. Routing rules assign each lesson type to a specific destination and owner.

Here's a routing scheme that holds up at scale:

1. 1
   Estimation gaps → estimation baseline owner. Any estimate delta over a threshold (say, work packages off by more than 25%) gets logged against the relevant estimation category for later model adjustment.
2. 2
   Prioritization mis-scores → scoring rubric owner. If a project scored high on the rubric but delivered thin benefits, the criteria may be mis-weighted. Route it to whoever maintains the scorecard.
3. 3
   Process/gate failures → the gate owner. If a stage gate let something through that should've been caught, the fix is a checklist or gate criteria change, owned by the gate owner.
4. 4
   Vendor/contract issues → procurement + contract template owner. Recurring clause problems become template updates.
5. 5
   Capability/skill gaps → resource/capability lead. Repeated shortages point to hiring or training, not to a document.

The rule of thumb: if a lesson can't be routed to a specific owner and a specific artifact, it's not a lesson — it's a comment, and you should either sharpen it or drop it. That single filter cuts the noise in most lessons logs by more than half.

“

Set a two-week acknowledgment SLA for routed lessons to ensure owners don't let inputs sit unreviewed.

Routing also needs an SLA. A lesson routed to the estimation owner should be acknowledged within a couple of weeks and either accepted as a pending model input or rejected with a reason. Without that, routing just moves the pile from one folder to another.

Conversion rules: turning lessons into model changes

Routing gets a lesson to the right owner. Conversion is the step where enough lessons accumulate to justify actually changing the model. You don't want to re-baseline your estimation model every time one project runs over — that's overfitting to a single data point. You want a threshold.

A conversion rule looks like this: when three or more independent lessons point at the same estimation category in the same direction, the baseline for that category gets adjusted.

Worked example — estimation baseline

Say your estimation library has a category "third-party API integration, medium complexity" with a baseline of 20 person-days. Over eight months, PIR evidence packs surface:

1. 1
   Project A

   estimated 20, actual 28 — "undocumented rate limits forced redesign"

2. 2
   Project C

   estimated 18, actual 25 — "sandbox behaved differently from production"

3. 3
   Project F

   estimated 22, actual 30 — "auth flow more complex than vendor docs implied"

Three independent projects, all overrunning the same category by roughly 35–40%, all for related "the vendor's docs understate the real work" reasons. That crosses the conversion threshold. The baseline moves from 20 to about 27 person-days, and a note attaches to the category: "add a spike for vendor API validation before committing estimates." Now every future estimate in that category starts from a number that reflects reality, and the estimator gets a nudge to de-risk it.

Worked example — prioritization rubric

Prioritization conversion works the same way but on scoring weights. Suppose your intake rubric gives heavy weight to "strategic alignment" and light weight to "delivery confidence." Over a year, several high-scoring, high-alignment projects deliver late and thin on benefits. PIR packs keep flagging that alignment was strong but delivery risk was underweighted at intake.

Three or four of those, and the conversion rule triggers a rubric change: raise the weight on delivery-confidence factors, or add a hard gate that prevents a project from scoring "high priority" if delivery confidence falls below a floor. This is where PIR learning connects directly to how you allocate — and it pairs naturally with tracking whether the promised benefits actually showed up, which is its own discipline covered in operationalizing benefits realization. The PIR tells you the estimate was wrong; benefits tracking tells you whether the project was worth prioritizing in the first place. Together they tune both halves of the intake decision.

The end-to-end workflow, in plain terms

A project hits closure. Based on its tier, the appropriate PIR triggers automatically. The PM assembles the lightweight evidence pack — mostly pulled from existing tracking data, 20–30 minutes of work. The pack's lessons get tagged by type (estimation, prioritization, gate, vendor, capability). Routing rules push each tagged lesson to its owner with an SLA. Owners acknowledge and log lessons as pending model inputs.

Here's a quick visual of that flow.

This diagram maps the trigger, routing, and conversion steps in one view.

Quarterly — which works for most portfolios — the estimation and prioritization owners review the pending inputs and apply conversion rules: where three or more independent lessons converge on the same category and direction, the model gets adjusted, versioned, and communicated back to PMs.

The versioning matters. When you change an estimation baseline, record what changed, why, and which projects triggered it. When someone asks "why did the migration baseline jump 35%?", the answer should be three named projects and their evidence — not "the PMO felt like it." That traceability is also what keeps the whole thing credible during a portfolio review, where you'll want to defend why numbers moved; that connects well to how you'd structure a decision-ready portfolio health review.

A real scenario

A mid-sized professional services firm — somewhere in the 45–55 projects a year range, mostly client implementations — had a lessons-learned process on paper. PIRs happened, reports got filed. But estimators kept using the same baselines they'd used for years, and roughly a third of projects were coming in 15–25% over on effort. The reviews described the overruns accurately every single time. Nothing changed.

They didn't add more process. They stripped it down. The 12-page report became the five-part evidence pack. They added type-tagging and routing so estimation gaps went straight to the two people who owned the estimation library. And they set up the quarterly conversion review with the three-lesson threshold.

Within about two quarters, the estimation library had adjusted baselines on four categories that were chronically underestimated — client data cleanup, third-party integration, UAT cycles, and one deployment category. The share of projects landing more than 20% over dropped from roughly a third to somewhere around 15–18% over the following year. Nothing exotic. The reviews had always contained the right information. It had just never been routed anywhere it could bite.

The interesting side effect: PMs started taking evidence packs seriously because they saw the baselines actually move. When people watch their input change the model, they stop treating the review as paperwork.

When this makes sense — and when it doesn't

This system earns its keep once you're running enough projects that no single person can hold the pattern in their head. Below roughly 20–25 projects a year, a lighter approach and a decent memory can carry you, and the routing and conversion machinery may be more overhead than it's worth.

It's a bad fit if your project data is too inconsistent to pull reliable variance snapshots. If planned-vs-actual is unreliable, your evidence packs will be garbage, and garbage lessons produce garbage model changes. Fix the data reliability first, then layer this on top.

And it won't work if leadership won't let the models actually change. If the estimation baselines are politically frozen — because someone committed to a number and doesn't want it revisited — then the conversion step has no teeth, and you've just built a more efficient way to write reports nobody acts on. The whole point is that the model is allowed to move when the evidence says it should.

The one thing to get right

Every lesson needs an address and a threshold. An address so it reaches the specific owner and artifact that can act on it. A threshold so you change the model based on a pattern rather than a single bad project.

Cadence and evidence packs are the plumbing. Routing and conversion are what turn a pile of retrospectives into a portfolio that actually estimates and prioritizes better next quarter than it did last quarter. That's the whole game — closing the loop between what you learned and what you do differently.

Cadence and evidence packs are the plumbing. Routing and conversion are what turn a pile of retrospectives into a portfolio that actually estimates and prioritizes better next quarter than it did last quarter. That's the whole game — closing the loop between what you learned and what you do differently.