Systems concept
Program Observability
The sensing and state-reconciliation layer of a proposed Program Operating System: evidence, claims, provenance, uncertainty, and 10 signals that can still be gamed.
The problem
Program status is monitoring, in the operational sense of that word. A fixed set of checks, defined in advance, answering only the questions somebody thought to ask when they built the template. Are we on schedule. Are we on budget. Is anything red.
Software operations abandoned that model roughly 15 years ago, for a specific reason: in a system of any real complexity, the failures that hurt are the ones nobody predicted. If your instrumentation only answers pre-registered questions, you find out about the unpredicted failure from a customer.
Programs have exactly that property and none of the response. The failures that hurt are the ones that weren’t on the risk register. The mechanism by which they stay hidden is well understood: work happens in dozens of places, a human samples it on a cadence, compresses it hard, and passes it up through people with an interest in how it lands.
Adding AI to that loop makes the sampling continuous and the compression more fluent. It does not make the result more true. The 3 months of hallucinated analytics that a company ran its strategy on were caught by an accidental sanity check, not by a control.
Simulation bias is the reason this model exists. Observability is the first architectural response: before a system reasons or acts, it should be able to distinguish what was observed, what was claimed, and what it derived.
So the question this model tries to answer: what would you have to capture and reconcile so that somebody could ask an unanticipated question about the state of a program and get an evidence-backed answer without convening a meeting?
Program Observability is the sensing and state-reconciliation subsystem of the proposed Program Operating System. Its job ends at derived state. It does not own the full decision and execution loop.
The model
4 stages, then a handoff to the runtime.
Evidence is whatever the program actually leaves behind: commits, pipeline runs, ticket transitions, capacity allocations, approval records, vendor confirmations, and direct observations. It is the raw material. Evidence can be incomplete, stale, forged, misunderstood, or inaccessible.
Claims are assertions about program reality. “The supplier will ship on September 10” is a claim. The email that supports it is evidence. A claim should carry that evidence and its provenance, observed time, freshness, asserting actor or source, authority, and confidence or uncertainty.
Reconciliation compares claims, applies explicit rules, and preserves contradictions it cannot resolve. Agreement between 2 claims that share the same underlying evidence is not corroboration.
Derived state is the system’s conclusion over reconciled claims rather than a field somebody types. If an input is stale or contradicted, the state degrades visibly instead of quietly aging.
The Program Runtime consumes that state. Policy produces deterministic decisions in some cases. Agents may recommend, prepare, or take bounded actions in others. Some decisions require explicit human authority. The important property is that authority is explicit and accountable. Actions produce events, and events become new evidence for the observability layer.
4 realities that must remain separate
Observed reality is what evidence directly shows. Claimed reality is what an actor asserts. Derived reality is what the system concludes from evidence and reconciliation rules. Authorized reality is what an accountable authority has formally changed or accepted.
For example, these can all be true at once:
- Observed: The supplier wrote “September 10.”
- Claimed: The supplier says shipment will occur September 10.
- Derived: The current shipment commitment misses the program need date.
- Authorized: The official program schedule has not changed.
Collapsing those into one status is how a persuasive representation outruns reality. The system needs
first-class states such as UNKNOWN, UNVERIFIED, CONTRADICTED, STALE, SUPPORTED,
AUTHORIZED, and OVERRIDDEN so uncertainty survives the trip to a decision.
The signals
Claims are the primitive. The 10 signals below are operational measures constructed from claims, grouped by the question they answer. The grouping is doing real work: most program dashboards mix all 3 categories into one color, which is how a program with clean inputs and untrustworthy reporting ends up looking identical to a program with the opposite problem.
I’ve included how each one gets gamed, because any signal used in a performance conversation becomes a target. Writing the gaming vector next to the metric is the only mitigation I know of that doesn’t depend on everyone being virtuous.
Are the inputs sound?
Whether the program has been specified well enough to execute against at all.
| Signal | What it measures | How it gets gamed |
|---|---|---|
| Requirement stability | Rate of change in requirement content after work starts against it, weighted by how much downstream work depends on the changed requirement | Freeze requirements on paper and move the real spec into Slack |
| Ownership clarity | Share of active work items with a single unambiguous owner who has accepted the item, versus items owned by a team name, a distribution list, or nobody | Assign everything to a manager who then holds it |
| Interface compliance | Whether cross-team handoffs match their declared contract in shape, timing, and completeness | Write the contract loose enough that nothing can violate it |
Is the work moving?
Whether execution is progressing, and where it is actually stuck.
| Signal | What it measures | How it gets gamed |
|---|---|---|
| Dependency health | For each declared dependency: is the providing side emitting evidence consistent with the committed date, and how old is that evidence | Re-baseline the date instead of reporting the slip |
| Decision latency | Wall-clock time between a decision being requested and being made, measured from first request rather than from when it reached the decider | Stop logging requests until you’re ready to answer them |
| Execution confidence | A stated confidence interval on completion, scored against outcomes over time so the estimate has a track record | Give wide intervals so you’re never wrong |
| Risk propagation | How far a realized risk travels before it changes another team’s plan, and how long that takes | Absorb the impact locally and never propagate it |
Can we trust what we’re being told?
The category almost nobody instruments, and the reason I built this model.
| Signal | What it measures | How it gets gamed |
|---|---|---|
| Evidence freshness | Age distribution of the evidence behind the current state, surfaced as a property of the state itself | Refresh a timestamp without refreshing the underlying fact |
| Human intervention rate | How often a human overrides, corrects, or discards an automated conclusion, and in which direction | Stop overriding, which reads as agreement and is indistinguishable from disengagement |
| Narrative-to-evidence divergence | Whether claims in the generated summary are supported by the evidence it cites, and whether that evidence is current | Generate narratives only from fields the checker can verify, so the unverifiable claims move to the voice-over |
What it makes visible
If this worked, 4 things that are currently invisible would stop being invisible.
Waiting. Decision latency is the largest hidden cost in most programs I’ve seen, and it’s structurally unreportable today because waiting isn’t an activity anyone owns. Nobody writes “blocked on a decision for 11 days” in a status field, they write “in progress.”
Staleness. Right now a program’s state is equally confident whether it was updated an hour ago or 3 weeks ago. Making age a property of the state changes what a green means.
Absorbed risk. Risk propagation catches the pattern where a team eats an impact locally, reports green, and the downstream team finds out at integration. That’s the watermelon program mechanism, made measurable.
Divergence. The gap between the generated account and the evidence beneath it. This is the one that doesn’t exist anywhere today, and it’s the one that matters most as narratives get more fluent.
Limitations
Stated plainly, because the version of this that pretends to be finished is worse than useless.
Goodhart applies to all 10. Every signal becomes a target the moment it appears in a performance conversation. The gaming column is a mitigation, not a solution. I don’t know whether this is fatal or merely expensive, and that’s the single biggest open risk in the model.
It assumes an instrumentable execution surface. This works best for software programs. It degrades for programs whose critical path runs through vendor lead times, physical installs, regulatory approvals, or a partner who will not give you API access. I believe it degrades to partial coverage rather than to nothing, but I haven’t demonstrated that.
It assumes an organization willing to be observed. Some reporting layers are load-bearing for somebody’s narrative, and a system that derives state from evidence is a direct threat to that. This is a political constraint, not a technical one, and political constraints are the ones that actually kill implementations.
Nothing here is validated. Zero deployments. The signal definitions are what seem right from reasoning about failure modes I’ve watched, and reasoning about failure modes is exactly the activity that produces plausible frameworks that don’t survive contact.
It could easily become governance theater. A dashboard of 10 signals that nobody acts on is worse than 1 color that somebody argues about, because it costs more and looks more responsible.
What I’m testing next
The honest next step is small. The ambitious version requires an organization; the first test needs a controlled codebase with full data access.
Narrative-to-evidence divergence. Generate a weekly summary from commits, issues, and CI results. Separately extract the claims in that summary, check each one against the underlying data, and report the unsupported ones. If that is hard with 1 stakeholder and complete access, it says something important about the enterprise version.
Decision latency, measured from first request. The measurement definition is the whole game here. I want to find out whether “when it was first asked” is even recoverable from normal artifacts, or whether it requires a discipline nobody will adopt.
Evidence freshness as a first-class property. Attach an age to every derived value and see whether it changes any decision I make. If it changes nothing at this scale, that’s evidence the signal only matters above some organizational size, which is worth knowing.
Orion, as the first structured runtime test. The planned experiment will compare this structure against unstructured LLM reasoning when supplier commitments conflict, Jira is stale, evidence is missing, and facts change over time. The full test and its evaluation criteria are described in the Program Operating System page. It has not been run yet.
I’ll update this page as those come back, including the parts that don’t work. The Field Notes are where the intermediate results land.