Skip to content

Stage Overview annotation pie cutover

The Phase 5.2 first slice is one stage-annotation endpoint and one independently reversible Stage Overview consumer: the current annotation pie, and nothing else on that page. The area chart, the annotation leaderboard, the allocation progress panel and every screening surface continue to use their existing contracts.

This is the sibling of the screening-only Project Overview cutover and follows it deliberately closely. Where the two consumers' rules are the same, they are the same because the second one reuses the first one's seams rather than restating them.

The endpoint

GET api/projects/{projectId}/stages/{stageId}/annotation-stats returns one StageAnnotationStats — the same value object the legacy full-stats response carries in its stageAnnotation collection, for the requested stage. A materialized answer and a fallback answer are byte-identical on the wire, so the flag is a pure substitution rather than a new contract.

The route requires project view authorization and the stage's ViewAnnotationProgressGraph permission, both rechecked on the loaded Project before either statistics source is consulted.

Within the controller, an unknown project and a stage the project does not have return the same bare 404, taken before any permission is evaluated, so the stage id cannot be used to probe. End to end the responses are not identical, because the project-view policy runs first: an unknown project id is answered by AuthorizationHandler with a 404 carrying a project with projectId ... not found body, and a project the caller may not view is answered 403. Only an unknown stage inside a project the caller can view reaches the controller's bare 404. Hiding project existence from an unauthorized caller is the shared policy's job and is unchanged by this slice.

Its dedicated materializedProjectStatisticsStageOverview flag defaults to false, is declared for both the API and the project-management host, and requires the materializedProjectStatisticsPages kill switch, global serving and the stage annotation family in the runtime catalogue. Both consumer switches, writes, serving, the annotation family and the explicit project allowlist are evaluated by StageAnnotationStatisticsQuery before it touches storage, and the bundle reader enforces durable freshness, fences, write epochs, catalogue/source/digest compatibility, the one-snapshot predicate and authorization inside its own snapshot.

There is exactly one free-gate decision, and the query owns it. StageStatisticsController reads no materializedProjectStatistics* flag of its own: it asks IStageAnnotationStatisticsQuery.IsMaterializedReadRequested, which is the same PreSnapshotGate the read itself runs, so the endpoint and the read it guards cannot disagree about whether this surface is switched on. This mirrors the screening consumer, where ReviewController gates on IProjectScreeningStatisticsQueryAdapter.IsMaterializedReadRequested.

With the flag off the endpoint's answer is byte-identical to today's numbers. With either page gate off the gate answers no and nothing is read: no snapshot is pinned on the projection's behalf, no bundle is fetched, and the endpoint answers from the guarded authoritative query, which runs the production StudyStatsQuery.GetFullProjectStatsAsync pipeline — the same $facet aggregation and the same mapper the Stage Overview renders today — and returns the requested stage's section from it. The stage-annotation section of that response is investigator-independent (its facets carry no investigator predicate), so the section this route returns for a member equals the section the page's existing full-stats call renders for that member.

The guarded authoritative query

The authoritative delegate is lazy: a Fresh materialized response invokes no study aggregation.

Every authoritative response, including a disabled-consumer request, opens a pinned read-only Mongo snapshot. It reads the current Project without the repository aggregate cache, rechecks Project.View and ViewAnnotationProgressGraph, and reads the global and project control rows before running the facets in that same session. Inclusion and definition-rewrite fences, the legacy inclusion-job flag and durable-mode disagreement return HTTP 503 with a typed reason; revoked permissions return 403; an unpinnable snapshot also returns 503. No refusal executes the aggregation. These admission reads are required source-consistency work even while the consumer flag is off, so performance comparisons must include them.

The fallback deliberately computes the whole FullStats bundle and takes one stage's section from it, reusing IProjectScreeningSourceReader so this path and the rebuild path cannot diverge from the live query. It therefore makes no facet-count claim: the saving this consumer can demonstrate is on the materialized path, not on its fallback. A narrower stage-only aggregation would be a second transcription of the same formulas and is deliberately not attempted here.

A refusal never authorizes use of the request's cached Project settings, and it is never converted back into raw facets.

Row validation, and what it reuses

The served row is validated by ProjectStatisticsDerivedSummaries.StageAnnotation — the same decoder the coherent derived summaries use — which checks row identity, publication state, provenance, catalogue and source versions, the configuration digest, the content digest and the family's counter keys. The consumer keeps no second copy of that predicate; a row the decoder rejects falls back with a distinct RowRejectedByDecoder reason so a rollout dashboard can tell corruption apart from "nothing is switched on". A response carrying the all-zero checkpoint identity is not auditable and falls back before the row is decoded at all.

Nothing on that path can break the page: a refusal, a bounded capacity failure, a decoder rejection and an unexpected exception all return the guarded authoritative section with a recorded reason. The two deliberate exceptions are a cancellation, which is the caller going away, and the authoritative query's own typed refusals, which are the source being genuinely unavailable and must reach the caller as a 503 rather than as numbers of unknown freshness.

A permission fix this slice required

StagePermission.IsStageAuthorized resolved the caller's ProjectMembership before evaluating the project-level grant, so an explicitly authorized nonmember — an application claim group, a public project — raised ArgumentOutOfRangeException instead of being admitted, and an ordinary nonmember raised it instead of being denied. The project grant is now evaluated first and the membership lookup is the non-throwing one. A caller with neither grant is simply not authorized.

This is an unflagged behaviour change on every stage-gated route — AuthorizationHandler, SignalRAuthorizationHandler, ProjectAuthorizationContext and Project.GetProjectPermissionReport all reach it through Stage.IsAuthorizedForActivity. A plain non-member moves from a 500 to a denial; a non-member carrying an application claim group the stage permission names moves from a 500 to being allowed. No shipped default is widened: every stage activity in ResourceSecurity.json has AllowAllApplicationUsers: false and an empty AllowedApplicationGroupNames, so the second case is unreachable until a deployment deliberately adds such a grant. Both directions are now pinned by direct tests.

Browser behaviour and rollback

With materializedProjectStatisticsPages and materializedProjectStatisticsStageOverview both on, the Stage Overview annotation pie renders from this endpoint. With either off nothing is requested and the pie renders from the existing store aggregation, so a rollback is immediate and needs no data change.

The component owns only that pie, and it is mounted from a @defer (when panelOpenStateAnnotation) block, so a collapsed annotation panel issues no request at all: mat-expansion-panel renders its body eagerly, and the defer block is what keeps the poll off until the panel is actually open. The panel is expanded on first load, so the ordinary case still requests immediately; once opened, the block stays rendered, which is @defer's contract, so collapsing the panel again does not stop the poll. It requests the current answer on mount and every 30 seconds; requests time out after 15 seconds; a failure hides the previous counts rather than showing numbers whose freshness is unknown, and retries on the next interval. A response whose project or stage does not match the request is refused. Changing stage or leaving the view cancels the pending request and the refresh timer. This is bounded polling pending the separately planned SignalR invalidation consumer, not a replacement for that protocol; a hidden browser tab may throttle timers.

Not in this slice

  • No stage-annotation backfill or rebuild entry point. ProjectStatisticsAdminController routes administrative backfills by family over the registered backfill services, and no stage-annotation service is registered. Until one lands, an operator has no supported way to establish the Fresh baseline this consumer would serve, so the flag cannot be activated even in a pilot. The read path, its gates and its fallback are complete and proven; the maintenance half is the next slice.
  • No per-member breakdown. The Stage Overview pie needs only the stage-scoped section; the membership-stage annotation family is untouched.
  • No activation, allowlist or configuration change. Every materializedProjectStatistics* flag stays off and ProjectStatistics:ProjectAllowlist stays empty.
  • No SignalR invalidation consumer (Phase 5.4) and no exports consumer (Phase 5.5).
  • No performance evidence. Reproducible before/after read benchmarks are a separate artifact; the correctness tests below assert no timing improvement on a shared host.

Local correctness proof

The real Mongo replica-set suite compares the served section to the legacy full-stats section for the same stage on a corpus with real tally cells, and covers the disabled consumer, an unlisted project, an unmaterialized stage, a caller without the stage graph permission, a nonmember, an unknown project, an unknown stage, a definition-rewrite fence and a durable-mode disagreement. The Core suite pins the query's whole decision table, including that a closed gate touches no storage and that a cancellation is never laundered into an authoritative answer. Endpoint tests cover lazy materialized reads, fallback, the gate's answer in both directions — including that a closed gate attempts no read at all — refusal mapping and the permission boundary. Browser tests cover the materialized, legacy and flag-off modes, the bounded refresh, the timeout, a mismatched response, an immediate rollback, the four flag combinations of selectStageAnnotationPieMaterialized, and that a collapsed annotation panel issues no request while an open one does. StagePermission.IsStageAuthorized's own decision table is pinned directly, and the deployed ResourceSecurity.json is asserted to grant no stage activity to a non-member.

Reproduce the focused checks from the worktree root:

dotnet test src/libs/project-management/SyRF.ProjectManagement.Core.Tests/SyRF.ProjectManagement.Core.Tests.csproj --filter 'FullyQualifiedName~StageAnnotationStatisticsQueryTests'
dotnet test src/libs/project-management/SyRF.ProjectManagement.Mongo.Data.Tests/SyRF.ProjectManagement.Mongo.Data.Tests.csproj --filter 'FullyQualifiedName~StageAnnotationStatisticsQueryTests'
dotnet test src/services/api/SyRF.API.Endpoint.Tests/SyRF.API.Endpoint.Tests.csproj --filter 'FullyQualifiedName~StageAnnotationStatisticsEndpointTests|FullyQualifiedName~RuntimeFeatureFlagMaterializedStatistics'

From src/services/web, run:

pnpm exec ng test --no-watch --include='src/app/stage/stage-overview/**/*.spec.ts'

Rollout also requires #3371's administrative mode-transition surface and the stage-annotation maintenance entry points named above. Until both are in place, keep this consumer disabled and do not admit a project to the allowlist.