Deliberationv4.6.1

How severe is a wrong PR description when the Terraform change it describes breaks only invalid inputs?

Question, reviewed by the architect expert on codex, gemini, grok and or:glm-5-3-flashx. It follows up terraform-aws-modules/terraform-aws-rds-aurora/pull/556.

converged after 4 rounds

Recovered from the logs of a run on 2026-11-01 on Deliberation 4.0.0, run e6d7dbbe, tool consensus-step. Reviewer text is verbatim, so it can disagree with itself and with the verdict line the engine read.

The prompt

Question

DECISION UNDER REVIEW: the severity of rule `compat.claim-contradicts-analysis` when the code break it names is only LOW, and whether it should block a pull request.

Sent to every reviewer

Full text, verbatim (3,425 characters)
DECISION UNDER REVIEW: the severity of rule `compat.claim-contradicts-analysis` when the code break it names is only LOW, and whether it should block a pull request.

BACKGROUND. A private tool reviews GitHub pull requests on Terraform modules and posts one comment under a maintainer's name. Its rule `compat.claim-contradicts-analysis` (owned by Check A, compatibility) fires "when a compatibility claim in the title or the body contradicts Check A's own conclusion". Today it is ALWAYS HIGH, "never CRITICAL", and is emitted every time next to the code finding (they are "two defects in one change: the code breaks callers, and the prose tells a maintainer it does not ... a maintainer who reads only the description merges it"). In the verdict ladder (first match wins) a HIGH finding blocks merge at rule 2 ("changes requested"); a MEDIUM blocks at rule 6; LOW findings are optional nits; unknowns (checks that could not run) give "inconclusive" at rule 4.

THE CASE. terraform-aws-modules/terraform-aws-rds-aurora#556, a +1/-1 change: `vpc_security_group_ids = compact(concat(aws_security_group.this[*].id, var.vpc_security_group_ids))` becomes `concat(...)`. The PR body (its "Breaking Changes" section) says "the only behaviour removed is the silent filtering of empty-string entries". The review found: besides empty strings, null elements of the caller's `var.vpc_security_group_ids` (type list(string)) are no longer dropped either, and a null element makes the plan fail at the head (observed with the real aws provider in a plan fixture). So the description is incomplete (it names empty strings only, and says nothing of nulls), the code break is rated LOW (compat.expression-behaviour-changed: nulls/empty strings are not valid security group ids; the variable description says security group ids), and the contradicts-analysis finding was rated HIGH, which turned the whole review into 'changes requested before merge' for a documentation sentence. Earlier runs with the same evidence produced 'inconclusive' with two LOW findings; the HIGH appeared only in one reviewer run, from the rule text. The owner's complaint: reviews of tiny changes are far too heavy and not practical; a block over a missing sentence in a PR description feels disproportionate.

OPTIONS (weigh at least these): (A) keep HIGH always (a wrong description makes a maintainer merge a break); (B) severity follows the break: the contradicts-analysis finding is exactly one band below the code finding it pairs with, never above (code break CRITICAL -> HIGH, HIGH -> MEDIUM, MEDIUM -> LOW, LOW -> LOW), still always emitted with the code finding and still rendered; (C) it never blocks: fixed LOW (an 'edit the description' nit) unless the code break is CRITICAL/HIGH; (D) split the finding in two kinds: 'the description claims safety that the analysis refutes' (claim false: HIGH) versus 'the description omits a behaviour' (incomplete, not contradicted: MEDIUM or LOW, rule `compat.description-incomplete`); (E) something better. Constraints: the verdict stays a deterministic function of the reviewer's findings (severity is chosen by the reviewer under fixed rule text), the rule must remain testable with one reviewer instruction, and a reader must not be misled into merging a real break because of an optimistic description. Give a recommendation with the exact new rule text (one paragraph) and the effect on the PR 556 verdict.

Round history

Round history
RoundBlind verdictReviewersAfter adjudicationOutcome
Round 1 4m 03sREQUEST CHANGEScodex REQUEST CHANGES[1]gemini REQUEST CHANGES[3]grok REQUEST CHANGES[5]or:glm-5-3-flashx APPROVE[3]REQUEST CHANGES revisedNot converged, 1 of 4 approved
Round 2 2m 58sAPPROVEcodex REQUEST CHANGES[2]gemini APPROVEgrok REQUEST CHANGES[4]or:glm-5-3-flashx APPROVE[1]REQUEST CHANGES revisedNot converged, 2 of 4 approved
Round 3 3m 13sAPPROVEcodex REQUEST CHANGES[1]gemini REQUEST CHANGES[3]grok REQUEST CHANGES[3]or:glm-5-3-flashx APPROVEREQUEST CHANGES revisedNot converged, 1 of 4 approved
Round 4 2m 40sAPPROVEcodex APPROVEgemini APPROVEgrok APPROVEor:glm-5-3-flashx APPROVEAPPROVEConverged, 4 of 4 approved

Round 1

Blind verdict. Before any reviewer answered, the host agent recorded REQUEST CHANGES with 2 issues of its own: Follow the break one band lower; treat omission as at most LOW.

  • codex REQUEST CHANGESgpt-6.1-sol, 1 issue, 1m 24s

    The recording holds no summary or response text for this answer.

  • gemini REQUEST CHANGESgemini-3.8-flash-high, 3 issues, 1m 58s

    The recording holds no summary or response text for this answer.

  • grok REQUEST CHANGESgrok-4.7, 5 issues, 4m 02s

    The recording holds no summary or response text for this answer.

  • or:glm-5-3-flashx APPROVEz-ai/glm-5.3-flashx, 3 issues, 16s

    The recording holds no summary or response text for this answer.

Round 1 rulings: every issue, with the reason the host agent wrote
IssueRaised byRulingReason
correctness Always-HIGH makes a LOW break block through prose alonecodex, gemini, grok, host agentacceptFive voices agree. Adopt banding: the description finding is one band below the paired code finding.
ambiguity Contradiction versus omission is undefined; wording like 'only' decides itgemini, grok, or:glm-5-3-flashxacceptRule text must define it: a claim is contradicted when it asserts something the analysis refutes (including a closed set such as 'only'); an omission is a behaviour the description never mentions. Under banding the definition no longer decides blocking, only the wording of the fix, so divergence stops mattering for the verdict.
correctness Option D alone would not fix PR 556 because of the word 'only'or:glm-5-3-flashxacceptRight, so banding (B) is the mechanism; D is dropped as a separate rule to keep one rule and one reviewer instruction.
security Prose severity must not exceed or replace the paired code findinggrokacceptBanding gives that: the description finding is never above the code finding minus one band, and it is still always emitted with it, so a missed code finding is still a Check A failure, not a prose one.
ops Severity was non-deterministic across runs; add a regression testgrok, or:glm-5-3-flashxacceptAdd the PR 556 evidence as a reviewer-level fixture: expected LOW description finding (or none above LOW), verdict not rule 2.
scope Blocking on prose goes beyond stopping an optimistic description from hiding a blocking breakgrokacceptBanding keeps the gate exactly where the code break is: with a LOW break the description finding is LOW.

Why the round closed this way. codex, gemini and grok voted REQUEST_CHANGES and openrouter:glm-5-3-flashx voted APPROVE, with the host also at REQUEST_CHANGES. The host accepted all 6 merged issues and revised the rule to option B (banding).

revision Option B (banding, one band below the paired code finding, never above, still always emitted) with a contradiction/omission definition that only changes the fix wording; fixture for PR 556.

Round 2

Blind verdict. Before any reviewer answered, the host agent recorded APPROVE: Banding by the paired code finding is deterministic, keeps both findings, and removes the disproportionate block; implementation list is complete.

  • codex REQUEST CHANGESgpt-6.1-sol, 2 issues, 1m 25s

    The recording holds no summary or response text for this answer.

  • gemini APPROVEgemini-3.8-flash-high, 0 issues, 2m 05s

    The recording holds no summary or response text for this answer.

  • grok REQUEST CHANGESgrok-4.7, 4 issues, 2m 57s

    The recording holds no summary or response text for this answer.

  • or:glm-5-3-flashx APPROVEz-ai/glm-5.3-flashx, 1 issue, 7s

    The recording holds no summary or response text for this answer.

Round 2 rulings: every issue, with the reason the host agent wrote
IssueRaised byRulingReason
correctness 'banded' is not a valid severities column value; registry and merge enforce enumerated severities and would raise LOW back to HIGHcodexacceptVerified need: the registry column lists HIGH, MEDIUM, LOW for this rule and the facts merge floor (merge.go) must not raise it back; the implementation includes the facts registry sync.
ops prefix.md line 225 independently instructs fixed HIGH; regenerate itcodexacceptYes: edit the source reference and run assemble.sh; skills-test checks it.
ambiguity 'Both emitted every time' conflicts with 'omission gets no finding'grokacceptBind: when the contradiction trigger fires, both findings are emitted; an omission fires nothing.
correctness Severity undefined when a claim is refuted but no compat.* code finding is pairedgrokacceptThe finding exists only paired with a code finding; with none, it is LOW and cannot block.
correctness 'A blocking break still carries a blocking description finding' is false for HIGH -> MEDIUM and MEDIUM -> LOWgrokacceptDrop the guarantee; only CRITICAL -> HIGH keeps rule 2. The code finding itself is what blocks.
ops Pin the PR 556 fixture; drop the 'banded' tokengrok, or:glm-5-3-flashxacceptRegistry value is the enum HIGH, MEDIUM, LOW; the rule text carries the banding. Fixture: LOW description finding, no rule-2 block, rule-4 inconclusive.

Why the round closed this way. codex and grok voted REQUEST_CHANGES on the banded rule; gemini and openrouter:glm-5-3-flashx approved. The host accepted all 6 issues and revised the rule again.

revision Enum severities in registry (HIGH, MEDIUM, LOW) with banding in rule text; emitted only with a paired code finding; no pairing -> LOW; blocking guarantee only for CRITICAL->HIGH; prefix regenerated; facts merge floor synced; PR 556 fixture pinned.

Round 3

Blind verdict. Before any reviewer answered, the host agent recorded APPROVE: The banded rule with an enum registry value, a pairing requirement and a pinned PR 556 fixture is consistent and implementable.

  • codex REQUEST CHANGESgpt-6.1-sol, 1 issue, 1m 51s

    The recording holds no summary or response text for this answer.

  • gemini REQUEST CHANGESgemini-3.8-flash-high, 3 issues, 3m 10s

    The recording holds no summary or response text for this answer.

  • grok REQUEST CHANGESgrok-4.7, 3 issues, 1m 56s

    The recording holds no summary or response text for this answer.

  • or:glm-5-3-flashx APPROVEz-ai/glm-5.3-flashx, 0 issues, 10s

    The recording holds no summary or response text for this answer.

Round 3 rulings: every issue, with the reason the host agent wrote
IssueRaised byRulingReason
correctness check-a-compatibility.md severity bullet (line 25) and verdict.md (line 59) still state fixed HIGH; the plan said they staycodex, gemini, grokacceptThree voices, verifiable. Both are listed for edit, preserving the unrelated migration-path rule in line 25.
ambiguity 'Emitted only together with a code finding' contradicts 'with no paired code finding it is LOW'gemini, grokacceptDelete the LOW fallback: a contradiction is by definition a claim the analysis refutes, so a refuting compat.* code finding always exists; with none, the rule does not fire.
correctness Multiplicity: one description finding per contradicted claim, banded from the code findings that refute that claim onlygemini, grokacceptAdopt exactly: one finding per contradicted claim; severity from the highest compat.* code finding that refutes that claim; unrelated compat.* findings never raise it.

Why the round closed this way. codex, gemini and grok voted REQUEST_CHANGES; only openrouter:glm-5-3-flashx approved. The host accepted all 3 issues and revised the rule a third time.

revision One description finding per contradicted claim, banded only from the code findings that refute it; no LOW fallback (no refuting code finding = rule does not fire); check-a line 25 and verdict.md line 59 edited.

Round 4

Blind verdict. Before any reviewer answered, the host agent recorded APPROVE: Rule text is now self-consistent (one finding per refuted claim, banded from refuting code findings, no fallback), and the edit list covers check-a, quoted-claims, verdict.md, registry, facts merge, prefix and fixtures.

  • codex APPROVEgpt-6.1-sol, 0 issues, 1m 42s

    Approve option B as specified. Effort: Short. Confidence: high. Top risk: an unsynchronized registry pin or stale fixed-HIGH instruction could restore unwanted escalation; the planned sync and regression checks address it.
    Full response, verbatim
    Approve. The plan resolves the earlier pairing, multiplicity, severity, and synchronization gaps. No critical security, correctness, scope, ambiguity, performance, or operational issue remains.
    
    1. Apply the listed source edits, regenerate `prefix.md`, and run the skills checks.
    2. Verify all four severity mappings, no finding without refuting code, and no escalation from unrelated findings.
    3. Confirm merge preserves LOW/MEDIUM and PR 556 remains inconclusive at rule 4. Preserve existing ladder precedence: a CRITICAL/HIGH pair may reach rule 0 before rule 2.
    
    **Effort:** Short (1–4h).  
    **Confidence:** High, based on the inspected rules, severity clamp, and render fixtures.
    
    <SUMMARY>
    Approve option B as specified. Effort: Short. Confidence: high. Top risk: an unsynchronized registry pin or stale fixed-HIGH instruction could restore unwanted escalation; the planned sync and regression checks address it.
    </SUMMARY>
    
    VERDICT: APPROVE
  • gemini APPROVEgemini-3.8-flash-high, 0 issues, 1m 24s

    The Round 4 plan thoroughly resolves previous banding, registry schema, and refutation trigger gaps for compat.claim-contradicts-analysis. Effort: Short (1-4h). Confidence: high. Top risk: prefix.md exceeding the 92,160-byte limit after regeneration via assemble.sh, mitigated by checking byte count immediately post-assembly.
    Full response, verbatim
    ### Essential
    
    - **Bottom line**: The Round 4 plan resolves all outstanding blockers from prior rounds, establishing a sound banding model, strict refutation semantics, and comprehensive test and registry synchronization. The rule logic cleanly couples claim findings to refuting code findings while preventing severity inflation in the merge and registry layers. The plan is complete, well-bounded, and ready for execution.
    - **Action plan**:
      1. Update [check-a-compatibility.md](file:///check-a-compatibility.md#L25), [quoted-claims.md](file:///quoted-claims.md), and [verdict.md](file:///verdict.md#L59) to replace fixed-HIGH text with the banding specification while preserving unrelated migration rules.
      2. Update [rule-facts.md](file:///rule-facts.md) to set the severities column to "HIGH, MEDIUM, LOW" and document the Final Step deduplication carve-out.
      3. Synchronize the facts registry and merge floor to support LOW and MEDIUM instances without upward promotion.
      4. Update [comment-format.md](file:///comment-format.md) and associated render tests to reflect re-rated or CRITICAL-paired findings.
      5. Regenerate [prefix.md](file:///prefix.md) with `assemble.sh` and verify file size remains within the 92,160-byte limit.
      6. Execute fixture verification for PR 556 to confirm code and description findings both emit at LOW with no rule-2 block and an inconclusive rule-4 verdict.
      7. Finalize the two-step pin bump for `SKILLS_COMMIT` and `REGISTRY_SHA256`.
    - **Effort**: Short (1-4h).
    - **Confidence**: high.
    
    ### Expanded
    
    - **Why this approach**:
      1. Eliminates false-positive blocking by ensuring description findings never exceed the severity of their refuting code finding (only CRITICAL code refutations yield a blocking HIGH finding).
      2. Resolves the omission versus contradiction ambiguity by requiring at least one refuting code finding before the rule can fire.
      3. Maintains registry schema and merge consistency by explicitly enumerating valid severities ("HIGH, MEDIUM, LOW") and clamping the merge floor.
      4. Preserves architectural dedup semantics through an explicit Final Step carve-out so paired findings are not collapsed.
    - **Risks**:
      1. *Prefix size expansion*: Assembling updated rule text into [prefix.md](file:///prefix.md) could breach the 92,160-byte threshold. *Mitigation*: Run a post-assembly size check in CI; trim surrounding commentary in [quoted-claims.md](file:///quoted-claims.md) if budget headroom is tight.
      2. *Merge floor regression*: Multi-rule PR evaluations could inadvertently promote LOW/MEDIUM description findings if registry sync logic is incomplete. *Mitigation*: Run the PR 556 fixture test across full multi-rule merge passes before finalizing pin bumps.
    
    <SUMMARY>
    The Round 4 plan thoroughly resolves previous banding, registry schema, and refutation trigger gaps for compat.claim-contradicts-analysis. Effort: Short (1-4h). Confidence: high. Top risk: prefix.md exceeding the 92,160-byte limit after regeneration via assemble.sh, mitigated by checking byte count immediately post-assembly.
    </SUMMARY>
    
    VERDICT: APPROVE
  • grok APPROVEgrok-4.7, 0 issues, 2m 37s

    Approve round 4. Banding (CRITICAL→HIGH, HIGH→MEDIUM, MEDIUM→LOW, LOW→LOW), one description finding per refuted claim, no fire without a refuting compat.* code finding, both emitted with a Final Step carve-out, and blocking description severity only for CRITICAL→HIGH. Registry floor, fixed-HIGH doc edits, prefix assemble, two-step pins, and the PR 556 LOW/LOW fixture are specified. Effort: Short. Confidence: high. Top risk: implementers reading “one band below” as “do not emit” for LOW — follow the map and fixture instead.
    Full response, verbatim
    The round-4 plan matches the settled model and is safe to implement as written.
    
    **Bottom line**: Banding, per-claim firing, and the no-refuter no-fire rule are consistent with rounds 1–3. Enumerated severities, the merge floor, the fixed-HIGH doc edits, prefix regeneration, and the PR 556 pin are all specified. The explicit map is normative, including LOW→LOW.
    
    **Action plan**:
    1. Treat the severity map as the spec: CRITICAL→HIGH, HIGH→MEDIUM, MEDIUM→LOW, LOW→LOW, never above the highest compat.* finding that refutes that claim.
    2. Fire once per refuted title or body claim; do not fire on omission or when no compat.* finding refutes that claim; emit the pair and keep the Final Step carve-out.
    3. Replace only this rule’s fixed-HIGH text in check-a line 25 and verdict.md line 59; leave the migration-path HIGH bullet unchanged.
    4. Set registry severities to HIGH, MEDIUM, LOW and set this rule’s merge floor so LOW/MEDIUM are not raised; confirm with PR 556 (LOW code, LOW description, no rule-2 block, rule-4 inconclusive).
    5. Update quoted-claims, SKILL.md severity mentions, and comment-format/render fixtures (pair a HIGH description with CRITICAL, or re-rate by the map).
    6. Regenerate prefix.md with assemble.sh and keep it ≤ 92160 bytes; two-step pin SKILLS_COMMIT and REGISTRY_SHA256.
    
    **Effort**: Short.
    **Confidence**: high — behavior is fully determined by the rule text plus the pinned fixture.
    
    **Why this approach**: The LOW fallback contradiction is gone. Multiplicity and banding are per contradicted claim, so unrelated compat.* findings cannot raise severity. Only CRITICAL→HIGH is a rule-2 blocking description finding. Doc, registry, and generated prefix stay on one severity story.
    
    **Risks**: “One band below” is false for LOW, but the map and PR 556 pin clamp to LOW — do not invent a fourth severity. “reviewer SKILL.md mentions” is vague; change severity language only, do not add new rules.
    
    <SUMMARY>
    Approve round 4. Banding (CRITICAL→HIGH, HIGH→MEDIUM, MEDIUM→LOW, LOW→LOW), one description finding per refuted claim, no fire without a refuting compat.* code finding, both emitted with a Final Step carve-out, and blocking description severity only for CRITICAL→HIGH. Registry floor, fixed-HIGH doc edits, prefix assemble, two-step pins, and the PR 556 LOW/LOW fixture are specified. Effort: Short. Confidence: high. Top risk: implementers reading “one band below” as “do not emit” for LOW — follow the map and fixture instead.
    </SUMMARY>
    
    VERDICT: APPROVE
  • or:glm-5-3-flashx APPROVEz-ai/glm-5.3-flashx, 0 issues, 7s

    This reply has no summary line. The full response is below.

    Full response, verbatim
    The plan is internally consistent and resolves every issue raised in rounds 1-3: no LOW fallback (rule does not fire without a refuting code finding), enumerated severities in the registry match the banding outputs (HIGH, MEDIUM, LOW), multiplicity is one finding per contradicted claim banded only from findings that refute that claim, the three textual voices (check-a line 25, verdict.md line 59, prefix.md via assemble.sh) are all listed for edit, the dedup carve-out and rule-2 blocking condition are stated, and the PR 556 fixture (LOW code finding → LOW description finding, no block) exercises the banded path end to end. The pin bump and facts merge floor are covered.
    
    Minor non-blocking notes: the "Fix is a corrected sentence" wording is slightly odd when the claim sits in the title rather than the body, but it is interpretive, not ambiguous enough to block; and "highest" among multiple refuting findings is clear enough given the explicit "never above" bound.
    
    VERDICT: APPROVE
    
    No critical issues.

No reviewer listed a critical issue, so there was nothing to rule on.

Why the round closed this way. All four voices and the host voted APPROVE with no critical issues, so the loop converged in round 4.

Outcome

CONVERGED in 4 round(s) (confidence: low)

Type to search the site.