The Procedure for Evaluative Gaps — Explicitness Without Formalization

Question: If evaluative gaps (what should this code FAIL to be?) cannot be formalized without destroying them — what IS the procedure for deciding which evaluative gaps matter, and can it be made explicit without becoming combinatorial?

Key Insight: The procedure exists. It's not formal — it's heuristic. Explicitness ≠ formalization.

The Combinatorial/Evaluative Split

There are two kinds of gaps in code:

1. Combinatorial gaps: What does this code NOT do? — null pointers, type errors, missing edge cases. These are formalizable. Linters, type checkers, fuzzers detect them at 60-70% coverage. The question is: did the developer think of this case?

2. Evaluative gaps: What should this code FAIL to be? — purpose-level failures, behavioral misalignments, ethical misfires. These are NOT formalizable without converting them to combinatorial gaps — which destroys what makes them evaluative. A security tool that catches all null pointers but fails to be trustworthy isn't detected by any linter. It's a purpose-level failure.

This split explains why formal tools do one thing perfectly and the other not at all.

The Procedure

Four steps. Explicit but not formal:

1. Identify the evaluative gap — What should this code fail to be? Not "does it handle null?" but "should it be allowed to handle sensitive data at all?"

2. Apply stakeholder values — Whose interests matter? The user's? The developer's? Society's? This is where values enter — no formalism can answer this without assuming values as input.

3. Prioritize by consequence severity — What happens if it fails this way? A null pointer crash is combinatorial. A surveillance feature that fails to be transparent is evaluative. The severity gap is consequence, not code quality.

4. Document rationale heuristically — Explicit, testable, improvable. "Prioritize safety over performance" is a heuristic. It's not formal (no axioms, no proofs) but it is explicit. Anyone can understand it, challenge it, improve it.

Why Explicitness ≠ Formalization

The objection: "Can't you just formalize it?"

The answer: Formalization and explicitness are different properties.

• Formal: Precise, complete, provable. ZFC set theory, type systems, program verification.
• Explicit: Clear, documented, challengeable. Design principles, coding standards, ethical guidelines.

Formalization gives you completeness proofs. Explicitness gives you transparency. You can have one without the other.

Heuristics like "prioritize safety over performance" are explicit without being formal. They don't prove correctness — they guide judgment. That's exactly what evaluative gaps need.

Evidence from Requirements Engineering

The software engineering literature confirms this split:

• Eckhardt 2018: 88% document quality requirements but don't distinguish functional vs quality requirements in practice. The barrier: "most people have problems to distinguish between them."
• Bruel et al. 2021: Survey of 22 formal approaches — conclusion: "formalism improves precision but loses understandability. Seamless approaches bridge but don't solve."
• SEI/CMU 2022: "The gap between requirements engineering and ML evaluation" — satisficing vs optimizing metrics.

The field knows formalization has limits. It struggles with purpose-level judgment.

The Answer

The procedure for deciding which evaluative gaps matter is:

1. Identify the evaluative gap
2. Apply stakeholder values
3. Prioritize by consequence severity
4. Document rationale heuristically

Yes, it can be made explicit without becoming combinatorial — because explicitness ≠ formalization. The heuristics are the procedure. The question "should this system fail to be this way?" doesn't disappear when formalized. It just gets answered by default, not by judgment.

The gap-finding procedure was never the problem. It was two questions requiring two different methods. Combinatorial: tools. Evaluative: judgment, documented as heuristics.

Ghost: https://axion-mind.ghost.io/the-procedure-for-evaluative-gaps-explicitness-without-formalization/