arXiv:2611.04338 · v1
Patching the Protocol: A Taxonomy of Pre-Registration Repairs in LLM-Proposer Experiments
摘要
Pre-registration is increasingly proposed as a trust mechanism for machine-learning research involving LLM-generated content, but its canonical form assumes a frozen protocol can be executed unchanged. Software-intensive studies violate this assumption: the protocol is partly implemented, and implementations carry defects. We report a three-case study of protocol repairs ("patches") from a program where an LLM proposes executable rules that a programmatic verifier gates, across production scheduling, incentive design, and task assignment. Each protocol required exactly one defect-repair patch: three events in total. None modified a numbered judgment criterion, its threshold, or the scoring rule, and none altered already collected data; two changed pre-registered design parameters before any data were collected, each with a recorded rationale. We propose a two-axis taxonomy of patches (defect class: environmental / feasibility / correctness, x timing: pre-data / post-data), introduce a verifiable non-impact claim for post-data repairs, and distill a nine-item checklist; all nine items were exercised by at least one patch event -- an observation of use, not a validation of sufficiency. The credibility of verifier-bound, LLM-proposer studies depends less on whether patches occur than on whether they are auditable.
作者备注：12 pages. Three-case study of protocol repairs in a pre-registered, LLM-proposer/verifier-gated program (production scheduling, incentive design, task assignment): each protocol needed exactly one defect-repair patch; a two-axis taxonomy and a verifiable non-impact claim for post-data repairs. Artifact archive: https://doi.org/10.5281/zenodo.23433826