arXiv:2505.20628 · v5
Position: Adopt Constraints Over Fixed Penalties in Deep Learning
摘要
Deep learning models are increasingly subject to explicit requirements, or constraints, such as safety levels, fairness limits, and sparsity budgets. In practice, such requirements are often handled through fixed weighted-sum penalization: the constraints are added to the task loss with fixed coefficients, and the resulting scalarized objective is minimized. This position paper argues that fixed penalization is often ill-suited to deep learning problems with non-negotiable requirements, for three main reasons. First, in non-convex settings, the penalized and constrained problems are not, in general, equivalent, so a minimizer of the penalized objective need not solve the constrained problem. Second, a penalty coefficient sets an objective-constraint trade-off rather than the constraint level, so finding an acceptable coefficient often takes trial and error, each trial a full training run. Third, fixed penalization weakens hard requirements into soft penalties to be traded off against task performance. As a result, compliance is no longer built into the problem, and requirements can be quietly relaxed. We therefore argue that, when a deep learning problem specifies non-negotiable requirements, one should solve the constrained problem itself instead of a surrogate defined by fixed penalization.
作者备注：Accepted at NeurIPS 2026 (Position Paper Track). 10 pages of main text plus appendix; 3 figures, 4 tables. Code: https://github.com/merajhashemi/constraints-vs-penalties