Safety-Aware Training Objectives for Generative Intelligence

Abstract

Large generative AI systems trained on web-scale corpora absorb not only the beneficial knowledge and capabilities of human culture but also its harmful content -- instructions for dangerous activities, dehumanising language, deceptive rhetoric, and content that violates fundamental ethical norms. Standard RLHF-based safety alignment addresses harmful outputs post-training through preference learning but does not modify the training objectives that determine what capabilities the model acquires during pre-training. This paper proposes the Safety-Aware Generative Training (SAGT) framework, a set of modified pre-training and fine-tuning objectives that integrate safety considerations directly into the generative learning process rather than addressing them solely through post-training alignment. SAGT comprises four training objective modifications: harm-weighted token loss (HWTL) that downweights the gradient contribution of harmful training tokens; capability-safety decoupling (CSD) that separates safety-relevant capability learning into a controllable parameter subspace; safety-consistent pretraining data filtering (SCPDF) that applies automated harm classifiers to curate training data before pre-training; and constitutional pre-training constraints (CPC) that enforce constitutional AI principles as soft constraints on the pre-training objective. SAGT is evaluated on safety benchmarks (TruthfulQA, HarmBench, BBQ bias), capability benchmarks (MMLU, HumanEval), and alignment tax (the quality degradation from safety training). Results demonstrate that SAGT reduces harmful output rates by 61.4% (SD = 5.8%) on HarmBench while retaining 96.2% of baseline capability on MMLU and HumanEval -- substantially improving the safety-capability trade-off compared to post-training RLHF alone. The alignment tax for SAGT is 3.8% (vs. 12.4% for equivalent RLHF-only safety), confirming that safety-at-training-time is more capability-efficient than safety-at-alignment-time. The study contributes the SAGT specification, a Safety-Capability Index (SCI), and empirical evidence for the superiority of training- time safety integration over post-training alignment alone.