Ethical Performance Metrics for Responsible AI Evaluation

Abstract

The evaluation of AI system performance has historically focused on predictive accuracy metrics -- AUC, F1-score, RMSE -- that measure technical capability without capturing the ethical dimensions of system behaviour that responsible AI governance requires. As AI systems are deployed in high-stakes contexts where fairness, transparency, privacy, and safety are legally mandated properties alongside accuracy, a comprehensive ethical performance metric suite is needed that enables systematic, comparable, and regulatory-aligned evaluation. This paper proposes the Ethical Performance Metric Suite (EPMS), a validated collection of thirty-two quantitative metrics organised under eight ethical performance dimensions: fairness, transparency, explainability, robustness, privacy protection, accountability, safety, and value alignment. Each metric is operationalised with a formal definition, measurement protocol, normative threshold, and EU AI Act / GDPR compliance mapping. The EPMS is validated through a psychometric study with 52 responsible AI evaluators and applied to benchmark evaluation of 36 AI systems across healthcare, finance, criminal justice, and education domains. Psychometric validation confirms strong inter-rater reliability (mean ICC = 0.83, 95% CI: 0.78-0.87) and convergent validity (r = 0.74 with established transparency instruments). Benchmark results reveal substantial cross-domain variation in ethical performance profiles: healthcare AI achieves the highest aggregate EPMS score (mean = 68.4/100, SD = 9.1) while criminal justice AI scores lowest (mean = 41.7/100, SD = 12.4). The EPMS contributes a standardised, psychometrically validated instrument for responsible AI evaluation, a cross-domain ethical performance benchmark, and an EU AI Act conformity assessment tool aligned with Articles 9, 10, 13, and 15.