Model Transparency Metrics for Responsible AI Evaluation

Abstract

Model transparency -- the degree to which an AI system's internal logic, decision process, and performance characteristics can be understood, verified, and communicated to relevant stakeholders -- is widely recognised as a foundational property of responsible AI. However, the field lacks a validated, comprehensive set of quantitative metrics for measuring transparency across its multiple dimensions, impeding systematic evaluation, benchmarking, and regulatory compliance verification. This paper proposes the Model Transparency Metric Suite (MTMS), a validated collection of nineteen metrics organised under five transparency dimensions: algorithmic transparency, data transparency, outcome transparency, process transparency, and governance transparency. Each metric is operationalised with a formal definition, measurement protocol, and normative threshold derived from regulatory requirements and expert consensus. The MTMS is validated through a psychometric study involving 44 responsible AI evaluators from eleven institutions and applied to benchmark evaluation of 28 publicly documented AI systems across healthcare, finance, law enforcement, and social media content moderation domains. Psychometric validation confirms strong inter-rater reliability (mean ICC = 0.84, 95% CI: 0.79-0.89) and convergent validity with existing transparency assessment instruments (r = 0.76, p < 0.001). Benchmark results reveal substantial variation in transparency profiles across systems and domains, with healthcare AI achieving the highest aggregate MTMS scores (mean = 71.4/100, SD = 8.3) and law enforcement AI the lowest (mean = 42.6/100, SD = 11.7). The MTMS contributes a standardised, psychometrically validated instrument for responsible AI transparency evaluation, directly applicable to EU AI Act Article 13 conformity assessment.