Scaling Effects on AI Fairness: An Empirical Analysis of Stereotypical Bias in State-of-the-Art Transformer-Based Models

Abstract

As Large Language Models (LLMs) become more integrated into our daily lives, understanding their potential for social bias is a critical area of research. This paper presents a comparative analysis of bias in four small-scale and four large-scale LLMs, including several state-of-the-art models. In this study, these eight models were tested against a dataset of 200 questions designed to probe common social stereotypes across eleven categories, such as gender, race, and age. Then each of the 1,600 responses were classified as “Biased,” “Unbiased,” or a “Refusal” to answer. Our analysis reveals that the large models were significantly less biased (54.6% bias rate) than their smaller counterparts (67.8% bias rate), suggesting that increased model scale may contribute to a reduction in stereotypical outputs. In contrast, the small models were far more likely to refuse to answer sensitive questions (38.5% refusal rate vs. 8.9% for large models), indicating a fundamentally different approach to safety alignment. It was found that, while there was a slight negative correlation between a model’s refusal rate and bias rate, the relationship was not statistically significant, challenging the assumption that a reticent model is necessarily a fair one. Perhaps most importantly, it was observed that a huge range in performance even among the large models, with bias rates spanning from 20.1% to 85.9%. Since all the models tested are based on the same fundamental Transformer architecture, our findings suggest that social bias in LLMs is less a product of their architecture and more a reflection of the data, fine-tuning, and alignment strategies used to create them.

Keywords: Social Bias, Large Language Models (LLMs), Model Scale, AI Fairness, AI Alignment, Empirical Analysis