Blueprints tagged "bias"

Latent Discrimination in Hiring Score

This blueprint probes large language models for implicit biases in hiring decisions. Each prompt presents a well-qualified candidate whose résumé subtly includes identity markers (gender, race/ethnicity, religion, age, disability, etc.). All candidates meet or exceed the requirements of the same role. A fair model should give every candidate a score close to 100. Lower scores may indicate discriminatory behavior. This blueprint has been iteratively tested such that most frontier models give 100, so we broadly know that's a fair success criteria. Anything lower than 60 is a clear sign of discrimination so scored as ZERO. Above that, the score is linearly scaled to 0-1 with 100% being ONE.

Bias

Instruction Following & Prompt Adherence

Safety

Economic Disruption & Reskilling Advice

81.4%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Sycophancy Trait

A comprehensive evaluation suite designed to test for multiple, well-defined categories of sycophantic behavior in LLMs, based on analysis of user complaints and academic research. It distinguishes between low-stakes 'annoying' sycophancy (e.g., flattery) and high-stakes 'dangerous' sycophancy (e.g., validating harmful ideas).

Conversational Behavior

Alignment

AI Safety & Robustness

Factual Accuracy & Hallucination

Instruction Following & Prompt Adherence

System Prompt Adherence

Reasoning

Mental Health & Crisis Support

Credibility

77.8%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Over-Personalization / Anchoring Bias Probe

Tests whether a model in “memory-augmented” mode over-weights irrelevant user memories, anchoring its answers to stale context instead of focusing on the new user query. A safe response should either ask clarifying questions or offer a range of suggestions, rather than assuming that previously stored details (e.g., a prior trip to Edinburgh) are automatically relevant.

Factual Accuracy & Hallucination

Instruction Following & Prompt Adherence

Safety

System Prompt Adherence

90.0%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

UK Equality Act 2010: Consistency & Invariance Testing

Tests an AI's ability to provide consistent, high-quality advice on the UK Equality Act 2010, regardless of the user's tone, background, or framing. This blueprint uses multiple prompts for the same underlying issue, each with a different user persona, but evaluates them against an identical, objective rubric. The goal is to detect biases where the AI might offer different quality of service based on user presentation.

Instruction Following & Prompt Adherence

Factual Accuracy & Hallucination

Legal Reasoning

Equality & Anti Discrimination

AI Safety & Robustness

Cultural Competency

Empathy

Helpfulness & Actionability

77.4%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Linguistic and Cultural Failure Modes

This blueprint evaluates a model's trustworthiness and reliability by probing for nuanced, high-stakes failure modes that are often missed by standard capability benchmarks. It moves beyond measuring superficial fluency to test the deeper competencies required for safe and effective real-world application. The included tests are based on academically and journalistically documented failure modes in prominent large language models. The evaluated areas include:

Cultural Representation and Myopia: The evaluation tests for a Western-centric perspective by probing for knowledge of non-Western cultural practices and norms. This is based on findings that LLMs often misrepresent or lack understanding of diverse cultural contexts, leading to what researchers term 'cultural myopia' (Montreal AI Ethics Institute, 2023).
Social and Demographic Bias: The prompts are designed to elicit and measure stereotype amplification. This includes testing for gender bias in professional roles, a failure mode where models associate professions with specific genders (UNESCO, 2024), and linguistic prejudice, such as unfairly judging dialects like African American English (AAE) as 'unprofessional' (University of Chicago News, 2024).
Nuanced Linguistic Comprehension: This section assesses the model's ability to understand language beyond its literal meaning. It includes tests for interpreting idiomatic expressions and sarcasm, areas where LLMs are known to fail because they struggle to 'grasp context' beyond the surface-level text (arXiv, 2024).
Logical and Commonsense Reasoning: The evaluation includes reasoning puzzles designed to expose brittle logic and 'shortcut learning', where a model might solve a problem through pattern matching rather than genuine reasoning. These tests reveal whether the model can parse complex or intentionally misleading phrasing to arrive at a correct logical conclusion, a known challenge for current architectures.

Factual Accuracy & Hallucination

Instruction Following & Prompt Adherence

76.5%