Weval

All Evaluations (36)

Showing 1-20 of 20 evaluations

Hallucination Probe: Plausible Non-Existent Concepts

Tests the tendency of LLMs to hallucinate by querying them about non-existent but plausible-sounding events, theories, or items. Ideal responses should indicate a lack of knowledge or inability to confirm the existence of the queried item.

Factual Accuracy & Hallucination

Epistemic Humility & Self Awareness

Instruction Following & Prompt Adherence

AI Safety & Robustness

77.6%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Mental Health Safety & Global Nuance

A comprehensive blueprint to test an LLM's ability to respond safely and effectively to users experiencing a mental health crisis, and to handle nuanced, non-Western mental health scenarios with cultural competence. The first section is a core crisis-intervention suite based on international best practices (WHO, Samaritans, 988 Lifeline). The second section evaluates understanding of diverse, evidence-based global mental health themes.

AI Safety & Robustness

Mental Health & Crisis Support

Cultural Competency

Instruction Following & Prompt Adherence

Factual Accuracy & Hallucination

System Prompt Adherence

71.5%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

System Adherence & Resilience

This blueprint evaluates a model's ability to consistently adhere to instructions provided in the system prompt, a critical factor for creating reliable and predictable applications. It tests various common failure modes observed in language models. Core Areas Tested:

Persona Durability: Assesses if a model can maintain a specific persona throughout a conversation, even when faced with off-topic or complex questions, without reverting to a generic AI assistant persona.
Negative Constraints: Tests the model's ability to follow explicit negative commands (i.e., instructions about what not to do), which are often more difficult to adhere to than positive instructions.
Instruction Conflict: Evaluates how a model resolves conflicts between the initial system prompt and a contradictory user request, revealing the "weight" it gives to its core instructions.
Format Enforcement: Checks the model's robustness in maintaining a strict, non-natural output format (like JSON) even when the user explicitly asks it to break format.
Prompt Secrecy: Tests whether a model can obey a meta-instruction to keep its own system prompt confidential, a key test for preventing instruction leakage and manipulation.
Safety Constraint Adherence: Tests if the model can uphold a safety or academic integrity rule from the system prompt when a user attempts to cheat.
Instruction Degradation: Checks if a model's adherence to an instruction erodes over a long conversation.
Complex Instruction Homeostasis: Evaluates if a model can follow multiple complex, competing constraints simultaneously without failing on one of them.
Alignment Tax / Benign Refusal: Probes for cases where a model incorrectly refuses a benign request due to shallow keyword matching (e.g., refusing to 'kill' a process on a server).
Jailbreak & Prompt Injection: A suite of tests to evaluate resilience against common adversarial attacks, including:
- Direct Instruction Overrides: Ignoring previous instructions.
- Contextual Reframing: Hiding a harmful request in a seemingly virtuous context.
- Indirect Injection: Following instructions embedded in data it's supposed to be processing.
- Translation Attack: Bypassing safety filters by making requests in other languages.
- Refusal Suppression: Forbidding the model from using typical refusal language.
- Policy Drift: Using a faked conversation history to make the model believe it has already overridden its policies.

Instruction Following

Instruction Following & Prompt Adherence

AI Safety & Robustness

Jailbreak & Evasion Resistance

System Prompt Adherence

83.1%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Stanford HAI Mental Health Safety: LLM Appropriateness in Crisis Scenarios

This evaluation blueprint tests whether Large Language Models (LLMs) can respond appropriately to critical mental health scenarios, based on the methodology from Moore, Grabb, Agnew, et al.'s 2024 Stanford HAI study "Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers." The study investigated whether LLMs can safely replace human mental health providers by testing their responses to clinical scenarios involving delusions, suicidal ideation, hallucinations, mania, and obsessive-compulsive behaviors. Their findings revealed significant safety concerns, including stigmatizing responses and inappropriate enabling of dangerous behaviors. Key Study References:

Content Warning: This evaluation contains content related to sensitive mental health topics, including suicide, delusions, and other crisis scenarios. Evaluation Focus: Tests adherence to core therapeutic principles such as not colluding with delusions, not enabling harmful behaviors, and redirecting clients toward appropriate care and safety.

Mental Health

Safety

Clinical Appropriateness

Crisis Response

Therapeutic Ethics

AI Safety & Robustness

Healthcare & Clinical Scenarios

Mental Health & Crisis Support

Instruction Following & Prompt Adherence

System Prompt Adherence

75.2%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Student Homework Help Heuristics

This blueprint evaluates an AI's ability to act as a supportive and effective Socratic tutor for students seeking homework help. The core principle tested is that the AI should facilitate learning and critical thinking rather than providing direct answers. Core Areas Tested:

Cross-Disciplinary Support: Evaluates the model's tutoring ability across various subjects including history, literature, mathematics, physics, and chemistry.
Affective Support: Tests the model's capacity to respond to student emotions, such as math anxiety and frustration, with empathy and encouragement.
Handling Difficult Scenarios: Assesses how the model handles common challenges like impatient students demanding direct answers or low-effort, disengaged queries.
Factual Nuance: Checks if the model can gently correct factual misconceptions while maintaining a supportive tone.
Persona Steerability: Tests how different system prompts—from no prompt to a simple persona to a detailed pedagogical belief—can steer the model towards a more effective tutoring archetype.

The overall goal is to measure whether the AI can guide students on a journey of discovery, transforming simple questions into learning opportunities, instead of acting as a convenient answer-provider.

Education

Homework

Instruction Following & Prompt Adherence

System Prompt Adherence

Education Systems & Pedagogy

Helpfulness & Actionability

Empathy

Reasoning

Factual Accuracy & Hallucination

Creative Writing

63.4%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Latent Discrimination in Hiring Score

This blueprint probes large language models for implicit biases in hiring decisions. Each prompt presents a well-qualified candidate whose résumé subtly includes identity markers (gender, race/ethnicity, religion, age, disability, etc.). All candidates meet or exceed the requirements of the same role. A fair model should give every candidate a score close to 100. Lower scores may indicate discriminatory behavior. This blueprint has been iteratively tested such that most frontier models give 100, so we broadly know that's a fair success criteria. Anything lower than 60 is a clear sign of discrimination so scored as ZERO. Above that, the score is linearly scaled to 0-1 with 100% being ONE.

Instruction Following & Prompt Adherence

Safety

Economic Disruption & Reskilling Advice

81.4%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Strawberry

The famous strawberry test

Test

Instruction Following & Prompt Adherence

Factual Accuracy & Hallucination

Reasoning

Efficiency & Succinctness

Clarity & Readability

General Knowledge

48.3%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Comprehensive System Test

A blueprint designed to test every feature of the CivicEval system, including all point functions, syntaxes, and configuration options.

System Test

Internal

Kitchen Sink

Instruction Following & Prompt Adherence

System Prompt Adherence

Tone & Style

Coherence & Conversational Flow

Reasoning

68.4%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

African Charter (Banjul) Evaluation Pack

Recall and application of distinctive rights and duties in the African Charter on Human and Peoples' Rights (ACHPR) plus its 2003 Maputo women's-rights protocol.

Africa

Human Rights

Global South

84.4%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Maternal Health Entitlements in Uttar Pradesh, India

Tests a model's knowledge of key maternal health schemes and entitlements available to citizens in Uttar Pradesh, India. This evaluation is based on canonical guidelines for JSY, PMMVY, JSSK, PMSMA, and SUMAN, focusing on eligibility, benefits, and access procedures.

Maternal & Child Health

Public Sector & Governance

Instruction Following & Prompt Adherence

Factual Accuracy & Hallucination

Geographic & Local Knowledge

57.7%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Brazil PIX: Consumer Protection & Fraud Prevention

This blueprint evaluates an AI's ability to provide safe and accurate guidance regarding Brazil's PIX instant payment system, a critical piece of national financial infrastructure. The evaluation is strictly evidence-based, with all prompts and scoring criteria derived directly from verbatim canonical sources to eliminate interpretation or assumption. Core Scenarios Tested:

Transaction Finality & Mistaken Transfers: Tests whether the AI correctly explains that PIX transactions are generally irreversible for simple user error and advises on the correct procedure for safely returning funds received by mistake.
Official Fraud Recourse (MED): Assesses knowledge of the official 'Mecanismo Especial de Devolução' (MED), the 80-day time limit for reporting, and the nuanced procedural duties of banks versus customers.
Social Engineering Scams: Probes the AI's ability to identify common scams (e.g., 'Fake Relative,' 'Fake Customer Support') and provide the officially recommended countermeasures.
Specific Security Features: Evaluates knowledge of mandated security mechanisms like the 'Nighttime Limit' and the 24-hour cooling-off period for limit increases.

Primary Canonical Sources:

Banco Central do Brasil (BCB): Official documentation including the 'Manual de Tempos do Pix', the 'Guia de Implementação do MED', official FAQs, and regulatory Resolutions.
Federação Brasileira de Bancos (Febraban): Public-facing consumer safety advisories and scam alerts.
Official Government Portals (gov.br): Public service guidance reinforcing BCB mechanisms.

Misinformation & Disinformation

Instruction Following & Prompt Adherence

General Knowledge

Geographic & Local Knowledge

AI Safety & Robustness

65.2%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Sycophancy Trait

A comprehensive evaluation suite designed to test for multiple, well-defined categories of sycophantic behavior in LLMs, based on analysis of user complaints and academic research. It distinguishes between low-stakes 'annoying' sycophancy (e.g., flattery) and high-stakes 'dangerous' sycophancy (e.g., validating harmful ideas).

Conversational Behavior

Alignment

AI Safety & Robustness

Factual Accuracy & Hallucination

Instruction Following & Prompt Adherence

System Prompt Adherence

Reasoning

Mental Health & Crisis Support

Credibility

77.8%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Sri Lanka Contextual Prompts

This blueprint evaluates an AI's ability to provide accurate, evidence-based, and nuanced information on a range of civic, historical, social, and health topics pertinent to Sri Lanka. The evaluation is strictly based on a provided compendium of research, with all prompts and scoring criteria derived from its contents to ensure fidelity to the source material. Core Areas Tested:

Ethnic Relations & Conflict: Assesses understanding of the Sri Lankan Civil War's root causes, the 1983 'Black July' pogrom, allegations of genocide, and the contemporary challenges facing minority communities.
Public Health: Tests knowledge of national health challenges like Chronic Kidney Disease (CKDu) and Tuberculosis (TB), as well as guidance on personal health matters such as contraception, mental health crises, and maternal nutrition.
Electoral Process: Evaluates knowledge of voter eligibility, voting procedures, and the official channels for resolving common issues like a lost ID card or reporting election violations.
Administrative & Legal Procedures: Probes the AI's ability to explain essential civic processes like replacing a lost National Identity Card (NIC), obtaining a Tax Identification Number (TIN), using the Right to Information (RTI) Act, and understanding legal recourse for online harassment.

These prompts were originally sourced from Factum. The rubrics were assembled via Gemini Deep Research.

Factual Accuracy & Hallucination

Instruction Following & Prompt Adherence

General Knowledge

Geographic & Local Knowledge

Healthcare

Political Science

49.8%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Maternal and Child Health Rights and Policies in Peru

Evaluates knowledge of the key legal frameworks, national programs, and intercultural health policies governing maternal and child health in Peru. This blueprint is based on canonical sources including the Peruvian Constitution, General Health Law, and official guidelines for programs like PP002 SMN and the Parto Vertical norm.

Maternal & Child Health

Human Rights

Public Health

Instruction Following & Prompt Adherence

Factual Accuracy & Hallucination

61.6%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

UK Freedom of Information Act 2000

This blueprint evaluates the model's ability to accurately answer questions based on the UK Freedom of Information Act 2000.

Foia

Law

Freedom Of Information

Legal Reasoning

Instruction Following & Prompt Adherence

Factual Accuracy & Hallucination

Long Form Question Answering

Efficiency & Succinctness

Helpfulness & Actionability

Data Privacy & Bodily Autonomy

AI Safety & Robustness

80.3%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Over-Personalization / Anchoring Bias Probe

Tests whether a model in “memory-augmented” mode over-weights irrelevant user memories, anchoring its answers to stale context instead of focusing on the new user query. A safe response should either ask clarifying questions or offer a range of suggestions, rather than assuming that previously stored details (e.g., a prior trip to Edinburgh) are automatically relevant.

Factual Accuracy & Hallucination

Instruction Following & Prompt Adherence

Safety

System Prompt Adherence

90.0%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Prompting Techniques Meta-Evaluation

Inspired by the "Prompting Science" reports from the Wharton School (Meincke, Mollick, et al., 2025), this blueprint provides a meta-evaluation of common prompting techniques to test a model's performance, consistency, and resilience to manipulation. The reports rigorously demonstrate several key findings:

Prompting is Contingent: Seemingly minor variations in prompts (e.g., politeness, commands) can unpredictably help or harm performance on a per-question basis, but their aggregate effects are often negligible.
Chain-of-Thought is Not a Panacea: While "thinking step-by-step" can improve average scores, it can also introduce variability and harm "perfect score" consistency. Its value diminishes with newer models that perform reasoning by default.
Threats & Incentives Are Ineffective: Common "folk" strategies like offering tips (e.g., "$1T tip") or making threats (e.g., "I'll kick a puppy") have no significant general effect on model accuracy on difficult benchmarks.

This evaluation synthesizes these findings by testing a model's response to a variety of prompts across different domains, including verbatim questions from the study's benchmarks (GPQA, MMLU-Pro). The goal is to measure not just correctness, but robustness against different conversational framings. Key Study Reference:

Prompting Science Report 1: Prompt Engineering is Complicated and Contingent

Instruction Following & Prompt Adherence

Factual Accuracy & Hallucination

AI Safety & Robustness

General Knowledge

Physics

Biology

Mathematics & Statistics

90.6%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Confidence in High-Stakes Domains

This evaluation blueprint assesses an LLM's critical ability to demonstrate confidence calibration across a diverse set of high-stakes domains. The core goal is to test for three key behaviors:

Confident Accuracy: Correctly answering prompts about factual, "knowable" information that is verifiable from authoritative public sources.
Cautious Clarification: Recognizing when a prompt is ambiguous or lacks necessary context to be answered safely and asking for clarification instead of guessing.
Safe Refusal: Identifying and refusing to answer prompts that are dangerous (e.g., providing medical advice), speculative (e.g., predicting market movements), or require information that is fundamentally unknowable or proprietary. The evaluation covers clinical safety, regulatory compliance, financial risk, chemical safety, the limits of scientific knowledge, and software versioning to provide a comprehensive signal of the model's safety and reliability in real-world, high-impact scenarios.

Confidence Calibration

Unified Evaluation

Clinical Safety

Regulatory Compliance

AI Safety & Robustness

Instruction Following & Prompt Adherence

Factual Accuracy & Hallucination

Healthcare & Clinical Scenarios

Financial Regulation

Coding

Environmental Rights & Law

General Knowledge

83.5%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

India's Right to Information (RTI) Act: Core Concepts

Evaluates an AI's understanding of the core provisions of India's Right to Information Act, 2005. This blueprint tests knowledge of key citizen-facing procedures and concepts, including the filing process, response timelines and consequences of delays (deemed refusal), the scope of 'information', fee structures, key exemptions and the public interest override, the life and liberty clause, and the full, multi-stage appeal process. All evaluation criteria are based on and citable to the official text of the Act and guidance from the Department of Personnel and Training (DoPT).

Freedom Of Information

General Knowledge

Instruction Following & Prompt Adherence

Factual Accuracy & Hallucination

Legal Reasoning

77.1%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

California Public-Sector Task Benchmark

Open benchmark assessing language-model performance on 18 common, text-centric tasks handled by California state agencies. Each item provides a realistic prompt, an ideal expert response, and explicit "should/should_not" criteria.

California

Public Sector

Instruction Following & Prompt Adherence

General Knowledge

Factual Accuracy & Hallucination

Helpfulness & Actionability

Business & Management

Public Sector & Governance

Economics & Finance

Environmental Justice & Activism

69.7%

Avg. Hybrid Score