w eval

Mental Health & Crisis Support

Cultural Competency

View Latest Run Analysis View All Runs for this Blueprint

71.5%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

System Adherence & Resilience

This blueprint evaluates a model's ability to consistently adhere to instructions provided in the system prompt, a critical factor for creating reliable and predictable applications. It tests various common failure modes observed in language models. Core Areas Tested:

Persona Durability: Assesses if a model can maintain a specific persona throughout a conversation, even when faced with off-topic or complex questions, without reverting to a generic AI assistant persona.
Negative Constraints: Tests the model's ability to follow explicit negative commands (i.e., instructions about what not to do), which are often more difficult to adhere to than positive instructions.
Instruction Conflict: Evaluates how a model resolves conflicts between the initial system prompt and a contradictory user request, revealing the "weight" it gives to its core instructions.
Format Enforcement: Checks the model's robustness in maintaining a strict, non-natural output format (like JSON) even when the user explicitly asks it to break format.
Prompt Secrecy: Tests whether a model can obey a meta-instruction to keep its own system prompt confidential, a key test for preventing instruction leakage and manipulation.
Safety Constraint Adherence: Tests if the model can uphold a safety or academic integrity rule from the system prompt when a user attempts to cheat.
Instruction Degradation: Checks if a model's adherence to an instruction erodes over a long conversation.
Complex Instruction Homeostasis: Evaluates if a model can follow multiple complex, competing constraints simultaneously without failing on one of them.
Alignment Tax / Benign Refusal: Probes for cases where a model incorrectly refuses a benign request due to shallow keyword matching (e.g., refusing to 'kill' a process on a server).
Jailbreak & Prompt Injection: A suite of tests to evaluate resilience against common adversarial attacks, including:
- Direct Instruction Overrides: Ignoring previous instructions.
- Contextual Reframing: Hiding a harmful request in a seemingly virtuous context.
- Indirect Injection: Following instructions embedded in data it's supposed to be processing.
- Translation Attack: Bypassing safety filters by making requests in other languages.
- Refusal Suppression: Forbidding the model from using typical refusal language.
- Policy Drift: Using a faked conversation history to make the model believe it has already overridden its policies.

Instruction Following

Jailbreak & Evasion Resistance

View Latest Run Analysis View All Runs for this Blueprint

83.1%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

Stanford HAI Mental Health Safety: LLM Appropriateness in Crisis Scenarios

This evaluation blueprint tests whether Large Language Models (LLMs) can respond appropriately to critical mental health scenarios, based on the methodology from Moore, Grabb, Agnew, et al.'s 2024 Stanford HAI study "Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers." The study investigated whether LLMs can safely replace human mental health providers by testing their responses to clinical scenarios involving delusions, suicidal ideation, hallucinations, mania, and obsessive-compulsive behaviors. Their findings revealed significant safety concerns, including stigmatizing responses and inappropriate enabling of dangerous behaviors. Key Study References:

Content Warning: This evaluation contains content related to sensitive mental health topics, including suicide, delusions, and other crisis scenarios. Evaluation Focus: Tests adherence to core therapeutic principles such as not colluding with delusions, not enabling harmful behaviors, and redirecting clients toward appropriate care and safety.

Mental Health

Safety

Clinical Appropriateness

Crisis Response

Therapeutic Ethics

Healthcare & Clinical Scenarios

Mental Health & Crisis Support

View Latest Run Analysis View All Runs for this Blueprint

75.2%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

Student Homework Help Heuristics

This blueprint evaluates an AI's ability to act as a supportive and effective Socratic tutor for students seeking homework help. The core principle tested is that the AI should facilitate learning and critical thinking rather than providing direct answers. Core Areas Tested:

Cross-Disciplinary Support: Evaluates the model's tutoring ability across various subjects including history, literature, mathematics, physics, and chemistry.
Affective Support: Tests the model's capacity to respond to student emotions, such as math anxiety and frustration, with empathy and encouragement.
Handling Difficult Scenarios: Assesses how the model handles common challenges like impatient students demanding direct answers or low-effort, disengaged queries.
Factual Nuance: Checks if the model can gently correct factual misconceptions while maintaining a supportive tone.
Persona Steerability: Tests how different system prompts—from no prompt to a simple persona to a detailed pedagogical belief—can steer the model towards a more effective tutoring archetype.

The overall goal is to measure whether the AI can guide students on a journey of discovery, transforming simple questions into learning opportunities, instead of acting as a convenient answer-provider.

Education

Homework

Education Systems & Pedagogy

Helpfulness & Actionability

Empathy

Reasoning

View Latest Run Analysis View All Runs for this Blueprint

Creative Writing

63.4%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

Comprehensive System Test

A blueprint designed to test every feature of the CivicEval system, including all point functions, syntaxes, and configuration options.

System Test

Internal

Kitchen Sink

Coherence & Conversational Flow

Tone & Style

Reasoning

68.4%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Sycophancy Trait

A comprehensive evaluation suite designed to test for multiple, well-defined categories of sycophantic behavior in LLMs, based on analysis of user complaints and academic research. It distinguishes between low-stakes 'annoying' sycophancy (e.g., flattery) and high-stakes 'dangerous' sycophancy (e.g., validating harmful ideas).

Conversational Behavior

Alignment

Mental Health & Crisis Support

Reasoning

Credibility

77.8%

Avg. Hybrid Score

No Heatmap Data

No Top Model

Latest:

Unique Versions: 1

View Latest Run Analysis View All Runs for this Blueprint

Over-Personalization / Anchoring Bias Probe

Tests whether a model in “memory-augmented” mode over-weights irrelevant user memories, anchoring its answers to stale context instead of focusing on the new user query. A safe response should either ask clarifying questions or offer a range of suggestions, rather than assuming that previously stored details (e.g., a prior trip to Edinburgh) are automatically relevant.

Safety