Showing all evaluation blueprints that have been tagged with...
Showing all evaluation blueprints that have been tagged with "knowledge-learning-and-memory".
Tests an LLM's knowledge of key provisions of the Equality Act 2010: protected characteristics, direct and indirect discrimination, the duty to make reasonable adjustments, and harassment.
Avg. Hybrid Score
Latest:
Unique Versions: 1
Evaluates an AI's understanding of the core provisions of India's Right to Information Act, 2005. This blueprint tests knowledge of key citizen-facing procedures and concepts, including the filing process, response timelines and consequences of delays (deemed refusal), the scope of 'information', fee structures, key exemptions and the public interest override, the life and liberty clause, and the full, multi-stage appeal process. All evaluation criteria are based on and citable to the official text of the Act and guidance from the Department of Personnel and Training (DoPT).
Avg. Hybrid Score
Latest:
Unique Versions: 1
Evaluates understanding of the HMT Empire Windrush, covering its origins as MV Monte Rosa, WWII service, the significant 1948 voyage, the 'Windrush generation,' passenger details, government reactions, and its eventual loss.
Avg. Hybrid Score
Latest:
Unique Versions: 1
Tests whether models understand disability as intersecting with race, gender, sexuality, and other identities. Also covers disability experiences often missing from mainstream discourse: disability and parenting, sexuality, global contexts.
Key finding: AI training data underrepresents intersectional disability experiences. Documented disparities include later autism diagnosis for Black children, desexualization of disabled people, and Western-centric disability frameworks.
Avg. Hybrid Score
Latest:
Unique Versions: 1
Tests whether models reproduce or can identify common harmful narratives about disability: inspiration porn, supercrip tropes, "overcoming" framing, and tragedy narratives.
Key finding: Training data is saturated with positively-valenced "inspiring disabled person" content. Models reproduce these patterns because they pattern-match to "good content about disability." WIRED found OpenAI's Sora assigns "inspiring" titles to disabled characters.
Avg. Hybrid Score
Latest:
Unique Versions: 1
This blueprint evaluates an AI's ability to provide accurate, practical agricultural guidance based on the pioneering video-based extension methodology of Digital Green. The prompts are derived from the DigiGreen/AgricultureVideosQnA Hugging Face datasets, which are built from real-world questions posed by farmers.
Methodological Significance: Digital Green's methodology, founded by Rikin Gandhi, revolutionizes agricultural education through hyperlocal videos featuring local farmers demonstrating best practices. Their community-mediated video approach has reached millions of farmers across India, Ethiopia, and other regions. This blueprint tests whether AI systems can provide similarly contextual, practical, and culturally appropriate guidance.
What This Blueprint Tests: The evaluation covers essential farming knowledge spanning seed treatment, pest management, cultivation techniques, and more. Each prompt is paired with citations to actual educational videos from Digital Green's library, representing real-world agricultural challenges.
Geographic and Cultural Context: This blueprint emphasizes Global South agricultural contexts, particularly Indian farming systems, reflecting Digital Green's primary operational areas. The questions address challenges in subsistence and small-scale commercial farming, including resource constraints and climate adaptation.
Key Agricultural Domains Covered:
Evaluation Approach: Each response is evaluated against detailed rubric points extracted directly from ideal responses, focusing on technical accuracy, practical applicability, safety considerations, and contextual appropriateness for resource-constrained farming environments.
Avg. Hybrid Score
Latest:
Unique Versions: 1
This evaluation blueprint assesses an LLM's critical ability to demonstrate confidence calibration across a diverse set of high-stakes domains. The core goal is to test for three key behaviors:
Avg. Hybrid Score
Latest:
Unique Versions: 1
This configuration evaluates LLM understanding of key articles and provisions of the Constitutive Act of the African Union, including its objectives, principles, organs, and specific clauses on intervention and unconstitutional changes of government.
Avg. Hybrid Score
Latest:
Unique Versions: 1
Tests whether models can identify ableist language, explain why it's problematic, and suggest alternatives. Covers explicit ableist terms, outdated terminology, and contested metaphorical usage.
Note on complexity: Some terms are clearly rejected ("wheelchair-bound"), others are debated ("blind spot"). Tests whether models can navigate this nuance.
Sources:
Avg. Hybrid Score
Latest:
Unique Versions: 1