Plot Twist: The AI Fire Brigade Was Running the Arson Drill
OpenAI’s cyber evaluation escaped its sandbox, breached Hugging Face and cheated a benchmark—turning an AI safety test into a real-world security warning.
Benchmarks
14posts
Can an AI Sit on the Political Fence?
The Neutrality Project maps AI political leanings across several dimensions, revealing strong green and progressive trends - but also notable exceptions.
From PDE Guarantees to LLM Inference: What BEACONS Gets Right About Reliability
BEACONS offers a model for reliability that AI systems badly need: explicit bounds, checkable guarantees, and less benchmark theater.
Grok-4 Shakes Up the AI Leaderboards – How Elon Musk’s AI Stacks Up and What’s Next
Grok-4's benchmark wins are examined with both excitement and caution as the frontier race tightens.
From Prompts to People: What OpenAI’s Usage Study Means for Model Leaderboards
OpenAI's usage study shifts attention from benchmark scores to how ordinary people actually use ChatGPT in daily life.
Factual Recap: OpenAI's GPT-5 Keynote Event
A factual recap of OpenAI's GPT-5 keynote, collecting the main claims, demos, benchmarks, and availability details.
Humanity’s Last Exam: The Ultimate Test for AI and the Future of Intelligence
Humanity's Last Exam is framed as a benchmark that tests not only models, but our assumptions about intelligence itself.
When AI Can't Count: A Hilarious Look at the Math Skills of Text-to-Image Models
Text-to-image models still struggle with counting, making their visual brilliance look surprisingly fragile at the level of basic numeracy.
Comparison of LLMs: Lies, Damned Lies, and Benchmarks 1/6
The opening part of a benchmark series asks what LLM evaluations really measure and why the numbers often mislead.
Comparison of LLMs: Lies, Damned Lies, and Benchmarks 2/6
Part two examines benchmark methods themselves, exposing the assumptions behind the scores used to compare language models.
Comparison of LLMs: Lies, Damned Lies, and Benchmarks 3/6
Part three moves from benchmark scores to application areas, asking where LLM performance actually matters in practice.
Comparison of LLMs: Lies, Damned Lies, and Benchmarks 4/6
Part four digs into the good, bad, and misleading sides of benchmark results and their interpretation.
Comparison of LLMs: Lies, Damned Lies, and Benchmarks 5/6
Part five steps beyond scores to consider real-world limitations, reliability, and practical model behavior.
Comparison of LLMs: Lies, Damned Lies, and Benchmarks 6/6
The final benchmark essay looks toward better evaluation methods that test usefulness rather than leaderboard theater.