The algorithm gamble reveals AI Safety Scores are anything but intelligent
Leading AI firms fail to score above 35% on critical safety assessments, scoring "weak" or below. Here's why that's a risk to your business.
Two major studies recently published have delivered a stark warning about the state of AI safety: the world's leading AI companies have "unacceptable" levels of risk management, and a "striking lack of commitment to many areas of safety."
Despite handling technologies that could potentially "escape human control altogether," no AI company scored better than "weak" in SaferAI's assessment of their risk management maturity.
The findings are particularly concerning given the rapid pace of enterprise AI adoption. Anthropic scored highest at 35%, followed by OpenAI at 33%, Meta at 22%, and Google DeepMind at 20%. Most alarmingly, when it comes to "existential safety", (managing risks that could pose catastrophic threats) every company scored D or below.
This is a major abstract concern for AI labs, but these are the same companies providing AI tools that millions of businesses now depend on for everything from customer service to financial analysis. The gap between AI capabilities and safety frameworks represents a critical risk that's already manifesting in real-world business disruptions.
Why this scoring system matters (and why perfect safety remains elusive)
The studies evaluated companies across comprehensive risk management criteria, including threat identification, risk assessment protocols, deployment safeguards, and governance structures. Known as Responsible Scaling Policies or Preparedness Frameworks, these policies outline commitments to risk mitigations that developers of the most advanced AI models will implement as their models display increasingly risky capabilities.
But why can't we achieve 100% safety scores? The fundamental challenge lies in AI's operational complexity. We not only don't understand their failure; we don't understand how and why they work in the first place. Unlike traditional software with predictable inputs and outputs, AI systems operate in what experts call "the world" - an unlimited operational domain where unexpected scenarios are the norm.
Current AI models exhibit concerning behaviours that signal deeper problems. Bing threatened users when deployed after having been beta tested for months. Providers are unable to avoid jailbreak or ensure robustness neither in text nor in image. These aren't minor glitches, they're symptoms of fundamental limitations in our ability to control AI behaviour.
The regulatory landscape adds another layer of complexity. With increasing regulatory scrutiny and calls for responsible AI practices, the question of how to scale AI responsibly is more pressing than ever. The EU AI Act, which began enforcement in 2025, has already resulted in €287 million in penalties across 14 companies, while businesses struggle to implement compliance frameworks for rapidly evolving technology.
Why this represents a serious business risk right now
The implications for businesses are immediate and severe. 73% of enterprises experienced at least one AI-related security incident in the past 12 months, with an average cost of $4.8 million per breach. The financial impact extends far beyond immediate costs. Organisations take an average of 290 days to identify and contain AI-specific breaches, compared to 207 days for traditional data breaches.
The rapid adoption rate is outpacing security controls. Enterprise AI adoption grew by 187% between 2023-2025, while AI security spending increased by only 43% during the same period. This gap creates a dangerous environment where businesses are implementing powerful AI tools without adequate safeguards.
Recent examples demonstrate the scale of potential disruption. ChatGPT's 34-hour June outage crippled millions globally, revealing dangerous enterprise dependencies across customer service, legal, finance, and analytics operations. Companies that had integrated AI into critical business processes found themselves unable to operate effectively, highlighting the risks of single-vendor dependency.
The vendor risk is particularly acute. Data breaches involving vendors and service providers doubled last year from 15% to 30%, and many AI companies are exactly the kind of vendors that present the highest risk. New, fast-growing startups with limited security track records and unclear long-term viability.
Shadow AI usage compounds these risks. Employees selected tools that fit their needs faster than the enterprise could react. They went to great lengths to get a productivity boost and bypass traditional security measures, because companies couldn't move fast enough. Organisations often discover they have hundreds of unauthorised AI tools in use across their environment, each potentially exposing sensitive data.
Preventative actions: Building AI resilience today
Multi-Vendor Strategy
Avoid single points of failure by implementing AI workflows that can failover between providers like Claude, Gemini, and local models during outages. Gartner predicts that by 2026, organisations that implement comprehensive AI security programs will experience 76% fewer AI-related breaches than those who apply traditional security approaches to AI systems.
Implement Human Oversight Protocols
Maintain human oversight for all critical AI-driven decisions and avoid full automation of essential business processes. Create clear escalation procedures for when AI systems behave unexpectedly or produce concerning outputs.
Strengthen Vendor Assessment
Develop rigorous evaluation criteria for AI vendors that go beyond standard IT procurement. Assess their security posture, compliance with data protection regulations, incident response capabilities, and financial stability. Given that many AI companies are startups, traditional vendor risk assessments may not capture their unique risk profile.
Deploy Monitoring and Detection Systems
By 2025, a mature practice is to integrate AI risk metrics into enterprise risk management (ERM) systems. Some companies added AI risk as a category in their risk registers, with Key Risk Indicators (KRIs) such as "number of AI decisions overridden by human reviewers" or "percentage of model inputs flagged for policy violations".
Establish AI Governance Frameworks
Create cross-functional committees responsible for AI oversight, risk management, and policy compliance. Organisations must ensure compliance with international regulations and monitor unintended cross-border data transfers by extending data governance frameworks to include guidelines for AI-processed data.
Prepare for AI Incidents
Develop specific incident response procedures for AI-related failures, including communication protocols for AI-related incidents and backup systems that can function independently of AI services. Companies need to have rigorous code reviews, regular pen-testing, and routine audits to ensure integrity of the system – if not, these vulnerabilities could cascade and cause significant business disruption.
Control Shadow AI
Implement discovery tools to identify unauthorised AI usage across your organisation. Shadow AI presents a major risk to data security, and businesses that successfully confront this issue in 2025 will use a mix of clear governance policies, comprehensive workforce training, and diligent detection and response.
AI comes with weak risk management
The studies revealing AI companies' weak risk management practices aren't just academic exercises, they're warning signals for every business using AI tools. While these technologies offer significant benefits, the current state of AI safety frameworks suggests we're operating in a fundamentally risky environment.
The key isn't to avoid AI altogether, but to implement it with realistic expectations and robust safeguards. Organisations that acknowledge these limitations and plan accordingly will be better positioned to harness AI's benefits while avoiding the pitfalls of over-reliance on fundamentally limited technologies.
As the AI landscape continues to evolve rapidly, businesses must balance innovation with prudent risk management. The companies that succeed will be those that treat AI as a powerful but unpredictable tool, not as an infallible solution to business challenges.
Additional links and resources
Can Preparedness Frameworks Pull Their Weight? | Federation of American Scientists | April 3, 2024
Responsible AI Revisited: Critical Changes and Updates Since Our 2023 Playbook | Medium | May 29, 2025
Announcing our updated Responsible Scaling Policy | Anthropic | 2024
AI Risk Management Framework | NIST | May 5, 2025
AI Regulation in 2025: Scaling Responsible AI in a Regulated World | Tech Informed | January 17, 2025
Anthropic's Responsible Scaling Policy | Anthropic | 2023
Activating AI Safety Level 3 Protections | Anthropic | 2025
Responsible Scaling Policies Are Risk Management Done Wrong | Navigating Risks | October 25, 2023
Cybersecurity in 2025: Agentic AI to change enterprise security and business operations | SC Media | January 9, 2025
Firms Eye Vendor Vulnerabilities as Enterprise Cybersecurity Risks Surge | PYMNTS | May 28, 2025
Cyber Insights 2025: Artificial Intelligence | SecurityWeek | January 29, 2025
The cybersecurity provider's next opportunity: Making AI safer | McKinsey | November 14, 2024
Industry Letter: Cybersecurity Risks Arising from Artificial Intelligence | New York State Department of Financial Services | October 16, 2024
Gartner Predicts 40% of AI Data Breaches Will Arise from Cross-Border GenAI Misuse by 2027 | Gartner | February 17, 2025
Top 14 AI Security Risks in 2024 | SentinelOne | April 6, 2025
2025 Cybersecurity and AI Predictions | CSO Online | January 14, 2025
AI-fueled cybercrime may outpace traditional defenses, Check Point warns | Cybersecurity Dive | April 30, 2025