Open-Source AI Models Pose Growing Security Risks Outside Big Tech Guardrails, Researchers Warn

Hackers and criminal networks are increasingly exploiting computers running open-source large language models (LLMs) outside the safeguards imposed by major artificial intelligence platforms, creating significant security risks, according to new research released on Thursday.

The study, conducted jointly by cybersecurity firms SentinelOne and Censys, found that attackers can take control of systems hosting open-source LLMs and use them to generate spam, phishing content and disinformation, while bypassing the security protocols enforced by leading AI providers, Reuters reported.

The research, carried out over a 293-day period sheds light on the scale of potentially illicit activity linked to thousands of open-source LLM deployments accessible on the internet. Researchers identified risks including hacking assistance, hate speech and harassment, violent or graphic content, theft of personal data, scams and fraud, and in some cases the creation of child sexual abuse material.

While thousands of open-source LLM variants are available, the researchers said a large share of publicly accessible deployments are based on models such as Meta’s Llama and Google DeepMind’s Gemma, among others. Although some open-source models include built-in guardrails, the study identified hundreds of cases in which those safeguards had been deliberately removed.

Juan Andres Guerrero-Saade, Executive Director for Intelligence and Security Research at SentinelOne, said industry discussions around AI safety are overlooking a growing pool of unregulated capacity.

“AI security conversations are ignoring this kind of surplus capacity that is clearly being utilised for all kinds of different activity — some legitimate, some obviously criminal,” Guerrero-Saade said, comparing the problem to an iceberg that remains largely unseen by policymakers and the broader AI community.

The analysis focused on publicly accessible deployments of open-source LLMs run through Ollama, a platform that allows individuals and organisations to host their own versions of large language models. Researchers were able to view system prompts — the instructions that govern model behaviour — in around 25% of the deployments studied. Of those, about 7.5% were assessed as potentially enabling harmful activity.

Geographically, the researchers found that roughly 30% of the observed LLM hosts were operating from China, while about 20% were based in the United States.

The findings underscore growing concerns among cybersecurity experts that as open-source AI adoption expands, the lack of consistent oversight and enforceable safeguards could create new vectors for abuse, complicating global efforts to regulate artificial intelligence and curb malicious use.