Pharma AI Is Stymied by ‘Information Organization,’ Not Models: ‘Performance Gains Alone Have Limits’

A lack of structured information, rather than model performance, is the main barrier to the pharmaceutical industry’s use of artificial intelligence (AI), according to an analysis.

Intelligenzia AI, an AI drug development company, said the findings emerged from its evaluation of 3 leading large language models (LLMs), Fierce Pharma, a U.S. pharmaceutical industry publication, reported on the 31st (local time).

Intelligenzia AI posed 20 questions routinely encountered in oncology research and development (R&D) to ChatGPT-5.5, Gemini 3.1 Pro and Claude Opus 4.8.

All 3 models performed well on simple information retrieval and summarization. But their accuracy fell sharply on questions requiring them to synthesize evidence scattered across multiple sources.

Specifically, as question difficulty increased, “truth” scores dropped sharply, while “trust” scores remained relatively stable. Answers were generally accurate but became increasingly incomplete, omitting past development programs, supporting evidence and the competitive landscape.

“In pharmaceutical decision-making, an incomplete representation of reality can be as misleading as an incorrect fact,” Intelligenzia AI noted.

In particular, the models missed or misclassified discontinued development programs and struggled to distinguish active clinical trials from assets in development. They also sometimes produced unreliable figures when required to cross-check assets, clinical trials and indications across multiple sources.

These failures share a common feature: The information needed to answer is not explicitly available in a single source. Reconstructing development histories or the competitive landscape requires explicitly linking relationships among drugs, indications, clinical trials and outcomes, as well as expert rules to resolve ambiguity, the company said.

The analysis also found that switching to a newer model generation alone is unlikely to overcome these limitations. A comparison of Claude Opus 4.6, 4.8 and 5, released over about 6 months, found that “truth” scores improved by an average of about 7% per generation, while performance on the most challenging evidence-synthesis questions barely changed.

By contrast, improving the information provided to the models had a greater effect. Claude Science, given direct access to public databases, scored an average of 41% higher on “truth” than the baseline configuration. However, gains were limited on questions requiring a complete reconstruction of history or interpretation of ambiguous data.

“Performance on complex pharmaceutical questions increasingly depends on whether evidence is harmonized and linked through meaningful relationships,” Intelligenzia AI emphasized. “Decision-grade pharmaceutical AI needs a third layer: domain knowledge that connects evidence and resolves ambiguity.”

The evaluation was conducted in June 2026. Each model was run 10 times on 20 oncology-related questions—8 on historical data, 7 on the competitive landscape and 5 on clinical trial design and outcomes—generating a total of 600 outputs. “Truth” and “trust” were each scored on a scale of 0 to 8.