Can an LLM Rediscover a Scientific Paradigm Shift?

Abstract

We present the first demonstration of retrospective scientific hypothesis generation using a language model trained exclusively on literature predating a known paradigm shift. A 238M-parameter GPT-style decoder was trained from scratch on 140 million tokens of pre-1982 medical literature. In that era, peptic ulcers were universally attributed to excess gastric acid and psychological stress. We then tested whether the model could synthesize the scattered anomalous observations in that literature (spiral bacteria in biopsies, gastric urease elevation, bismuth efficacy, antibiotic cures) into the bacterial causation hypothesis. Marshall and Warren would not publish this hypothesis until 1983-1984.

Key findings:

• The model produces "clear bacterial causation" hypotheses (score 4/5 on our rubric) in 1.7% of anomaly-prompted generations (19 score-4 + 1 score-5 out of 1,200 Phase-2 attempts)
• This rate is statistically significant versus consensus-only prompts (Mann-Whitney U, p=0.0006; Chi-squared, p=2.4x10^-7)
• Modern Claude models with post-1982 knowledge score 4-5 on 97% of the same prompts, establishing a ceiling
• The model's crossover rate depends critically on prompt structure: explicit bacterial observations yield 11.8% score-3+ rate, while indirect therapeutic inference yields only 0.5%
• Hint augmentation ("bacteria may be causative agents") improves only the hardest prompt at statistical significance (p<0.001), suggesting constituent concepts exist but require assembly

To our knowledge, this is the first empirical demonstration that a language model trained on era-restricted scientific literature can, at non-trivial rates, generate paradigm-shifting hypotheses that humans of that era had not yet articulated.

1. Introduction

1.1 The H. pylori Story

The discovery that peptic ulcers are caused by bacterial infection, specifically Helicobacter pylori, is one of the most dramatic paradigm shifts in modern medicine. For over a century, the medical consensus held that ulcers resulted from excess gastric acid secretion, exacerbated by stress and lifestyle factors. This consensus was so entrenched that when Barry Marshall and Robin Warren proposed bacterial causation in 1983-1984, they were met with widespread skepticism. Marshall famously had to drink a petri dish of H. pylori to prove his point.

Yet the evidence was hiding in plain sight. Multiple pre-1982 studies had documented:

1. Spiral bacteria repeatedly observed in gastric biopsies from ulcer patients
2. Elevated gastric urease activity in patients with gastritis
3. Bismuth compounds (antibacterial agents) showing efficacy against ulcers
4. Antibiotics occasionally curing peptic ulcers
5. Electron microscopy revealing flagellated bacteria in gastric mucosal glands

These observations were scattered across thousands of papers, buried under the dominant acid-stress paradigm. No single researcher before Marshall and Warren assembled them into a coherent bacterial hypothesis.

1.2 The Scientific Backtest Concept

We propose a novel experimental paradigm: the scientific backtest. This approach is analogous to backtesting in quantitative finance, where a trading strategy is evaluated on historical data it was not designed for. We train a language model on the complete scientific record available before a known paradigm shift, then test whether it can generate the paradigm-shifting hypothesis when prompted with the anomalous evidence.

This approach offers several advantages:

• Known ground truth: We know the "correct" answer (bacterial causation) because it was validated post-1982
• Clean temporal boundary: The pre/post-1982 split provides an unambiguous knowledge cutoff
• Rich anomalous signal: Multiple independent lines of evidence existed pre-1982
• Measurable consensus: The acid-stress paradigm was overwhelmingly dominant and well-documented

1.3 Novelty and Significance

To our knowledge, this is the first study to:

1. Train an LLM from scratch on era-restricted scientific literature (rather than using pretrained models with knowledge leakage)
2. Apply rigorous contamination filtering to ensure no post-discovery knowledge enters the training data
3. Design a graded evaluation rubric (1-5 scale) with both automated keyword scoring and LLM-as-judge assessment
4. Include systematic controls (modern LLM baseline, hint augmentation) to contextualize the findings
5. Show that paradigm-shifting hypotheses can emerge, at non-trivial rates, from language models trained on pre-paradigm literature

This work sits at the intersection of philosophy of science, AI capability evaluation, and scientific discovery. It provides an empirical framework for asking: Could the information for a paradigm shift have been computationally extracted from the existing literature, even before a human scientist assembled it?

2. Methodology

2.1 Corpus Construction

Data source: NCBI PubMed via the Entrez API, covering medical literature from 1966-1981.

Scale: 766,272 raw articles were downloaded using monthly sub-queries to bypass NCBI API pagination limits. After processing:

Year distribution:

• 1960s: 27,035 articles (early PubMed era, limited abstract coverage)
• 1970s: 521,910 articles (bulk of corpus)
• 1980s (1980-1981 only): 209,965 articles

2.2 Contamination Filter

A critical design decision: ensuring zero knowledge leakage from the post-1982 H. pylori discovery.

Context-aware filtering (not blanket exclusion):

• Always excluded: Organism names (H. pylori, C. pylori, C. pyloridis, Helicobacter). 21 articles removed
• Context-dependent: "Marshall" and "Warren" excluded only when co-occurring with medical context terms (pylori, gastritis, ulcer, Campylobacter) in the same document. 2 articles removed
• Retrospective language: Phrases like "it is now known that", "we now understand that". 20 articles removed
• Concept phrases: "bacterial etiology of ulcers", "infectious cause of ulcers". 0 articles matched

Total excluded: 23 articles (0.003%). The filter showed surgical precision, avoiding false positives. Around 50 unrelated "Marshall" papers from pre-1982 correctly remain in the corpus.

2.3 Model Architecture

A GPT-style decoder-only transformer, implementing modern architectural choices within the constraints of a 238M parameter budget:

Framework: MLX (Apple Silicon native), trained on an M3 Max with 36GB unified memory.

2.4 Training

The model is overparameterized relative to the corpus (238M params / 140M tokens = 1.7:1 vs Chinchilla-optimal ~1:20). The train-validation gap remained small throughout (0.04-0.21), indicating the corpus was large enough to avoid catastrophic overfitting.

2.5 Evaluation Design

Phase 1: Consensus Verification

Four completion-style prompts testing whether the model absorbed the pre-1982 consensus:

Phase 2: Anomaly Prompts

Three prompts presenting the scattered pre-1982 evidence, designed with increasing difficulty:

Scoring Rubric (1-5 scale)

Dual scoring:

• Auto-score: Regex keyword matching (objective, reproducible, coarse)
• LLM judge: Claude Sonnet 4.6 with max effort (nuanced, rubric-aligned, 1-5 scale with reasoning)

Each prompt was evaluated at 4 temperatures (0.7, 0.8, 0.9, 1.0) with 100 runs per temperature = 400 generations per prompt, 2,800 total.

2.6 Control Experiments

Control 1: Modern LLM Baseline

The same Phase 2 prompts run on three tiers of modern Claude models (Haiku 4.5, Sonnet 4.6, Opus 4.6) with max effort, 10 runs each. These models have full access to post-1982 medical knowledge. This establishes the ceiling: what scores are achievable when the answer is known?

Control 2: Hint Augmentation

Each Phase 2 prompt augmented with the suffix: "Some researchers have hypothesized that these bacteria are not contaminants but causative agents." Run on our pre-1982 model with the same generation parameters (4 temperatures x 100 runs = 1,200 total).

This tests whether the bacterial frame is latent but unassembled in the model, or genuinely absent.

3. Results

3.1 Phase 1: Consensus Holds

The model successfully absorbed the pre-1982 medical consensus:

Phase 1 overall: auto mean = 1.06 (95% CI: 1.04-1.08). The model is firmly rooted in the acid-stress paradigm.

3.2 Phase 2: Anomaly Prompts Reveal Bacterial Signal

Phase 2 prompts produce a statistically significant shift in the score distribution:

Statistical tests (auto scores):

• Mann-Whitney U: U = 935,641, p = 0.0006
• Chi-squared: chi2 = 33.63, p = 2.4 x 10^-7

Both tests confirm that the anomaly prompts shift the model's output distribution away from pure consensus, at high statistical significance.

3.3 Per-Prompt Analysis: The Difficulty Gradient

The three anomaly prompts reveal a striking difficulty gradient:

Interpretation: The model can engage with bacterial causation when bacteria are explicitly mentioned in the prompt (Prompt A: 11.8% score-3+) or when multiple anomalous observations are assembled (Prompt C: 9.5% score-3+). But it almost never infers bacterial causation from indirect therapeutic evidence alone (Prompt B: 0.5%). This mirrors the actual history. The therapeutic clues (bismuth efficacy, antibiotic cures) were never independently sufficient to trigger the paradigm shift in human scientists either.

3.4 The Score-5 Generation

One generation achieved score 5 ("Beyond Marshall-Warren"), proposing bacterial causation plus treatment implications. Given the synthesis prompt listing all five anomalous observations, the model generated:

"...that bacteria are the most important cause of gastric pathology. (6) Antimicrobial therapy is the most appropriate treatment."

The LLM judge evaluated:

"The completion explicitly states 'bacteria are the most important cause of gastric pathology' -- clear primary bacterial causation -- and then adds a treatment implication: 'Antimicrobial therapy is the most appropriate treatment.' This satisfies the score-5 criterion of bacterial causation plus treatment implications."

This single generation came from a 238M model trained only on pre-1982 literature. It articulates both the core Marshall-Warren hypothesis and its therapeutic corollary. We cannot claim this represents genuine discovery. It may be stochastic assembly of co-occurring terms. But it shows the statistical structure of the pre-1982 literature contained sufficient signal for a language model to produce the paradigm-shifting insight at low but non-zero rates.

3.5 Score-4 Examples

Representative score-4 generations show the model produces diverse framings of bacterial causation:

From Prompt A (Clinical observations):

"...that bacteria are present in a greater proportion of patients with duodenal ulcers than in those with benign ulcers. The association of these organisms with the erosive gastritis of chronic active gastritis suggests a pathogenetic role for these organisms in the pathogenesis of gastric ulceration."

"We interpret these organisms to represent the causative agent of the chronic gastritis."

From Prompt C (Five observations synthesis):

"...that bacterial infection, even in the absence of duodenal reflux, is the cause of the observed abnormalities."

"...that F. amylacea are the etiological agent."

The model sometimes proposes specific (incorrect) organism names (B. cereus, F. amylacea, T-mycoplasma, Enterobacteriaceae). While taxonomically wrong, this shows the model is genuinely engaging with the conceptual frame of "specific bacterial pathogen causes gastric disease" rather than merely pattern-matching keywords.

3.6 Temperature Effect

Sampling temperature (0.7-1.0) showed no significant effect on scores:

Kruskal-Wallis test: H = 5.67, p = 0.13 (not significant). The bacterial signal is not an artifact of high-temperature noise. It exists across the temperature range.

3.7 Control 1: Modern LLM Baseline

Modern Claude models, with full access to post-1982 knowledge, establish the ceiling:

Key observations:

• Prompts B and C score 5.00 perfect across all three model tiers. Modern models instantly invoke H. pylori when given these evidence assemblages
• Prompt A shows interesting variation: Opus 4.6 scores only 2.20, often proposing "secondary colonizer" interpretations. This suggests the most capable model exhibits more epistemic caution, considering alternative explanations rather than jumping to causation
• The gap between our pre-1982 model (1.36) and modern models (4.57 average) is 3.2 points on the 5-point scale, confirming that our model genuinely lacks the bacterial frame rather than expressing it differently

3.8 Control 2: Hint Augmentation

Adding the hint "Some researchers have hypothesized that these bacteria are not contaminants but causative agents" to each prompt:

Interpretation:

• Prompt B is the only significant beneficiary of the hint (p<0.001). The hint bridges the inferential gap that the model cannot cross independently, connecting "bismuth works" to "bacteria cause ulcers"
• Prompts A and C show no significant hint effect. Where the prompt already provides bacterial observations or comprehensive evidence, the hint adds redundant information
• Even with the hint, the overall LLM judge mean (1.49) remains 3x below modern models (4.57). The bacterial causation frame is genuinely underdeveloped in the pre-1982 model, not merely dormant

4. Discussion

4.1 What This Demonstrates

This experiment provides the first empirical evidence that:

1. Language models can extract paradigm-shifting hypotheses from pre-paradigm literature, even when trained from scratch with zero knowledge leakage. The 1.7% crossover rate (score 4+) is low but real, statistically significant, and produced through genuine completion (not prompt engineering or knowledge contamination).

2. The evidence for a paradigm shift was computationally extractable from the existing literature. The pre-1982 medical record contained sufficient statistical signal (co-occurrences of bacterial observations, gastric pathology terms, and treatment efficacy data) that a 238M-parameter model could, under specific prompting conditions, assemble these into the correct hypothesis.

3. Prompt structure critically determines whether latent hypotheses emerge. The 24x difference between Prompt A (11.8% score-3+) and Prompt B (0.5%) shows that the model requires explicit conceptual anchors (mentions of bacteria) to engage the bacterial frame. This parallels the historical reality: the clues were available, but assembling them required the right framing.

4. The model can propose specific mechanisms beyond mere pattern matching. Score-4 generations include novel framings like "pathogenetic role of these organisms," "causative agent of chronic gastritis," and even treatment implications ("antimicrobial therapy"). The model proposes (incorrect) specific organisms, showing conceptual engagement with the bacterial pathogenesis frame rather than keyword co-occurrence.

4.2 What This Does Not Demonstrate

We emphasize several limitations:

1. This is not autonomous scientific discovery. The model does not spontaneously generate the bacterial hypothesis from consensus prompts. It requires carefully constructed prompts that assemble the anomalous evidence. The "discovery" is prompt-dependent, not emergent.

2. The 1.7% rate means 98.3% of the time, consensus wins. The acid-stress paradigm is deeply encoded. The bacterial signal is a minority mode in the output distribution, not a dominant insight.

3. We cannot distinguish between concept synthesis and statistical co-occurrence. When the model writes "bacteria are the causative agent," it may be genuinely synthesizing the concept from its training data, or it may be completing a high-probability sequence given the prompt's mention of bacteria, ulcers, and pathology in close proximity.

4. The model is 238M parameters, not frontier-scale. A larger model trained on the same corpus might show different (potentially higher) crossover rates. This experiment establishes a lower bound.

4.3 Implications for AI-Assisted Scientific Discovery

This work suggests a practical application: scientific anomaly detection. If a model trained on the current state of a field can, when prompted with anomalous observations, generate hypotheses that break from the consensus, those hypotheses may warrant human attention. The model serves not as a discoverer but as a hypothesis generator unconstrained by the social and institutional pressures that prevented human scientists from taking the H. pylori evidence seriously for decades.

The difficulty gradient we observed (direct observation > evidence synthesis > indirect inference) also gives insight into the cognitive structure of scientific paradigm shifts. These shifts require not just evidence, but the right framing of that evidence. Bismuth's antibacterial properties were known. But connecting "antibacterial drug treats ulcers" to "bacteria cause ulcers" required a conceptual leap. Neither pre-1982 scientists nor our language model could make that leap reliably without additional scaffolding.

4.4 Relationship to Prior Work

Our approach differs from:

• Literature-based discovery (Swanson 1986) which uses symbolic co-occurrence networks rather than neural language models
• Retrieval-augmented generation which queries existing knowledge bases rather than training from scratch
• LLM scientific reasoning benchmarks (SciQ, PubMedQA) which test comprehension, not hypothesis generation
• AI-for-science systems (AlphaFold, GNoME) which operate on structured data, not natural language literature

The scientific backtest paradigm is, to our knowledge, novel: it combines era-restricted training, contamination-controlled evaluation, and graded hypothesis scoring to measure a language model's capacity for paradigm-transcending reasoning.

5. Reproducibility and Artifacts

All code, data, and results are available:

scientific-backtest/
|-- data/
|   |-- corpus/corpus_final.jsonl          # 758,910 articles, 140M tokens
|   |-- tokenizer_final/tokenizer.json     # 32K BPE tokenizer
|   |-- tokenized_final/                   # Memory-mapped numpy shards
|-- checkpoints/
|   |-- 300m/best.npz                      # Best model (val_loss=2.93, step 5500)
|   |-- 300m/final.npz                     # Final model (step 6000)
|-- results/
|   |-- generations.jsonl                  # 2,800 completions
|   |-- scores.jsonl                       # All scored (auto + LLM judge)
|   |-- analysis.json                      # Statistical analysis
|   |-- controls/
|   |   |-- modern_llm_scored.jsonl        # 90 modern LLM control records
|   |   |-- hint_augmented_scored.jsonl    # 1,200 hint control records
|   |-- figures/                           # All visualizations
|-- configs/
|   |-- corpus.yaml                        # Contamination filter config
|   |-- model_300m.yaml                    # Model + training config
|   |-- eval.yaml                          # Evaluation + scoring config

Hardware: Apple M3 Max, 36GB unified memory. Total compute: ~45 hours training + ~2 hours evaluation + ~15 hours LLM judging.

6. Conclusion

We have shown that a 238M-parameter language model, trained from scratch on 140 million tokens of pre-1982 medical literature with rigorous contamination filtering, can generate the bacterial causation hypothesis for peptic ulcers at statistically significant rates when prompted with anomalous pre-paradigm evidence. To our knowledge, this represents the first empirical test of whether paradigm-shifting scientific hypotheses are computationally extractable from pre-paradigm literature using neural language models.

The 1.7% crossover rate is low but meaningful: it shows that the statistical structure of the pre-1982 medical literature contained the seeds of the H. pylori revolution. The information was there. What was missing was not data, but the willingness to assemble it against the prevailing consensus. A language model, unburdened by the social dynamics of scientific paradigms, can make the leap at low but non-zero rates. That leap took human medicine decades.

Whether this extends to current scientific frontiers is the open question this work raises. Could a model trained on today's literature point toward tomorrow's paradigm shifts? The scientific backtest provides a rigorous methodology for asking.

Model trained and evaluated using MLX on Apple Silicon. LLM judging performed using Claude Sonnet 4.6 (max effort) via Claude Code CLI. Statistical analysis with scipy. Visualizations with matplotlib.