Scientific inquiry stands as one of the most intricate human endeavors. Initially, scientists must assimilate the existing corpus of knowledge and pinpoint a critical lacuna. Following this, they must craft a research query, devise a methodology, and execute an experiment in search of answers. Upon gathering results, the process culminates in analysis and interpretation, often leading to new questions.
Can Science Be Mechanized?
Can such a sophisticated process be mechanized? Recently, Sakana AI Labs heralded the creation of an “AI scientist”—an artificial intelligence system purported to autonomously pioneer discoveries within the realm of machine learning.
Harnessing generative large language models (LLMs) akin to those powering ChatGPT and other AI conversational agents, this system can conjure ideas, code algorithms, generate plots, and author scientific manuscripts, inclusive of citations, encapsulating the entirety of the research cycle at a mere US$15 per paper—a fraction of the cost of a scientist’s midday meal.
These declarations are indeed monumental. But do they hold water? And even if they do, would an onslaught of AI-generated papers, produced at a superhuman pace, genuinely benefit the scientific community?
How Computers ‘Do’ Science
A significant portion of scientific work occurs in the public domain, and the vast majority of scientific knowledge is recorded in literature, making it accessible. Millions of scholarly articles are freely available through repositories like arXiv and PubMed.
LLMs trained on this extensive data reservoir capture the lexicon and patterns of scientific discourse. Hence, it is perhaps unsurprising that a generative LLM might produce something resembling a credible scientific paper—it has ingested myriad examples from which it can emulate.
What remains ambiguous is whether an AI system can generate a novel and interesting scientific paper. Novelty is the cornerstone of valuable science.
The Pursuit of Novelty in Scientific Papers
Scientists are not interested in reiterations of known facts. They seek to uncover novel insights, especially those that deviate significantly from established knowledge. This necessitates discernment regarding the scope and worth of new contributions.
The Sakana system addresses the issue of “interestingness” through two mechanisms. Initially, it assigns a “similarity score” to new paper ideas by comparing them to existing research indexed in the Semantic Scholar repository, discarding anything too derivative.
Additionally, Sakana’s system incorporates a “peer review” phase—utilizing another LLM to assess the quality and originality of the generated paper. This review process is informed by abundant examples of peer critique available on platforms like openreview.net, which LLMs have also absorbed.
Limitations of AI in Evaluating Scientific Quality
The reception to Sakana AI’s output has been mixed, with some critiques labeling it as generating “endless scientific slop.” Even the system’s internal evaluations often rate its papers as weak, at best. While improvements may occur as the technology advances, the value of automated scientific papers remains questionable.
The proficiency of LLMs in appraising research quality is also uncertain. My own forthcoming work in Research Synthesis Methods reveals that LLMs struggle to accurately assess bias in medical research, although this capability may evolve over time.
Sakana’s system automates discovery within computational research—a domain where experiments involve code, a form of structured text that LLMs can be trained to generate, making this type of automation more feasible compared to other scientific fields requiring physical experimentation.
AI as an Assistant, Not a Replacement
AI researchers have long been developing tools to assist scientists. Given the sheer volume of published research, even locating relevant studies can be daunting.
Advanced search engines leverage AI to aid scientists in identifying and synthesizing existing knowledge. Examples include Semantic Scholar, Elicit, Research Rabbit, scite, and Consensus.
Text mining tools like PubTator delve deeper into papers, extracting crucial information such as specific genetic mutations and associated diseases, which is invaluable for curating and organizing scientific data.
Machine learning has also been employed to support the synthesis and analysis of medical evidence, exemplified by tools like Robot Reviewer. Summaries that juxtapose claims from various papers, provided by services like Scholarcy, facilitate comprehensive literature reviews.
All these tools are designed to enhance scientists’ efficiency, not to supplant them.
Potential Risks of AI-Driven Research
Despite Sakana AI’s assurances that the role of human scientists will remain intact, the company’s vision of a “fully AI-driven scientific ecosystem” poses significant implications for the field.
One concern is that a deluge of AI-generated papers could saturate the scientific literature, leading to future AI systems being trained on AI-generated content, which may trigger a “model collapse.” This scenario could result in diminishing returns on innovation.
However, the consequences extend beyond AI science systems themselves. The existence of “paper mills,” which produce fraudulent papers, is already a recognized issue. The ability to produce a scientific paper for US$15 with a rudimentary prompt could exacerbate this problem.
The burden of sifting through an avalanche of automatically generated research could overwhelm human scientists. The peer review system, already strained, may not withstand the influx of research with dubious quality.
Science fundamentally relies on trust. The integrity of the scientific process is paramount, ensuring that our understanding of the world (and its technologies) remains accurate and progressive.
Conclusion: The Future of AI in Science
The emergence of a scientific ecosystem dominated by AI systems raises profound questions about the nature and value of this process, and to what extent we should place our trust in AI-generated science. Is this the direction we wish to take?
