A Japanese tech startup, Sakana AI, has claimed to have created the first AI-generated peer-reviewed scientific paper. The company announced that a paper produced by its AI Scientist-v2 passed the peer-review process at a workshop in the International Conference on Learning Representations (ICLR), a top AI conference. Sakana AI partnered with the University of British Columbia and the University of Oxford for this project.
The AI-generated paper underwent the same rigorous peer-review process that human researchers face. The company believes that future generations of The AI Scientist will usher in a new era in science. However, the achievement is not as groundbreaking as it might initially appear.
Sakana admitted that the AI occasionally made citation errors, such as misattributing a method to a 2016 paper instead of the original work from 1997.
Sakana AI paper faces review concerns
Additionally, the paper was withdrawn after initial peer review and did not go through further scrutiny, such as a “meta-review,” which could have resulted in its rejection.
Matthew Guzdial, an AI researcher and assistant professor at the University of Alberta, cautioned that Sakana’s results could be misleading. “The Sakana folks selected the papers from a number of generated ones, meaning they used human judgment in picking outputs they thought might get in,” he stated. “This shows that humans plus AI can be effective, not that AI alone can create scientific progress.”
Mike Cook, a research fellow at King’s College London specializing in AI, questioned the rigor of the peer reviewers and the workshop itself.
“New workshops are often reviewed by more junior researchers,” he observed. “It’s arguably easier to get an AI to write convincingly about failures.”
Sakana emphasized that its goal wasn’t to showcase groundbreaking scientific work but to assess the quality of AI-generated research and to underscore the need for norms in AI-generated science. The company aims to continue engaging with the research community to ensure that AI-generated research does not undermine the scientific peer review process.
Photo by; Artem Podrez on Pexels
