Nature flags doubts over Google AI study, pulls commentary – Retraction Watch
Retraction Watch
Tracking retractions as a window into the scientific process
A new editor’s note in Nature highlights concerns about a paper by Google researchers who claimed computer chips designed in just a few hours using artificial intelligence beat chip plans that human experts took months to develop.
In the note, published September 20, the journal stated:
Readers are alerted that the performance claims in this article have been called into question. The Editors are investigating these concerns, and, if appropriate, editorial action will be taken once this investigation is complete.
The day after the editor’s note, Nature pulled an accompanying News & Views article extolling Google’s results. According to the retraction notice, the article’s author, Andrew B. Kahng of the University of California, San Diego, who was also a reviewer of the internet company’s research, had changed his assessment of the work following publication.
Kahng was not available for an interview. A spokesperson for Nature told us the journal takes “all concerns raised about papers we have published seriously,” adding:
We are currently assessing concerns that have been raised with us, but we cannot discuss the specifics of those concerns relating to any particular paper while such post-publication assessments are underway.
Google’s paper, “A graph placement methodology for fast chip design,” has been embroiled in controversy since it was published in 2021. It has been cited well over 100 times, but critics argue the article didn’t include enough detail to allow others to vet the findings and say Nature‘s decision to publish it was a mistake.
In an effort to reproduce the results using reverse-engineering, Kahng and his colleagues found human experts typically beat Google’s machine-learning system, as they reported in a preprint conference paper published earlier this year. However, they had no way “to replicate the pre-training described in the Nature paper,” they wrote.
What’s more, a Google researcher who was fired after he questioned the paper in Nature alleged in a lawsuit earlier this year that the tech giant’s “fraudulent” claims to have revolutionized chip layout using AI were tied to efforts to commercialize its software, as reported by The Register.
The company referred us to a statement issued earlier to the New York Times and Wired by Zoubin Ghahrahmani, vice president at Google DeepMind:
We thoroughly vetted the original Nature paper and stand by the peer-reviewed results — the authors’ many contributions to chip design have already helped us design more energy-efficient and useful AI hardware. We’ve open-sourced this work for the external research community, and multiple teams continue to advance this important area of research.
Anna Goldie, one of the corresponding authors of the paper and formerly a researcher at Google Brain, pointed us to a statement she issued in March with first author Azalia Mirhoseini, also a former Google researcher.
They noted, among other things:
Our [reinforcement learning] method has been used in production for multiple generations of Google’s flagship AI accelerator (TPU), including the latest, and chips with layouts generated by our method have been manufactured and are currently running in Google datacenters.
Being physically manufactured at sub-10 nm is a level of validation that goes far beyond what is present in almost any paper.
The pair also took issue with Kahng’s preprint conference paper, which they said “mischaracterizes” their work for a number of reasons laid out in their statement.
“Ultimately,” they added, “we believe that this is just the beginning and that learning-based methods for chip design will have profound impact on both hardware and machine learning itself.”
Like Retraction Watch? You can make a tax-deductible contribution to support our work, follow us on Twitter, like us on Facebook, add us to your RSS reader, or subscribe to our daily digest. If you find a retraction that’s not in our database, you can let us know here. For comments or feedback, email us at [email protected].
Your email address will not be published. Required fields are marked *
This site uses Akismet to reduce spam. Learn how your comment data is processed.
