UvA student discovers flaw in research on psychedelics
Not every student takes the trouble to reproduce the results of a research article themselves. Artificial intelligence student Paul Lodder (25) did, and when he discovered an error, it earned him a co-authorship. “The authors responded with incredible academic integrity.”
In preparation for the Neural Dynamics and Deep Learning course, student Paul Lodder browsed through the literature. He wanted to know what had already been done and what he might be able to build on. He did not need the course to complete his master's degree, but hoped to find the direction in which he wanted to write his thesis.
LSD as therapy
During his search, he came across a paper from the University of Valparaíso in Chile led by researcher Rubén Herzog. The article was about what happens in the brain when taking the psychedelic drug LSD.
Based on a computer model, the paper claimed that LSD increases the amount of chaos—also known as entropy—in the brain. That result was consistent with studies conducted in human brains. LSD, if combined with therapy, could therefore help to break through entrenched thought patterns.
Lodder first wanted to be able to reproduce the results of the paper. “Because then I knew for sure that I was building on something and not working on an island.” The study used a computer model to simulate the brain that he — due to his above-average enthusiasm for programming — was able to get to work on his computer.
Instructions from Chile
Some data was missing from the paper, so Paul turned to Chilean author Rubén Herzog. “Rubén immediately responded very helpfully and provided me with instructions on how to run the simulations.”
Lodder got the model up and running with his own programming code and then had to calculate the entropy with the data from the model. Except he calculated a completely different entropy. “My results showed that LSD caused less entropy in the brain. Exactly the opposite of what the paper claimed.”
He e-mailed the Chilean scientist again. At that point, Lodder thought he had a parameter wrong, something that can happen quite easily in programming. After a few suggestions from Rubén Herzog, he still didn’t get the same results as in the paper. “I wrote: ‘You must be busy, but unfortunately I still don’t get the same results.’”
Instead of leaving it at that, Herzog sent his own analysis code to calculate the entropy. Lodder was able to reproduce the same results as in the paper with that analysis code. So he searched further in Herzog’s analysis code. What was different from his own code? Lodder explains: “Then I saw that there was a typing error in the formula that was supposed to calculate the entropy. As a result, the formula was no longer correct. When I adjusted that, I got the same result with Rubén’s code as with my own analysis code. That meant that the results in the paper were not correct.”
In slight confusion, he sent his results to Valparaíso and soon received a response. The Chilean agreed that there was an error in the code. He also wrote that he was extremely grateful for the work of the Dutch student and emphasized the importance of reproducing results. In the e-mail, Lodder was also invited to co-author the follow-up paper Herzog wanted to write once the error was corrected.
Disappointed
Although Lodder calls the reaction of the Chilean researcher "extremely academically honest" and only praises their collaboration, he was initially disappointed. “If I happen to find this error,” he reasoned, “how much more research is there whose results cannot be reproduced?”
Lodder says it is distressing that this occurs precisely in computer science. In that field, you can reproduce the results better than in any other field by supplying the code with the article. “The script with code can do the complete analysis and recreate every figure and table in the paper.”
But in practice, this is often not the case, he says. And if the code is already included, it is often incomplete. “That makes it unnecessarily difficult to build on science that has already taken place.”
According to Lodder at the UvA, more can be done about this. “Computer science students could be employed to review papers and provide well-documented, reproducible code with instructions to get it working. They could do this in exchange for a possible co-authorship.”
Withdrawn
Nature, the publisher of the paper, has also become aware of the error and withdrew the article on September 15, 2022. Above the online article, the words “RETRACTED ARTICLE” now appear.
Paul Lodder has kept in touch with Rubén Herzog and met him this summer at a conference in Amsterdam. Herzog also introduced him to other scientists in the field. Lodder is now writing his master’s thesis with a group of researchers from Argentina and Mexico.
The follow-up paper with Paul Lodder’s name on it has not yet been published. “It is a great honor to stand among the big names in the field on the paper.” Incidentally, the follow-up paper uses a model for the brain that is closer to biology. And in that model, LSD does increase entropy in the brain.
