Karolina Drożdż wins UvA thesis prize for research on AI
Polish student Karolina Drożdż has been named this year’s winner of the UvA Thesis Prize. The jury awarded her first prize for her research into the language comprehension of AI models, earning her €3,000. “In some respects, the AI models clearly outperformed the humans.”
This year’s first prize in the UvA Thesis Prize went to Karolina Drożdż (25). The Polish student completed the UvA’s Research Master’s in Psychology majoring in Brain and Cognitive Sciences, where she conducted research into the semantic abilities of language models. In other words: how well does an AI model actually understand the texts it processes?
How did you go about finding that out?
“I presented a series of short stories to both humans and large language models. In one such story, a character – let’s call her Dora – is cleaning her room. First, Dora picks up a pen and puts it in a box. In that same box, she finds a wallet, which she then places in a basket. Using stories like these, we examine how well humans and AI models are able to keep track of different objects. Do they understand what each move implies, and where the pen or the wallet ends up as a result?”
And how did the AI models perform compared to humans?
“When it came to complexity, the AI models significantly outperformed humans. For people, the more complicated the stories become, the worse they perform. The more advanced language models clearly had much less trouble with this. Even when a story contained the maximum number of different objects and movements, the AI models were able to handle it very well.”
“However, there is an important caveat: the models perform better when we look directly at how likely they rate each possible answer. But when we ask them a question and they have to answer in a single word, they perform worse than humans. In other words, the model’s predictions point to the right answer, but that doesn’t always come through when it has to state it. This goes wrong more often than it does for humans.”
Why is it important to know this?
“Because we are currently dealing with what is known as the black box problem: we feed input into an AI model, all sorts of things happen inside that we cannot directly observe or interpret, and eventually we see the output. To maintain control over these models, we need to understand better how they work. Some people discuss their most personal problems with these kinds of models, so we need to understand how they arrive at certain answers. How do they process information? And what kind of higher-level internal understanding are they capable of developing?”
Should we be interacting with AI differently?
“I don’t know whether my findings could directly change the way we interact with AI, but I do hope they highlight an important aspect of how language models work. They sometimes appear to behave like humans because they are trained on human language, but the way they process language is completely different from the way humans do.”
Finally, what was it like to win the prize?
“It was really special. I hadn’t expected to win, but when I did, I was incredibly happy that my family had travelled all the way from Poland to be there. It was a wonderful way to bring my five years in Amsterdam to a close. For me, this is confirmation that I'm on the right track with this subject, which I will also be researching during my PhD at the University of Warsaw and IDEAS Research Institute.”
