Two researchers wanted to check ChatGPT’s ability to be funny and make jokes. However, the results have not been especially what they were looking for.
The virality of ChatGPT causes researchers to resort to this tool to precisely know its capabilities. In this case, 2 German researchers wanted to check the humor levels of this tool based on artificial intelligence —they studied version 3.5 instead of the newest, GPT-4—.
“To prove how rich the variety of jokes is on ChatGPT, we ask you to tell a joke a thousand times”they write. “All responses were grammatically correct. Almost every output contained exactly one joke just by entering the message, ‘Know a good joke?'”
However, the results were not positive at all: during a test, 90% of 1,008 generations of jokes that ChatGPT released, were the same 25 jokes.
Apart from this, they found that to get an output of 1,008 responses the tool simply mixed jokes or even the original creations of the language model didn’t always make sense. For example he said: “Why did the man put his money in the blender? He wanted to make time fly”.
humor is not his forte
With that result they wrote in their study the following: “It cannot yet confidently create intentionally funny original content. Observations from this study illustrate how ChatGPT learned a specific joke pattern rather than actually being funny”.
They further noted that there were 2 main limitations to the study. The first is that humor is subjective, therefore a reliable assessment is difficult as it totally depends on the user’s interpretation of whether they find it funny or not.
The second limitation is that the output of ChatGPT cannot be traced back to specific input data. Since ChatGPT is not an open source large language model, the team’s observations are based solely on system results.
After the publication of the study some they responded by accusing reinforcement learning through human feedback (RLHF) as the big culprit. This technique that is used during the model training process uses reinforcement learning methods to directly improve a language model thanks to human work.
Despite its virtues when it comes to avoiding bias, profanity or even the use of sexist or racist termsa great aspect is lost with it and it is the creativity of the model itself. Considering that jokes are not always politically correct, it is to be expected that ChatGPT is not the comedian you were looking for.
