Sometimes you read a question you never thought of before. I never wondered how we understand emoji. But still, the answer can be something relevant and interesting. Because even when emoji are used to substitute for words, we still understand the sentence. Isn’t that strange?

From the press release:

Emoji are small image icons that originated in Japan: Japanese “e” means picture, “moji” means word. “We set out to explore to what extent reading emoji is more similar to reading words or interpreting pictures,” explains Tatjana Scheffler. To this end, the researchers conducted an online study where participants read sentences in which some words were replaced by emoji, and measured the reading time for each word. Subsequently, they asked questions to test whether the participants understood the sentences.

“As we had suspected and as other studies suggest, our participants easily understood sentences with emoji instead of nouns,” the researcher points out. The comprehension questions were even answered correctly slightly more often – but not significantly so – following the sentences containing emoji than following the same sentences without any emoji.

Normal words, however, do not only lead to the meaning while they are being processed; they also have other properties that are activated during reading: for example their pronunciation. And words can be homophones: when two words have the same pronunciation but different meanings, for example like the knight and the night. The researchers wanted to find out if emoji can also trigger this phenomenon. “Therefore, we had our test participants read sentences in which the emoji doesn’t show the intended object but its homophone,” Tatjana Scheffler explains.

“We showed that the sentences were almost always understood correctly, even when homophones were used,” points out Tatjana Scheffler. “This indicates that an emoji can be broken down into a complete ‘lexicon entry’ that contains information about its pronunciation. Based on this information, the participants then derive the other, homophonic meaning.”

Still, the team also showed that the reading times differed considerably. The average reading time for a written-out word is approximately 450 milliseconds, for a corresponding emoji approx. 800 milliseconds and for an emoji depicting a homophone more than 900 milliseconds.

A detour via the image takes time

This leads the researchers to conclude that the image must be interpreted first when reading emoji. Emoji are generally less recognisable and more unusual, and therefore not as easy to read as written words. “This is also supported by the fact that those test participants who self-assessed as using emoji often, read matching emoji more quickly,” Tatjana Scheffler explains. Since readers can also derive a pronunciation for emoji just like for words, homophone emoji are also readily understood. This takes longer, however, because the visual information has to be suppressed and the meaning of the homophonic word has to be recalled. “In this case, the fact that someone is used to emoji is no longer helpful. Participants who use emoji more often read the homophone emoji just as slowly as the others,” Tatjana Scheffler concludes.

Together with the team at Charité, she plans to conduct a similar study with people living with schizophrenia. Since some of them have difficulty identifying non-literal meanings, a comparison with a control group should provide additional insight into language processing, the linguistic structure of non-literal meaning and the linguistic effects of schizophrenia.

Abstract of the study:

In computer-mediated communication, emojis can be used for various purposes. As small graphical images, many emojis depict abstract or concrete objects ideogrammatically. We report on a self-paced reading experiment of sentences containing emojis. We tested to what extent emojis encode lexical meanings when used in a sentence context. First, we confirm earlier findings that sentence comprehension does not suffer when emojis replace words. Second, we show that in addition to the graphically encoded concept, emojis in some cases enable the retrieval of an entire lexical entry, including the phonological value of the associated word. This means that even emojis showing a homophonous noun to the target word, such as “palm (tree)” for “palm (of hand)” can be interpreted correctly in context. Based on measured differences in the reading times between words, emojis depicting the intended target referent, and emojis depicting a homophonous noun, we propose a context dependent account of emoji interpretation.