For a translator to turn one language (say, English) into another (say, Greek), she has to be able to understand both languages and what common meanings they point to, because English is not very similar to Greek.

It turns out that a similar task must be carried out within our own minds when translating visual information into linguistic information — a task Alon Hafri, an assistant professor in the Department of Linguistics and Cognitive Science at the University of Delaware and director of the Perception and Language (PAL) Lab, is trying to better understand. His research is on the connection between language and visual perception, or, in other words, between what we say and what we see.

Understanding these two activities — seeing and speaking — may at first seem trivial, Hafri said. Indeed, we're so good at these things that we do them without thinking. But seeing and speaking are some of the most complex processes that the human mind carries out. Hafri’s goal is to understand how they work and how those two systems share information.

Hafri gives a seemingly simple example: a cat on a mat. If we saw a cat on a mat, we'd have no problem describing the scene. Or if someone said, "Look, a cat on a mat," we'd easily be able to identify it out in the world. Yet at first glance, an image of a cat on a mat and the sentence “a cat on a mat” have nothing in common: the image has colors, edges, shapes and locations, while the sentence has sounds, words, phrases and such. 

“If you think about the problem that the mind has to solve when you're going from a visual scene to a sentence that you produce about it, there's nothing in common really between words in a certain order and a visual image,” Hafri said. “The kind of information that they start with is very different. Something I'm really interested in is how visual information gets translated — sort of like an English-to-Greek translator would do — into a format that language can use, such that we can talk about what we see. And vice versa, so that we can recognize the things people are talking about in the world. The kind of questions I ask are, well, how does that happen? What is the translation process?”