Research Areas

Multimodal communication

Virtually all sorts of human communication and meaning-making within and across national and cultural borders relies on elaborate “cocktails” of linguistic and non-linguistic semiotic resources or modalities: for example, words, sentences, pictures, films, facial expressions, gestures, colours, shapes, tastes, and smells. The examples span from printed and electronic news media, through ads, brands, fashion and product packaging, to opera performances and job interviews.

While any in-depth analyses of the ingredients of such “semiotic cocktails” are at present mainly performed in isolation by self-contained academic disciplines such as linguistics, visual communication and sensory science, there is a vast potential for addressing them in integration (for further arguments, see e.g. Jewitt, Bezemer & O’Halloran, 2016). Such integration can generate new insights and methodological tools for the practice-oriented domains and disciplines where these “cocktails” are mixed and exert their societal, cultural, and commercial influence within and across societies. Areas of particular relevance to MSC include, but are not restricted to, intercultural management, marketing, branding, and advertising.

Going beyond the meta-scientific orientation of traditional semiotics, we investigate “hands on”, i.e. on empirical/experimental grounds, how various sorts of “semiotic cocktails” work in practice, observing and testing their capability to support specific societal, cultural, and commercial goals. Existing approaches to the different elements of multimodal communication are naturally combined with each other in addressing real-life  challenges in areas spanning from product development and advertising to consumer protection, such as how to achieve fair communication through food packages and how to support a healthier lifestyle.

Main contact person: Viktor Smith

Visual communication

Vision is one of the main human senses and visual communication one of the main forms of human communication along with language. Examples of visual media are found everywhere we look: they include still images (e.g., drawings, paintings, and photographs), moving images (e.g., film and television), and an ever increasing number of digital images and associated platforms (e.g., computer-generated imagery, internet web pages, video games, and virtual reality).

At CogLab, we have conducted experiments in visual communication using a variety of stimulus materials and response measures. The stimulus materials that we have used include natural scenes, schematic and photographic faces, pictures of food, drink and other products, short film sequences, and different types of user interface. The response measures include accuracy scores, rating scales, reaction times, and eye tracking (all indirect measures of visual attention and cognitive processing). These experiments have addressed questions pertinent to a range of disciplines including psychology, marketing, consumer behaviour, and usability studies.

Another line of research concerns the influence of culture on visual perception. Studies in cross-cultural psychology suggest that Westerners tend to think analytically in terms of categories and rules, whereas East Asians tend to think holistically in terms of similarities and relationships. In a series of eye tracking studies, we have tested the related hypothesis that Westerners attend more to focal objects in a natural scene, while East Asians attend more to backgrounds and the relations between objects. So far, we have collected data for participants in Denmark, Russia, and China. This is of direct relevance for the department’s research on cross-cultural marketing, as well as for cross-cultural psychology and cultural studies.

Main contact person: Daniel Barratt

Language processing

Human language processing involves many steps, each of which is impressive in its complexity. To understand a spoken or written text, we have to recognise several hundred words per minute, with all their inherent and associated meanings; decode how those words are related in the sentence structure; and understand how individual sentences relate to our existing knowledge, the current situation, and the rest of the text. In spite of the complexity, language processing is usually fast, automatic and relatively error-free. In CogLab, we investigate these processes using eye-tracking and reaction time experiments that reflect the cognitive processes during language processing.

Understanding this process is relevant for learning more about contexts where language is instrumental. This includes advertising and PR (often in conjunction with images, as we explore in our research on multimodal communication), accounting and auditing, organisational communication, foreign language learning and cross-cultural communication. For instance, we investigate the information processing of loan officers and how it influences their decision whether to grant a loan or not; this gives us an insight into which types of information are most useful in accounting and auditing. We look at how the emotional valence of words influence their processing in first and second languages, knowledge which is applicable in marketing  as well as important for understanding differences between first and foreign languages. We also look at which characteristics of texts make them easy or difficult to understand, empirically investigating the underpinnings of good written communication.

Main contact person: Daniel Barratt