TY - ABST

T1 - Digitalizing the visible world:

AU - Nielsen, Ann Merrit Rikke

AU - Due, Brian Lystgaard

AU - Lüchow, Louise

PY - 2021

Y1 - 2021

N2 - Knowledge of objects in spatial environments is obtained through multisensorial resources, such as sight, hearing, touch and smell (Mondada, 2019). For most people the visual sense takes priority in the perception of objects’ spatial relation to the sensing body and the social world (Workman, 2016). We mostly use our eyes to unnoticeably perceive objects and their position in the environment (e.g. Goodwin & Goodwin, 1996). Visually impaired people (VIP), however, have no or very limited access to the visual aspects of these spatial relations, when orienting towards their immediate surroundings. The emergence of computer vision and natural language processing (NLP) in accessible mainstream technology, such as smartphones, enables the device to “translate” the visual world into language descriptions. We explore in this paper the use of the app Microsoft SeeingAI which enables VIP to receive computerized descriptions of “visual” information about objects, people and places by scanning the environment - like using a flashlight to receive information about objects in the dark. Based on a video ethnographic collection of VIP scanning the shelfs when grocery shopping, we investigate how this ‘digitalizing’ of an everyday practice is done in situ. Through ethnomethodological multimodal conversation analysis (Streeck et al., 2011) we investigate environment scanning as a specific aspect of distributed perception (Due, 2021a), that is focusing on the co-operative actions between sense-able agents. As VIP rely on other multisensorial resources than sight alone, the question is how the practice of scanning is achieved in situ. Our analysis shows, that the practice of scanning inanimate distant objects require complex coordination of the device used to scan in relation to the object(s) being scanned with regards to both distance, angle of the device, and the location of the relevant feature of the object. This paper focuses on three particular phenomena: i) scanning nearby surroundings for the location and identification of objects, ii) scanning specific objects to obtain information about the object, and iii) scanning text on the object. This paper contributes to research into blind and visually impaired people, the senses and perception in interaction, shopping activities (Due, 2017) and interactions with non-human robotic agents (cf. Due, 2021b). Due, B. L. (2017). Respecifying the information sheet: An interactional resource for decision-making in optician shops. Journal of Applied Linguistics and Professional Practice, 14(2), 127–148. https://doi.org/10.1558/jalpp.33663Due, B. L. (2021a). Distributed Perception: Co-Operation between Sense-Able, Actionable, and Accountable Semiotic Agents. Symbolic Interaction, 44(1), 134–162. https://doi.org/10.1002/symb.538Due, B. L. (2021b). RoboDoc: Semiotic resources for achieving face-to-screenface formation with a telepresence robot. Semiotica, 238, 253–278. https://doi.org/10.1515/sem-2018-0148Goodwin, C., & Goodwin, M. H. (1996). Formulating Planes: Seeing as a Situated Activity. In Cognition and Communication at Work, (eds.) David Middleton and Yrjö Engestrom (pp. 61–95). Cambridge University Press.Mondada, L. (2019). Rethinking Bodies and Objects in Social Interaction: A Multimodal and Multisensorial Approach to Tasting. In U. T. Kissmann & J. van Loon (Eds.), Discussing New Materialism: Methodological Implications for the Study of Materialities (pp. 109–134). Springer Fachmedien. https://doi.org/10.1007/978-3-658-22300-7_6Streeck, J., Goodwin, C., & LeBaron, C. D. (2011). Embodied interaction: Language and body in the material world /. University Press.Workman, J. (2016). Phenomenology and Blindness: Merleau-Ponty, Levinas, and an Alternative Metaphysical Vision. Electronic Theses and Dissertations. https://digitalcommons.du.edu/etd/1210

AB - Knowledge of objects in spatial environments is obtained through multisensorial resources, such as sight, hearing, touch and smell (Mondada, 2019). For most people the visual sense takes priority in the perception of objects’ spatial relation to the sensing body and the social world (Workman, 2016). We mostly use our eyes to unnoticeably perceive objects and their position in the environment (e.g. Goodwin & Goodwin, 1996). Visually impaired people (VIP), however, have no or very limited access to the visual aspects of these spatial relations, when orienting towards their immediate surroundings. The emergence of computer vision and natural language processing (NLP) in accessible mainstream technology, such as smartphones, enables the device to “translate” the visual world into language descriptions. We explore in this paper the use of the app Microsoft SeeingAI which enables VIP to receive computerized descriptions of “visual” information about objects, people and places by scanning the environment - like using a flashlight to receive information about objects in the dark. Based on a video ethnographic collection of VIP scanning the shelfs when grocery shopping, we investigate how this ‘digitalizing’ of an everyday practice is done in situ. Through ethnomethodological multimodal conversation analysis (Streeck et al., 2011) we investigate environment scanning as a specific aspect of distributed perception (Due, 2021a), that is focusing on the co-operative actions between sense-able agents. As VIP rely on other multisensorial resources than sight alone, the question is how the practice of scanning is achieved in situ. Our analysis shows, that the practice of scanning inanimate distant objects require complex coordination of the device used to scan in relation to the object(s) being scanned with regards to both distance, angle of the device, and the location of the relevant feature of the object. This paper focuses on three particular phenomena: i) scanning nearby surroundings for the location and identification of objects, ii) scanning specific objects to obtain information about the object, and iii) scanning text on the object. This paper contributes to research into blind and visually impaired people, the senses and perception in interaction, shopping activities (Due, 2017) and interactions with non-human robotic agents (cf. Due, 2021b). Due, B. L. (2017). Respecifying the information sheet: An interactional resource for decision-making in optician shops. Journal of Applied Linguistics and Professional Practice, 14(2), 127–148. https://doi.org/10.1558/jalpp.33663Due, B. L. (2021a). Distributed Perception: Co-Operation between Sense-Able, Actionable, and Accountable Semiotic Agents. Symbolic Interaction, 44(1), 134–162. https://doi.org/10.1002/symb.538Due, B. L. (2021b). RoboDoc: Semiotic resources for achieving face-to-screenface formation with a telepresence robot. Semiotica, 238, 253–278. https://doi.org/10.1515/sem-2018-0148Goodwin, C., & Goodwin, M. H. (1996). Formulating Planes: Seeing as a Situated Activity. In Cognition and Communication at Work, (eds.) David Middleton and Yrjö Engestrom (pp. 61–95). Cambridge University Press.Mondada, L. (2019). Rethinking Bodies and Objects in Social Interaction: A Multimodal and Multisensorial Approach to Tasting. In U. T. Kissmann & J. van Loon (Eds.), Discussing New Materialism: Methodological Implications for the Study of Materialities (pp. 109–134). Springer Fachmedien. https://doi.org/10.1007/978-3-658-22300-7_6Streeck, J., Goodwin, C., & LeBaron, C. D. (2011). Embodied interaction: Language and body in the material world /. University Press.Workman, J. (2016). Phenomenology and Blindness: Merleau-Ponty, Levinas, and an Alternative Metaphysical Vision. Electronic Theses and Dissertations. https://digitalcommons.du.edu/etd/1210

M3 - Conference abstract for conference

Y2 - 23 June 2021 through 25 June 2021

ER -