Author: Saroosh Bilal (UAM)

The following REPAIRS toolbox is available under the CC-BY licence (Creative- Commons:  https://creativecommons.org/). This implies that others are free to share and adapt our works under the condition that appropriate credit to the original contribution (provide the name of the REPAIRS consortium and the name the authors of the toolbox when available, and a link to the original material) is given and indicate if changes were made to the original work. 

Be Ecologically Correct: A Toolbox for Developing Functional Sensory Substitution Devices

Chapter 1: Introduction to Toolbox

            According to WHO nearly 250 million (Bourne et al., 2017) people suffer from moderate to severe blindness. This large number of individuals implies the need for effective rehabilitation aids to improve their quality of life. Among many other challenges, they find indoor and outdoor navigation quite problematic (Goldschmidt, 2017).  Scientific communities are striving to design better rehabilitation aids to address the navigation troubles being faced by visually impaired individuals in their day-to-day lives.

            This toolbox aims to provide readers with the ecological framework as well as a list of components for designing the navigation aids for blinds and provide the readers with a consolidated checklist for development, testing, and improvement in existing or future devices based on the ecological perspective.

            We will start this discussion by first understanding human perception from two different schools of thought, the first one is the cognitive/ traditional approach, and the other one is the ecological approach. In the third chapter we will provide readers an understanding of how individuals with visual impairments perceive their surroundings and a brief description of sensory substitution devices (SSDs). Later, in chapter 4, we will build the narrative for understanding the need of considering the ecological approach while designing the rehabilitative navigating aids for blinds. Lastly, in chapter 5, we will provide readers with a consolidated checklist that satisfies the ecological framework and device components to design effective navigation aids for blinds.

Chapter 02: Cognitive vs. Ecological Approach of Perception

• Cognitive Approach

      The cognitive approach has taken the perspective that the pivotal system that underlies human movement behavior is a representation-based information processing system. It often defines certain regularities of behavior in terms of centralized computational process by receiving, processing, manipulating, storing, and generating information to realize an action (von Eckardt, 1993; Lakoff & Johnson,1999). It considers that perception entails the enrichment of ambiguous ambient energy patterns by means of, say, unconscious inference, statistical weighting, or use of internal representations and memory (Fodor & Pylyshyn, 1981). The main understanding of traditional cognitive approach is that sense organs provide information about surroundings (e.g. objects, events) in an impoverished way, and therefore, nervous system provides numerous additions to the stimulation that leads to “perception”, which is enriched and meaningful. Structuralism of early nineteenth century purposed “perception as summation of multiple meaningless sensations”. As otherwise these are merely physical attributes of the impinging energy such as intensity, frequency etc.  These multiple meaningless sensations then must be added with prior images of the same nature in the memory; as a single meaningless sensation fails to identify the objects or events (Michaels & Carello, 1981).

      So, this approach suggests the existence of unconscious inference to fix the poorly learned cues and rules and are unconsciously applied to create mental representations of the world. Helmholtz (1910/2000) suggested that every ambiguity in the retinal image could be solved by a combination of unconscious inference and the principle of maximum likelihood. The result was a mental representation (a cleaned-up and interpreted copy) of the world that could be used to plan and execute behavior. It focusses on isolated stimuli and internal mental processes without adequately accounting for the rich and dynamic perceptual information present in the environment, thus, overlooks the dynamic interactions between organism and environment.

• Ecological Approach

Information

      J.J Gibson proposed the ecological approach to understand human perception. For him, the way an organism establishes a contact with his environment will determine the perception and it is a phenomenon to be understood in terms of lawful regularities and symmetrical principles defined at the ecological scale of organisms and environments, rather than in terms of mental states or formal languages of representation and computation (e.g., Turvey et al.,1981). The underlying essence of this approach is that information is particular to the environmental properties such as objects, surfaces, events etc., and perception is specific to that information (Turvey & Shaw, 1999). Gibson’s (1979) ecological approach explores the specificity between the structured energy distributions or in other words “informational variables” available to a perceptual system and the environmental properties causally responsible for that structure. This specificity is what is meant by information. Furthermore, this approach claims that “perception is direct”, meaning that perception is a direct process with no need for intermediate process transforming or supplementing the incoming data. In real life we are not interacting with single objects, rather we interact with an entire event. The patterns of energy (in case of vision it is light (optical array)) created by events in the environment, change as we move through space and time. Hence, these energy patterns or flow fields show variants and invariants over these changes and these are information for the perceptual systems.

In realms of the ecological approach, the process of detecting information is carried out by a functional system distributed throughout an organism (Gibson 1979). Adjustments of peripheral organs, such as turning the eyes and head, play as significant a role in direct perception as the activity of the brain and the nervous system (Richardson et al. 2008). In this way the perception is coupled to action and not segregated.  Awareness of the environment is based on the adjustment of the organism’s entire perceptual system to the available information. This adjustment includes a range of processes, all of which may be described as the simultaneous extraction of persisting and changing properties of stimulation, invariants despite disturbances of the array of information (Gibson, 1979). The invariant is a higher-order property of the stimulus array that exists whether the organism knows it or not, and whether the organism attends to it or not. Observers can perceive themselves, their environments, and the changing relationship between themselves and their surroundings. This adjustment of the perceptual system requires component processes such as delimiting the range of variables in stimulation, establishing co-variations of information across different perceptual systems, distinguishing information specifying the self from information specifying the environment, and extracting information of the affordances of the objects, places, events and other people in one’s habitat (Gibson, 1966).

Affordances

            Gibson’s (1979) introduced the concept of “affordance”. Affordances can be described as the properties of the environment which are meaningful to the observer, that can be perceived from the pattern of sensory stimulation alone, without needing prior experience. Affordances are specified by higher order ambient energy patterns which can be confined to a single type of ambient energy pattern (light, sound, gravito-inertial energy array etc.) or may span through multiple types of energy (Gibson, 1979). According to Fajen, Riley & Turvey (2008), there are five distinguished features of affordances:

• Affordances are ontological properties of organism- environment system.
• Affordances are not intrinsic properties of environment rather these are relational properties defined with respect to the perceiver.
• Affordances capture the reciprocity of perception and action, it implies that the perception of the environment is in terms of the possible actions that the perceiver can produce, and, at the same time, affordances are perceived through active exploration of the environment.
• Affordances allow prospective control of action, implies that affordances allow perceiver to make behavioral adjustments to the future event, lawfully predicted from the current state.
• Affordances are meaningful so that instead of perceiving the environment in neutral terms as extent, mass, and so forth, affordances are perceiver-relevant properties as climbability, catchability, graspability etc.

            Fajen et al. (2008) further distinguished body- scaled and action scaled affordances. Action-scaled affordances referred to as the possibilities of action made possible by the dynamic action- capabilities of the perceiver.  Studies references braking control (Lee 1976), catching fly balls (Fajen, Diaz, & Cramer, 2011; Oudejans, Michaels, Bakker, & Dolné, 1996), and walking through sliding doors (Fajen & Matthis, 2011; Fajen et al., 2011).  Body-scaled affordances refer to properties that are scaled to anthropometric dimensions. Research concerning this type of affordance has addressed stair climbing (Konczak et al., 1992; Mark, 1987; Warren, 1984; Wraga, 1999), prehension (Newell et al., 1993; Newell et al., 1989; Van der Kamp et al., 1998), sitting (Mark, 1987), passing under a barrier (Van der Meer, 1997), fitting the hand through an aperture (Ishak et al., 2008), and walking through apertures tightly scaled to the inter-shoulder dimension (Warren & Whang, 1987).

Chapter 3. Sensory Substitution Devices: Concepts, Examples, Active Perception, and Limitations

“Vision is indispensable to provide information about body-space interaction, movement precision and orientation and motor action timing” (Carretti et al., 2023).Visually impaired individuals have limited access to the global array, or we can also ask to what extent it may affect their ability to detect their affordances (Huges, 2001).

3.1 What Are Sensory Substitution Devices?

Sensory substitution devices (SSDs) are a group of noninvasive devices that enable the detection of information about the surroundings through one sensory modality and convey it to the user through another (Chebat et al., 2018). With the development of the first famous SSDs by Bach-y-Rita, numerous such devices were introduced (Ptito et al., 2021). Most commonly such devices are substituting for vision using touch or audition (Bach-y-Rita et al., 1969; Bach-y-Rita & Kercel, 2003; Meijer, 1992; Ward & Meijer, 2010). SSDs thereby provide individuals with a visual impairment to access information about their environment. Studies have shown performing certain tasks with SSDs is possible such as grasping objects (de Paz et al., 2023), wayfinding (Kilian et al., 2022), walking towards targets (Lobo et al., 2018), and obstacle avoidance (Lobo et al., 2019).

According to popular opinion, the conceptual underpinning of SSDs is based on the plasticity of the brain. According to this idea that even if the visual system is compromised, the brain can process information delivered through other sensory channels (Bach-y-Rita, 1972; Ptito et al., 2021). The ecological theories of perception, however, also stress the role of active perception and the importance of direct interaction with the environment for the detection of affordances and sensorimotor contingencies (Gibson, 1979; Varela et al., 1991; Kolarik et al., 2014; Travieso et al., 2015).

3.2 Classic and Modern Examples of SSDs

Classic Examples:

• Braille: The earliest and most successful SSD for reading, Braille allows individuals to read and write via touch, representing a tactile substitute for visual print (Ptito et al., 2021).
• White Cane: Used for mobility, the cane extends tactile perception to detect obstacles on the ground, but does not protect the upper body or head (Rizzolatti et al., 1997; Manduchi & Kurniawan, 2011).

Electronic SSDs:

• TVSS (Tactile Vision Sensory Substitution System): Developed by Bach-y-Rita et al. (1969), this device uses a camera and a tactile stimulator array to translate visual information into tactile stimuli on the skin, first applied to the back, then to the fingers and tongue.
• vOICe: This auditory SSD translates visual images into complex soundscapes by mapping pixel brightness to volume, pitch, and timing, enabling users to “hear” visual scenes (Meijer, 1992; Ward & Meijer, 2010).
• Tongue Display Unit (TDU): Converts visual images to electrotactile patterns on the tongue, taking advantage of the tongue’s high sensitivity and dense cortical representation (Bach-y-Rita et al., 1998; Sampaio et al., 2001; Ptito et al., 2012).
• Enactive Torch: A more intuitive device that provides vibrotactile feedback proportional to the distance from obstacles, supporting direct sensorimotor interaction (Froese et al., 2011).

Electronic Travel Aids (ETAs):

• ETAs comprise ultrasonic or laser-based sensors (such as the Miniguide, K-Sonar, and UltraCane) to detect obstacles. These devices relay distance information through tactile or auditory cues, often as vibration or changes in pitch (Kolarik et al., 2014; Ptito et al., 2021).

3.3 Role of Active Perception

Current research is focusing on the fact that SSDs require active perception. This implies examining how perception is not simply the passive reception of sensory inputs. Gibson’s ecological approach argue that perception involves active exploration of the environment, (Gibson, 1979; Michaels & Carello, 1981).

Active perception is crucial for SSD use: This approach emphasize that users must move and explore their environment with the device—whether feeling vibrations on the skin, listening to auditory cues, or manipulating a cane in order to detect the useful information. For example, users of the TVSS or vOICe must learn to scan their environment to interpret tactile or auditory representations of space, and Braille reading involves active finger movements.

Studies have shown that active exploration enables the detection of affordances—possibilities for action offered by the environment—regardless of the specific sensory modality (Mark, 1987; Warren, 1984; Kolarik et al., 2014; Travieso et al., 2015). Moreover, active sensorimotor engagement is essential for perceptual learning, adaptation, and the functional use of SSDs in real-world tasks.

3.4 Limitations and Pitfalls of Current SSD Designs

Despite substantial progress, current SSDs face several limitations:

1. Increased Cognitive Load: Many SSDs (e.g., TVSS, vOICe) attempt to translate the full richness of vision (color, shape, spatial relations) into alternative modalities, resulting in complex, non-intuitive signals that are difficult to interpret without extensive training. Whereas, also with a few simpler one, users find under or overload of information with SSDs problematic (Bilal et al., 2025b). This can lead to cognitive overload and reduced usability in daily life (Auvray et al., 2007; Guarniero, 1974, 1977; Jicol et al., 2020).
2. Training Demands: Successful use of many SSDs, especially those with arbitrary or non-intuitive coding schemes, require prolonged training. Although the possibilities SSDs offer, enhance with practice (Bilal et al., 2025a) but users must learn to map new forms of stimulation to meaningful environmental features—essentially acquiring a new “language” for perception (Lenay et al., 2003; Kushnir & Müller, 2020; Auvray & Myin, 2009)
3. Limited Practical Adoption: Although users can achieve impressive performance in laboratory settings, few SSDs have been widely adopted for daily life (Elli et al., 2014). Most visually impaired individuals continue to rely on the white cane, which is although simple, is limited to ground-level obstacle detection and does not support richer spatial understanding or upper body protection (Bilal et al., 2025b).
4. Design Trade-offs: There is a continuing debate about whether SSDs should aim for general-purpose use (replacing all functions of vision) or be highly specific (targeting a single task, such as obstacle detection). Recent research favors the SSDs that are task specific and provide minimal information crucial to the specific action (Loomis et al., 2012; Maidenbaum et al., 2014).
5. Sensory Modality Constraints: There is no consensus on whether touch or audition is superior for substituting vision. There are pros and cons of each modality. For example, audition provides high temporal resolution but is less private, whereas tactile feedback can be more discreet but is limited in bandwidth and can interfere with the use of hands for other tasks (Auvray & Myin, 2009).
6. Social and Practical Barriers: Individuals with visual impairments believe that devices which are obtrusive, stigmatizing, or require conspicuous equipment (e.g., large headsets, electrode arrays) are not very useful with regard to the social aspects. They believe such devices make it hard for them to integrate into society with minimal difference or attention drawn to their impairment. Thereby, users would like to favor devices which fulfill their basic need of sociability, along with autonomy and control, however, they don’t feel that it is fulfilled by current devices (Bilal et al., 2025b)

Chapter 4: Why do we need an ecological approach while developing navigation aids for visually impaired individuals?

• SSDs designed based on affordance allow us to experiment and understand the issues that are relevant to the control of action (de Paz et al., 2019).
• Environmental properties are not associated with any particular sensory modality, rather affordances can be perceived as by means of different perceptual systems.
• Studies supporting the perception of body scaled affordances with SSDs (de Paz et al., 2019, 2023; Lobo et al., 2014; Travieso et al., 2015)
• Affordances are perceived through exploratory actions (Gibson, 1979). And studies have shown the affordance-based approach naturally lead to enhanced exploratory behavior with SSDs (Bilal et al., 2025a, de Paz et al., 2019; Lobo et al., 2014) .
• Travieso et al. (2015) said: “…the majority of SSDs allow an active control of the sensor component, and the effector component is lawfully coupled to the sensor component, according to such criteria the majority of SSDs may produce distal attribution.” They further emphasized that some devices provide a direct substitution of sensory information (true substitution), while others provide aids for cognitive processing (cognitive aids). True substitution devices mimic the natural laws governing sensory perception, like how light behaves in optics. Cognitive aids, on the other hand, might use arbitrary codes that don’t necessarily follow these natural laws.
• Distance to vibration-based SSDs allow real time information-based control of action (de Paz et al., 2023; Kolarik et al., 2017; Lobo et al., 2019). Cancar et al. (2013) suggested that vibrotactile flow can be generated by distance-based activation of the actuators and this stimulation on the skin allows users to perceive time-to-contact of approaching ball in both simulated and real environments.

So far, we established the argument that the ecological approach offers a more suitable framework for designing navigation aids by emphasizing the importance of directly perceiving environmental information and exploiting the affordances present in the surroundings for guiding action. Future developments in this field should develop a conceptual framework containing criteria based on ecological psychology approach for designing aids for rehabilitating blinds. In addition, we should describe basic device components and features to develop useful navigating aids.

Table 1: Ecological Framework Criteria

Criterion Name	Description
Affordance	Design aids that clearly indicate potential actions or interactions, enabling the blind individual to perceive what actions are possible in their environment.
Perception-Action Coupling	Device must ensure coupling between the sensory information provided by the aids and the actions required for navigation and grasping, facilitating seamless interaction.
Adaptability	Create aids that can adapt to different environments and tasks, allowing for flexible usage in various contexts encountered in daily life. For example, devices aiding navigation should be able to detect static and dynamic obstacles.
Sensory Substitution	Present spatial information through alternative sensory channels such as sound or touch.
Information detection/ Real-time Feedback Mechanisms	Incorporate feedback mechanisms that provide information (real-time) about the consequences of actions taken, aiding in the refinement of motor skills and spatial understanding.
Perceptual Learning	Focus should not be only on designing a device but also determine methods to allow better perceptual learning through repeated interaction with the aids, enabling blind individuals to develop more refined spatial awareness and motor skills over time.
Minimal Cognitive Load	Design aids that minimize cognitive load, ensuring that the blind individual can focus attention on navigating their environment and performing tasks rather than on interpreting complex information.

We would like to point the readers to Elmannai et al. (2017) who presented a detailed overview of currently used sensor technologies in designing navigation aids for blinds. They also compared various technical aspects, shortcomings, and techniques for detection, recognition, and localization. We recommend readers to consult this text.

Finally, we did not explain in this toolbox other general considerations of developing viable rehabilitation devices for helping visually impaired individuals such as portability, cosmesis, ease of use, regulatory compliances, economic factors etc. as it was out of scope of this toolbox. However, we emphasize here that the non-inclusion of potential end- users while researching and developing rehabilitation aids, left the users with only two options accepting or rejecting of devised solution but not to embed their experiences into the development or improvement of device with respect to design and application (Rogers & McClelland, 2004). Understanding the user’s need should be one of foremost objectives while developing the rehabilitation aids for them (Bilal et al., 2025b).

References

Araujo, D., & Davids, K. (2009). Ecological approaches to cognition and action in sport and exercise: Ask not only what you do, but where you do it. International Journal of Sport Psychology, 40(1), 5.

Auvray, M., & Myin, E. (2009). Perception with compensatory devices. From sensory substitution to sensorimotor extension. Cognitive Science, 33, 1036-1058.

Beall, A. C., & Loomis, J. M. (1997). Optic flow and visual analysis of the base-to-final turn. The International Journal of Aviation Psychology, 7(3), 201-223.

Bourne, R. R. A., Flaxman, S. R., Braithwaite, T., Cicinelli, M. V., Das, A., Jonas, J. B., Keeffe, J., Kempen, J., Leasher, J., Limburg, H., Naidoo, K., Pesudovs, K., Resnikoff, S., Silvester, A., Stevens, G. A., Tahhan, N., Wong, T., Taylor, H. R., Ackland, P., … Zheng, Y. (2017). Magnitude, temporal trends, and projections of the global prevalence of blindness and distance and near vision impairment: a systematic review and meta-analysis. The Lancet Global Health, 5(9), e888–e897. https://doi.org/10.1016/S2214-109X(17)30293-0

Carretti, G., Manetti, M., & Marini, M. (2023). Physical activity and sport practice to improve balance control of visually impaired individuals: a narrative review with future perspectives. Frontiers in Sports and Active Living, 5, 1260942. https://doi.org/10.3389/FSPOR.2023.1260942/BIBTEX

de Paz, C., Ibáñez-Gijón, J., Travieso, D., & Jacobs, D. M. (2023). Grasping objects with a sensory substitution glove. International Journal of Human Computer Studies, 170. https://doi.org/10.1016/j.ijhcs.2022.102963

de Paz, C., Travieso, D., Ibáñez-Gijón, J., Bravo, M., Lobo, L., & Jacobs, D. M. (2019). Sensory substitution: The affordance of passability, body-scaled perception, and exploratory movements. PLoS ONE, 14(3). https://doi.org/10.1371/JOURNAL.PONE.0213342

Fodor, J. A., & Py Lyshyn, W. (1981). Discussion How direct is visual perception?: Some reflections on Gibson’s “Ecological Approachm*. In Cognition (Vol. 9).

Gibson, J. J. (1950). Perception of the visual world. Houghton Mifflin.

Gibson, J. J. (1959). Perception as a function of stimulation. In S. Koch (Ed.), Psychology: A study of a science (Vol. I, pp. 456–473). New York, NY: McGraw-Hill.

Gibson, J. J. (1979). The ecological approach to visual perception.

Goldschmidt, M. (2017). Orientation and mobility training to people with visual impairments. In Mobility of Visually Impaired People: Fundamentals and ICT Assistive Technologies (pp. 237–261). Springer International Publishing. https://doi.org/10.1007/978-3-319-54446-5_8

Elmannai, W., & Elleithy, K. (2017). Sensor-based assistive devices for visually impaired people: Current status, challenges, and future directions. Sensors, 17(3), 565.

Huges B (2001). Specification, information, and the loss of receptor systems. Behavioral and Brain Sciences,24, 223.

Lobo, L., Travieso, D., Barrientos, A., & Jacobs, D. M. (2014a). Stepping on obstacles with a sensory substitution device on the lower leg: Practice without vision is more beneficial than practice with vision. PLoS ONE, 9(6). https://doi.org/10.1371/JOURNAL.PONE.0098801

Lobo, L., Travieso, D., Barrientos, A., & Jacobs, D. M. (2014b). Stepping on obstacles with a sensory substitution device on the lower leg: Practice without vision is more beneficial than practice with vision. PLoS ONE, 9(6). https://doi.org/10.1371/JOURNAL.PONE.0098801

Michaels, C. F., & Carello, C. (1981). Direct perception. Prentice-Hall.

Richardson, M. J., Shockley, K., Fajen, B. R., Riley, M. A., & Turvey, M. T. (2008). Ecological psychology: Six principles for an embodied–embedded approach to behavior. In Handbook of cognitive science (pp. 159-187). El

Rogers, T. T., & McClelland, J. L. (2004). Semantic Cognition: A Parallel Distributed Processing Approach. The MIT Press.

Travieso, D., Gómez-Jordana, L., Díaz, A., Lobo, L., & Jacobs, D. M. (2015). Body-scale affordances in sensory substitution. Consciousness and Cognition, 38, 130–138. https://doi.org/10.1016/J.CONCOG.2015.10.009

Turvey, M. T., & Shaw, R. E. (1999). Ecological foundations of cognition. I: Symmetry and specificity of animal-environment systems. Journal of Consciousness Studies, 6(11-12), 95-110.

von Eckardt , B. ( 1993 ). What is Cognitive Science? Cambridge, MA : The MIT Press .

Warren, W. H. (2021). Information Is Where You Find It: Perception as an Ecologically Well-Posed Problem Introduction: Perception as an Ill-Posed Problem. I-Perception, 12(2), 1–24. https://doi.org/10.1177/20416695211000366