| abstract |
| Background Cross-modality of auditory and visual senses is the interaction between these two different sensory channels. Today media environments are evolving towards the reinforcement of multi-modality. In various fields using media, the interaction between hearing and sight is an element that strengthens communication effect and user’s immersion level. Therefore, it is needed to study the way to visualize sound for the coming digital media era, by analyzing the framework of interactions between two different senses: auditory and visual.
Methods Firstly, we discussed about mechanisms of synesthesia and cross-modal integration. Then the cross-modal patterns of auditory and visual senses were analyzed by the cross-case analysis methodology. And the properties of cross-modality of auditory and visual senses was derived. Also, data applied with properties of audio-visual cross-modal integration was used to illustrate matching framework in variety of ways.
Result Consistent association between pitch and brightness was found in all studies. Chroma had the biggest relevance to overtone and was also related to pitch, volume, and tempo. We could not define exact match of color sets for synesthesia and cross-modal integration. However, there was a tendency of showing similar color responses through shared emotional experiences from musical and tactile experiences.
Conclusion Until now, most of the researches for audio-visual synesthesia were conducted with limited conditions and too small number of samples to generalize the result. In this study, however, we were able to derive the pattern of synesthesia and cross-modal integration by combining studies performed under variety of conditions and methods. These studies were conducted in different cultural areas and times. Also, based on gathered data, we could study the way to visualize sound from media. Various frameworks for mapping auditory data with visual data were used. |
|
|
| Key Words |
| 소리 시각화, 공감각, 교차감각, Sound Visualization, Synesthesia, Cross-Modality |
|
|
|
 |
|
Full Text