Digital Preservation and Access to Aural Heritage Via A Scalable, Extensible Method

FAIN: PR-263931-19

RIT (Rochester, NY 14623-5698)
Sungyoung Kim (Project Director: June 2018 to present)

The development of capture protocols, standards, and tutorials for long-term preservation and virtual representations of aural heritage.

Aural heritage preservation documents and recreates the auditory experience of culturally important places, enabling virtual interaction through physics-based reconstructions. A form of “intangible” cultural heritage, aural heritage is captured via spatial acoustics techniques, creating digital audio data for auralizations (reconstructions for listening). This project will 1) codify a protocol for the capture, verification, and auralization of aural heritage, demonstrated in case-study application on three culturally distinct sites; 2) create extensibility pathways for the widespread adoption of this protocol, including workshops, web-based tutorials, and other freely disseminated resources that enable non-acoustical specialists to apply the method to a diversity of sites. Case study demonstrations of the method will serve as models for site constituencies, Humanities researchers, and other cultural heritage practitioners, while providing a digital archive of endangered aural heritage.

Associated Products

AuralHeritage.org – Project Website (Web Resource)
Title: AuralHeritage.org – Project Website
Author: Sungyoung Kim
Author: Doyuen Ko
Author: Miriam Kolar
Abstract: In the first year of the project (2019), we developed and launched our project website (auralheritage.org), a key public interface for project dissemination and auralization demonstrations.
Year: 2019
Primary URL: http://auralheritage.org
Primary URL Description: The project website serves as a key public interface for project dissemination and publication of auralization demonstrations (to be produced during the second and third years of the project), which also provides project information, including objectives, references to NEH support, an introduction to aural heritage research, and press discussions about the project.

Sound Hounds: How One Research Team Is Helping Preserve the Acoustics of Historic Places (Web Resource)
Title: Sound Hounds: How One Research Team Is Helping Preserve the Acoustics of Historic Places
Author: Nicholas Som
Abstract: The National Trust for Historic Preservation profiled the NEH-Funded Research project case-study aural heritage research conducted in May 2019 at Columbia Studio A on Nashville's Historic Music Row. The research project, co-organized by Doyuen Ko (an associate professor of audio engineering technology at Belmont University), Sungyoung Kim (an associate professor at the Rochester Institute of Technology) and Miriam Kolar (a visiting scholar from Amherst College), will produce public demonstrations of aural heritage and provide tools for other researchers to do the same.
Year: 2019
Primary URL: https://savingplaces.org/stories/sound-hounds-how-one-research-team-is-helping-preserve-the-acoustics-of-historic-places#.XoLBim57mFw
Primary URL Description: Project profile article on the website of the National Trust for Historic Preservation.

Team receives grant to recreate the ‘sound signature’ of cultural heritage sites (Web Resource)
Title: Team receives grant to recreate the ‘sound signature’ of cultural heritage sites
Author: Michelle Cometa
Abstract: Advanced audio technologies being developed are helping to preserve the unique sounds of historic sites from recording studios in Nashville, Tenn., to a pre-Columbian archeological site in Peru. Sungyoung Kim, an associate professor of audio engineering technology at Rochester Institute of Technology, is leading a team of researchers to develop a set of tools using advanced augmented and virtual reality technology to preserve and replicate the acoustics of historical venues. The project would bring attention to the overlooked work in preserving aural heritage.
Year: 2019
Primary URL: https://www.rit.edu/news/team-receives-grant-recreate-sound-signature-cultural-heritage-sites
Primary URL Description: RIT News Website coverage of the project announcement.

Belmont Faculty to Research Digital Preservation of Music Row Studios’ Aural Heritage (Web Resource)
Title: Belmont Faculty to Research Digital Preservation of Music Row Studios’ Aural Heritage
Author: April Hefner
Abstract: Dr. Doyuen Ko, colleagues receive $350,000 National Endowment of Humanities grant to replicate acoustics of historic structures. Ko, along with co-principal investigator Dr. Sungyoung Kim (RIT) and aural heritage consultant Dr. Miriam Kolar, research how building acoustics influence human experience, particularly in religious, performance and other public gathering spaces. The researchers will develop, test and share their aural heritage process via three case studies, which represent culturally, architecturally and temporally distinct examples of endangered aural heritage.
Year: 2019
Primary URL: https://news.belmont.edu/belmont-faculty-to-research-digital-preservation-of-music-row-studios-aural-heritage/
Primary URL Description: Belmont University News & Media Website coverage of the project announcement.

Preserving Human Perspectives in Cultural Heritage Acoustics: Distance Cues and Proxemics in Aural Heritage Fieldwork (Article)
Title: Preserving Human Perspectives in Cultural Heritage Acoustics: Distance Cues and Proxemics in Aural Heritage Fieldwork
Author: Sungyoung Kim
Author: Miriam A. Kolar
Author: Doyuen Ko
Abstract: We examine the praxis implications of our working definition of aural heritage: spatial acoustics as physically experienced by humans in cultural contexts; aligned with the aims of anthropological archaeology (the study of human life from materials). Here we report on human-centered acoustical data collection strategies from our project “Digital Preservation and Access to Aural Heritage via a Scalable, Extensible Method,” supported by the National Endowment for the Humanities (NEH) in the USA. The documentation and accurate translation of human sensory perspectives is fundamental to the ecological validity of cultural heritage fieldwork and the preservation of heritage acoustics. Auditory distance cues, which enable and constrain sonic communication, relate to proxemics, contextualized understandings of distance relationships that are fundamental to human social interactions. We propose that source–receiver locations in aural heritage measurements should be selected to represent a comprehensive range of proxemics according to site-contextualized spatial-use scenarios, and we identify and compare acoustical metrics for auditory distance cues from acoustical fieldwork we conducted using this strategy in three contrasting case-study heritage sites. This conceptual shift from architectural acoustical sampling to aural heritage sampling prioritizes culturally and physically plausible human auditory/sound-sensing perspectives and relates them to spatial proxemics as scaled architecturally.
Year: 2021
Primary URL: https://doi.org/10.3390/acoustics3010012
Primary URL Description: Acoustics 2021, 3, 156–176.
Access Model: Open Access
Format: Journal
Periodical Title: Acoustics
Publisher: MDPI

Web Tutorial: BINAURAL AURALIZATION #1: Comparison of Two Case-Study Listening Perspectives (Web Resource)
Title: Web Tutorial: BINAURAL AURALIZATION #1: Comparison of Two Case-Study Listening Perspectives
Author: Doyuen Ko
Author: Miriam A. Kolar
Author: Sungyoung Kim
Abstract: Comparison of LISTENING PERSPECTIVE: Centralized Source with Close Listener (Proportional) in Music Row Recording Studio, Nashville, TN (left plan, below) vs. 19th-century Byzantine Revival Bank, Rochester, NY (right plan, below). This computational auralization is scaled for individual listening over headphones or earbuds. To create each auralization, a "dry", close-up recording of a mandolin (using one microphone, without acoustical context) was convolved with binaural/two-channel acoustical impulse responses as documented in the measurement session photographs below each site plan. Convolution is the name of a digital signal process that can be used to virtually locate one sound in another place, by mathematically combining audio (the mandolin recording) with measured data (the impulse response) from that space. How would you describe the differences and similarities of the sonic environments presented in these auralizations? How do you think the acoustical differences affect social interactions in these spaces during their intended uses? These are some of the social and historical questions that can be explored and tested using aural heritage data and auralization interfaces, for both researchers and public audiences.
Year: 2020
Primary URL: http://auralheritage.org/auralization-CSAvsRSB.html
Primary URL Description: BINAURAL AURALIZATION #1: Comparison of Two Case-Study Listening Perspectives on the Project Website for Digital Preservation and Access to Aural Heritage Via A Scalable, Extensible Method

Preserving Aural Heritage (Film/TV/Video Broadcast or Recording)
Title: Preserving Aural Heritage
Writer: Michelle Cometa
Producer: Michelle Cometa
Abstract: Documentary produced by the RIT News Service.
Year: 2021
Primary URL: https://www.youtube.com/watch?v=xKhoNnrVMSU&t=1s
Primary URL Description: Researchers Sungyoung Kim, Associate Professor at Rochester Institute of Technology, Doyuen Ko, Associate Professor at Belmont University (Tenn.), and Miriam Kolar, Visiting Scholar at Amherst College (Mass.) worked to develop a set of tools using advanced augmented and virtual reality technology to preserve and replicate acoustics of cultural heritage site. Their work was part of a three-year national Endowment for the Humanities grant, an international project bringing attention to the overlooked work in preserving aural heritage.
Format: Video
Format: Web

A "human-centered" approach to aural heritage preservation and access (Article)
Title: A "human-centered" approach to aural heritage preservation and access
Author: Miriam A. Kolar
Author: Sungyoung Kim
Author: Doyuen Ko
Author: Xuan Lu
Abstract: Aural heritage research documents, reconstructs, and preserves the sonic interactivity of sites important to humans across time and around the world. Here, we discuss acoustical data acquisition per the definition of aural heritage we proposed: "spatial acoustics as physically experienced by humans in cultural contexts." Two factors support ecological validity (realism). First, the use of places change over time, so aural heritage documentation and reconstruction should be informed by contextual knowledge of both present-day situations as well as past scenarios suggested by historical records or archaeological materials. Second, and our focus here: aural heritage documentation requires acoustical measurements that represent humanly plausible perspectives on a soundfield, including measurements that document human-surface relationships. To ensure accuracy in the presentation of reconstructions of past acoustics to present-day humans, it is necessary to understand how spatial acoustical data translate across different audio rendering systems used for auralizations. Our "human-centered" approach to data collection addresses ecological validity in aural heritage preservation, and in concurrent research we have conducted perceptual evaluations of soundfields reconstructed from these data in multichannel listening rooms. Aural heritage data collection and auralization research require separate methodologies that intersect in the presentation of past acoustics to present-day listeners.
Year: 2022
Primary URL: https://auralheritage.org/publications/ICA2022Proceedings_KolarEtAl.pdf
Format: Journal
Format: Other
Periodical Title: Proceedings of the 24th International Conference on Acoustics (ICA 2022), Gyeongju, South Korea, October 2022
Publisher: 24th International Conference on Acoustics (ICA 2022), Gyeongju, South Korea, October 2022

Influence of the presence of congruent visual media on spatial auditory fidelity (Article)
Title: Influence of the presence of congruent visual media on spatial auditory fidelity
Author: Xuan Lu
Author: Sungyoung Kim
Author: Miriam Kolar
Author: Doyuen Ko
Abstract: In this research, we investigated the influence of congruent visual media on the recognition of spatial audio attributes in a multimodal virtual reality context. For the immersive auditory data, we recorded solo piano music using three multichannel microphone techniques: a 5-channel conventional surround microphone technique, and a 7-channel and a 9-channel immersive microphone array that both include height-channel microphones. In total, 38 normal-hearing participants were invited to the experiment and separated into two groups for presenting immersive audio with and without congruent visual media. The visual data was collected using a 360-degree camera and projected to a 250-degree wraparound screen for immersive visual rendering. Participants were asked to rate the timbral and spatial attributes for each of the three immersive auditory presentations (solo piano music via the three multichannel recording techniques), and to provide an overall preference rating by using a continuous quality scale (CQS). Experiment results show that the presentation of congruent visual media modulates reported perceptions of auditory attributes from simultaneously presented immersive audio. The experiment demonstrated that the presence of a realistic visual image congruent to the presented auditory material made listeners require higher spatial auditory fidelity in the immersive multimodal context. Consequently, this study's findings can be used to inform immersive and holistic audiovisual and interaction design.
Year: 2022
Primary URL: https://auralheritage.org/publications/ICA2022Proceedings_LuEtAl.pdf
Format: Journal
Format: Other
Periodical Title: Proceedings of the 24th International Conference on Acoustics (ICA 2022), Gyeongju, South Korea, October 2022
Publisher: 24th International Conference on Acoustics (ICA 2022), Gyeongju, South Korea, October 2022

Perceptual evaluation of a new, portable three-dimensional recording technique: “W-Ambisonics” (Article)
Title: Perceptual evaluation of a new, portable three-dimensional recording technique: “W-Ambisonics”
Author: Doyuen Ko
Author: Xuan Lu
Author: Sungyoung Kim
Author: Miriam Kolar
Abstract: In order to exploit strengths and avoid weaknesses of the First Order Ambisonics (FOA) microphone technique, we devised a new, portable 3D microphone recording technique, "W-Ambisonics." This new technique incorporates a spaced stereo cardioid microphone pair (for frontal information) with two FOA microphone arrays (for lateral, rear, and height information). In W-Ambisonics, two FOA microphones are spaced 17cm apart to capture and represent interaural cues precisely, with two cardioid microphones spaced 50cm apart, 50cm in front, which improves frontal directionality. Varying the elevation of the cardioid pair enhances spatialization techniques such as reproduction with height channels. The combination of these two microphone pairs enables the translation of recorded audio into various reproduction formats according to practical limitations in reproduction peripherals. The design focus of this technique was efficiency in the recording stage and scalability in the reproduction stage. We conducted three perceptual experiments whose results show that the W-Ambisonics method enables improved lateral localization, provides comparable sound quality to the conventional spaced array technique, and translates spacious yet precise sound images in listening evaluations of a binauralized headphone rendering. The W-Ambisonics microphone technique is practical, precise, and scalable across multiple reproduction scenarios, from binaural to multichannel systems.
Year: 2021
Primary URL: https://auralheritage.org/publications/LuEtAl2021_AES_W_Ambisonics.pdf
Format: Journal
Format: Other
Periodical Title: Audio Engineering Society Convention e-Brief, Presented at the 151st Convention 2021 October
Publisher: Audio Engineering Society (AES)

La Arqueoacústica experimental: Estudios para la preservación de “patrimonio auditivo” / Experimental Archaeoacoustics: Studies for Aural Heritage Preservation (Article)
Title: La Arqueoacústica experimental: Estudios para la preservación de “patrimonio auditivo” / Experimental Archaeoacoustics: Studies for Aural Heritage Preservation
Author: Miriam Kolar
Abstract: Archaeoacoustical studies based in experiments made in the archaeological sites of Chavín de Huántar and Huánuco Pampa, Perú make possible the preservation of information pertinent to understanding the sonic communication possible in these sites of great Andean importance. For archaeological interpretation, we relate the physical dimension of sound with its human perceptual affordances, to make new archaeological discoveries. The new methodologies to document, preserve, and reconstruct aural heritage are applied in diverse sites including work in historical sites in the United States and in decorated caves from the Upper Paleolithic in France –– all important to the "aural heritage" record, a term that we are developing in practice and publications.
Year: 2021
Primary URL: https://doi.org/10.36044/EC.V3.N1.5
Access Model: Open Access
Format: Journal
Periodical Title: Revista Ecos
Publisher: Universidad de la Republica Uruguay

Real-time spatial sound rendering system using LiDAR sensor for auditory augmented reality application (Conference Paper/Presentation)
Title: Real-time spatial sound rendering system using LiDAR sensor for auditory augmented reality application
Author: Sungyoung Kim
Author: Rai Sato
Author: Takumi Ito
Abstract: Conference Poster: Augmented reality Room Acoustic Estimator (ARAE) is the iOS platform to compute variable characteristics of room acoustics for AR and produce spatialized sounds instantaneously using Unity and audio middleware, Wwise. This platform needs LiDAR-equipped iPhone and AirPods with gyrosensor. Participants’ movements and transmission of sounds can be stored at an external storage. Any sound attributes are controllable from both inside and outside. This research platform provides a practical experimental environment that supports a variety of studies on cognition in AAR, with applications in aural heritage access.
Date: 10/24/2022
Primary URL: https://auralheritage.org/publications/ICA2022Poster_SatoEtAl.pdf
Conference Name: 24th International Conference on Acoustics (ICA 2022), Gyeongju, South Korea, October 2022

Reconstructing the Aural Heritage of the Historic Rochester Savings Bank (Public Lecture or Presentation)
Title: Reconstructing the Aural Heritage of the Historic Rochester Savings Bank
Abstract: In cultural heritage preservation, visual and architectural aspects of heritage sites are emphasized while little attention has been given to sensory and acoustic features. Because human experience is holistic, the contribution of auditory information is significant. In fact, many built environments have been specifically designed and used for conveying particular auditory information. For example, concert halls and recording studios are constructed to create pleasing acoustics for musicians and audiences. In such buildings, acoustics translate to auditory information that can uniquely identify a space. Moreover, visual information is dominant for 'informatic' experiences, while auditory information has been strongly associated with the 'emotional' aspects of those experiences, as well as with communication properties. In convincing and meaningful renderings of virtual experiences, therefore, successful delivery of auditory importation is not supplementary, but essential.
Author: Sungyoung Kim
Date: 04/23/2022
Location: Rochester, NY, USA
Primary URL: https://scholarworks.rit.edu/frameless/vol4/iss1/18/
Primary URL Description: This Open Access article describes research behind the public event, an Auralization Workshop-Demonstration at Imagine RIT creativity and innovation festival, April 2022.

Aural Heritage Case-Study Auralizations with 360-degree visuals (Web Resource)
Title: Aural Heritage Case-Study Auralizations with 360-degree visuals
Author: Doyuen Ko
Author: Sungyoung Kim
Author: Ben Jordan
Abstract: This website auralization interface demonstrates aural heritage data collection with 360-visual perspectives in the case-study sites Columbia Studio A and the Rochester Savings Bank.
Year: 2021
Primary URL: https://auralheritage.org/360auralizations/Omnitone.html

Preserving aural heritage, starting with historic recording studios in Nashville’s Music Row (Blog Post)
Title: Preserving aural heritage, starting with historic recording studios in Nashville’s Music Row
Author: Doyuen Ko, Miriam Kolar, Sungyoung Kim
Abstract: ‘Heritage acoustics’ and the preservation and presentation of historic acoustic environments is a growing field of research within cultural heritage. Discover the role that it plays in American music heritage - part of a EuropeanaTech focus.
Date: 12/20/2022
Primary URL: https://pro.europeana.eu/post/preserving-aural-heritage-starting-with-historic-recording-studios-in-nashville-s-music-row
Blog Title: EuropeanaTech is the community of experts, developers, and researchers from the R&D sector within the greater Europeana Network Association. We coordinate research and development endeavors that seek to improve the standing of European digital cultural heritage and facilitate knowledge across the sector at large.
Website: Europeana Pro

---