International Conference on Auditory Display (ICAD)

Series

International Conference on Auditory Display (ICAD)

Permanent Link

https://hdl.handle.net/1853/70916

Series Type

Event Series

Associated Organization(s)

Organizational Unit

Sonification Lab

Full item page

Publication Search Results

Now showing 1 - 10 of 35

FACIAL BEHAVIOR SONIFICATION WITH THE INTERACTIVE SONIFICATION FRAMEWORK PANSON

(Georgia Institute of Technology, 2023-06) Nalli, Michele ; Johnson, David ; Hermann, Thomas

Facial behavior occupies a central role in social interaction. Its auditory representation is useful for various applications such as for supporting the visually impaired, for actors to train emotional expression, and for supporting annotation of multi-modal behavioral corpora. In this paper we present a prototype system for interactive sonification of facial behavior that works both in realtime mode, using a webcam, and offline mode, analyzing a video file. The system is based on python and Jupyter notebooks, and relies on the python module sc3nb for sonification-related functionalities. Facial feature extraction is realized using OpenFace 2.0. Designing the system led to the development of a framework of reusable components to develop interactive sonification applications, called Panson, which can be used to easily design and adapt sonifications for different use cases. We present the main concepts behind the facial behavior sonification system and the Panson framework. Furthermore, we introduce and discuss novel sonifications developed using Panson, and demonstrate them with a set of sonified videos. The sonifications and Panson are Open Source reproducible research available on GitHub.
SONECULES: A PYTHON SONIFICATION ARCHITECTURE

(Georgia Institute of Technology, 2023-06) Reinsch, Dennis ; Hermann, Thomas

This paper introduces sonecules, a flexible, extensible, enduser friendly and open-source Python sonification toolkit to bring 'sonification to the masses'. The package comes with a basic set of what we define as sonecules which are sonification designs tailored for a given class of data, a selected internal logic for sonification and offering a set of functions to interact with data and sonification controls. This is a design-once-use-many approach as each sonecule can be reused on similarly structured data. The primary goal of sonecules is to enable novice users to rapidly get their data audible – by scaffolding their first steps into auditory display. All sonecules offer a description for the user as well as controls which can be adjusted easily and interactively to the selected data. Users are supported to get started as fast as possible using different sonification designs and they can even mix and match sonecules to create more complex aggregated sonecules. Advanced users are enabled to extend/modify any sonification design and thereby create new sonecules. The sonecules Python package is provided as open-source software, which enables others to contribute their own sonification designs as a sonecule – thus it seeds a growing/growable library of well-documented and easy-to-reuse sonifications designs. Sonecules is implemented in Python using mesonic as the sonification framework, which provides the path to renderingplatform agnostic sonifications.
AltAR/table: A Platform for Plausible Auditory Augmentation

(Georgia Institute of Technology, 2022-06) Weger, Marian ; Hermann, Thomas ; Höldrich, Robert

Auditory feedback from everyday interactions can be augmented to project digital information in the physical world. For that purpose, auditory augmentation modulates irrelevant aspects of already existing sounds while at the same time preserving relevant ones. A strategy for maintaining a certain level of plausibility is to metaphorically modulate the physical object itself. By mapping information to physical parameters instead of arbitrary sound parameters, it is assumed that even untrained users can draw on prior knowledge. Here we present AltAR/table, a hard- and software platform for plausible auditory augmentation of flat surfaces. It renders accurate augmentations of rectangular plates by capturing the structure-borne sound, feeding it through a physical sound model, and playing it back through the same object in real time. The implementation solves basic problems of equalization, active feedback control, spatialization, hand tracking, and low-latency signal processing. AltAR/table provides the technical foundations of object-centered auditory augmentations, for embedding sonifications into everyday objects such as tables, walls, or floors.
Real-time auditory contrast enhancement

(Georgia Institute of Technology, 2019-06) Weger, Marian ; Hermann, Thomas ; Höldrich, Robert

Every day, we rely on the information that is encoded in the auditory feedback of our physical interactions. With the goal to perceptually enhance those sound characteristics that are relevant to us - especially within professional practices such as percussion and auscultation - we introduce the method of real-time Auditory Contrast Enhancement (ACE). It is derived from algorithms for speech enhancement as well as from the remarkable sound processing mechanisms of our ears. ACE is achieved by individual sharpening of spectral and temporal structures contained in a sound while maintaining its natural gestalt. With regard to the targeted real-time applications, the proposed method is designed for low latency. As the discussed examples illustrate, it is able to significantly enhance spectral and temporal contrast.
Data-driven auditory contrast enhancement for everyday sounds and sonifications

(Georgia Institute of Technology, 2019-06) Hermann, Thomas ; Weger, Marian

We introduce Auditory Contrast Enhancement (ACE) as a technique to enhance sounds at hand of a given collection of sound or sonification examples that belong to different classes, such as sounds of machines with and without a certain malfunction, or medical data sonifications for different pathologies/conditions. A frequent use case in inductive data mining is the discovery of patterns in which such groups can be discerned, to guide subsequent paths for modelling and feature extraction. ACE provides researchers with a set of methods to render focussed auditory perspectives that accentuate inter-group differences and in turn also enhance the intra-group similarity, i.e. it warps sounds so that our human built-in metrics for assessing differences between sounds is better aligned to systematic differences between sounds belonging to different classes. We unfold and detail the concept along three different lines: temporal, spectral and spectrotemporal auditory contrast enhancement and we demonstrate their performance at hand of given sound and sonification collections.
Wave Space Sonification

(Georgia Institute of Technology, 2018-06) Hermann, Thomas

This paper introducesWave Space Sonification (WSS), a novel class of sonification techniques for time- (or space-) indexed data. WSS doesn’t fall into the classes of Audification, Parameter- Mapping Sonification or Model-based Sonification and thus constitutes a novel class of sonification techniques. It realizes a different link between data and their auditory representation, by scanning a scalar field – defined as wave space – along a data-driven trajectory. This allows both the highly controlled definition of the auditory representation for any area of interest, as well as subtle yet acoustically complex sound variations as the overall pattern changes. To illustrate Wave Space Sonification (WSS), we introduce three different WSS instances, (i) the Static Canonical WSS, (ii) Data-driven Localized WSS and (iii), Granular Wave Space Sonification (GWSS), and we demonstrate the different methods with sonification examples from various data domains. We discuss the technique and its relation to other sonification approaches and finally outline productive application areas.
CardioSounds: A portable system to sonify ECG rhythm disturbances in real-time

(Georgia Institute of Technology, 2018-06) Aldana Blanco, Andrea Lorena ; Grautoff, Steffen ; Hermann, Thomas

CardioSounds is a portable system that allows users to measure and sonify their electrocardiogram signal in real-time. The ECG signal is acquired using the hardware platform BITalino and subsequently analyzed and sonified using a Raspberry Pi. Users can control basic features from the system (start recording, stop recording) using their smartphone. The system is meant to be used for diagnostic and monitoring of cardiac pathologies, providing users with the possibility to monitor a signal without occupying their visual attention. In this paper, we introduce a novel method, anticipatory mapping, to sonify rhythm disturbances such as Atrial Fibrillation, Atrial flutter and Ventricular Fibrillation. Anticipatory mapping enhances perception of rhythmic details without disrupting the direct perception of the actual heart beat rhythm. We test the method on selected pathological data involving three of the most known rhythm disturbances. A preliminary perception test to assess aesthetics of the sonifications and its possible use in medical scenarios shows that the anticipatory mapping method is regarded as informative discerning healthy and pathological states, however there is no agreement about a preferred sonification type.
Augmentation of an institute’s kitchen: An ambient auditory display of electric power consumption

(Georgia Institute of Technology, 2018-06) Groß-Vogt, Katharina ; Weger, Marian ; Höldrich, Robert ; Hermann, Thomas ; Bovermann, Till ; Reichmann, Stefan

Efficient feedback on energy consumption is regarded as one step towards a more sustainable lifestyle. Sonification is very apt to convey such information continuously in an ambient and effective way. This paper presents a pilot system for sonifying the electric power consumption of an institute’s kitchen. The reverberation of the kitchen is changed depending on the actual consumption and its difference to a weekly baseline. If the actual consumption is low, it is mapped to a plausible kitchen reverberation. If it is high compared to the baseline, the reverberation becomes unnatural. Evaluating the system gave insights on perceptibility and acceptance of auditory augmentation in a semi-home context.
Parallel Computing of Particle Trajectory Sonification to Enable Real-Time Interactivity

(Georgia Institute of Technology, 2017-06) Yang, Jiajun ; Hermann, Thomas

In this paper, we revisit, explore and extend the Particle Trajectory Sonification (PTS) model, which supports cluster analysis of high-dimensional data by probing a model space with virtual particles which are ‘gravitationally’ attracted to a mode of the dataset’s potential function. The particles’ kinetic energy progression of as function of time adds directly to a signal which constitutes the sonification. The exponential increase in computation power since its conception in 1999 enables now for the first time to investigate real-time interactivity in such complex interweaved dynamic sonification models. We speeded up the computation of the PTS model with (i) data optimization via vector quantization, and (ii) parallel computing via OpenCL. We investigated the performance of sonifying high-dimensional complex data under different approaches. The results show a substantial increase in speed when applying vector quantization and parallelism with CPU. GPU parallelism provided a substantial speedup for very large number of particles comparing to using CPU but did not show enough benefit for a low number of particles due to copying overhead. A hybrid OpenCL implementation is presented to maximize the benefits of both worlds.
Optimizing aesthetics and precision in sonification for peripheral process-monitoring

(Georgia Institute of Technology, 2015-07) Hermann, Thomas ; Hildebrandt, Tobias ; Langeslag, Patrick ; Rinderle-Ma, Stefanie

We have developed the SoProMon process monitoring system to evaluate how real-time sonifications can increase awareness of process states and to support the detection and resolving of critical process situations. Our initial design conveys analogue information as process-data-driven soundscape that users can blend out in favor of a primary task, however the sonification attracts the user's attention even before things become critical. As result of a first user study we gained and present here insights into usability and acceptance of the sounds. Although effective, the aesthetic qualities were not rated highly. This motivated us to create a new design that sacrifices some functional aspects to emphasize long-term use compatibility. We present and compare the new designs and discuss our experiences in creating pleasant sonifications for this application area.