FEPlab

Reactive Message Passing for FEP agents

Scientific challenge

The main scientific challenge for this research topic is to realize robust and scalable FEP agents. This WP builds on research topic 1, where robust and resilient reactive message passing-based inference in generative models is developed. Here we further develop the methods and tools to facilitate real-time inference in practical FEP agents. An important issue is to endow practical models with both temporal depth (to support predicting the future) and hierarchical depth (to support abstract goal settings such as “end user must be happy”). Automated inference in these hierarchical dynamical systems leads to a series of computational scaling issues that will be addressed in this research. There is currently very little knowledge about these computational issues when they are dealt with in a reactive programming environment. An important thread that we wish to explore is the early stopping of passing messages when the FE seems to have converged. In a reactive programming environment, there is no preset message-passing schedule and since FE minimization is the only objective, there is no reason to keep passing messages after FE convergence. We conjecture that this mechanism may be a key factor in maintaining real-time inference in large systems.

Generative Probabilistic Models for Audio Processing

Scientific challenge

An FEP agent mainly comprises FE minimization in a Probabilistic Generative Model (PGM) for its sensory signals. In this project, the agent’s sensory signals are audio signals from the hearing device and performance appraisals by the end user. The main research challenge in this PhD research is to develop a PGM for the audio signals that are processed by the hearing device. In general, acoustic signals in the real world are composed of mixtures of source signals (such as a mixture of a speech signal from a conversation partner, plus background chatter and traffic noise). We plan to develop a PGM for acoustic mixtures that can be applied to hearing devices. Crucially, the model must be compact enough to support the real-time inference of the constituent sources, and yet accurate enough to be accepted by hearing device users. These constraints lead to an engineering design trade-off (accuracy vs. model complexity) that is the essential research issue in this PhD research project.

Thirty-plus years of generative modeling research for real-world acoustic signals have not yet resulted in a compact yet accurate PGM for acoustic mixtures. In this PhD research project, we take a novel approach by framing audio processing as a reactive message-passing process. In research direction 1, we develop an RMP toolbox that maintains inference performance even if model components fail randomly (reliability issue). In this view, in-situ model structure adaptation should then also maintain the performance of the inference process. In this PhD research, we take advantage of this property and aim to explore in-situ model structure adaptation. Since FE can be decomposed as “model complexity minus accuracy”, we hypothesize that minimization of FE by model structure adaptation should lead to a highly accurate yet compact acoustic model.

Generative Probabilistic Models for Covert User Interactions

Scientific challenge

An FEP agent mainly comprises FE minimization in a Probabilistic Generative Model (PGM) for its sensory signals. In this project, the agent’s sensory signals are both audio signals from the hearing device and performance appraisals by the end user. As discussed in research project 5, the user interface must be covert, very lightweight, and yet as informative as possible. In this research project, the challenge is to translate these ideas into a formal specification of a probabilistic user interaction model. A particular challenge in this research project will be to accommodate an adaptive user interaction protocol, where the protocol itself optimizes in situ under the pressure of FE minimization. Our goal will be to learn from user interactions which interaction protocol is most preferred (least cognitive burden, yet informative) from a parameterized model for user interactions. That is, in addition to adapting the parameters of the interaction model based on the modeling of covert and overt determinants of acceptance and trust (identified and tested in research project 5), this research project will create a model that also incorporates parameters of characteristics of the interaction itself (e.g., interaction modalities and resolution, interaction context, etc). For example, when the user is busy with another task (e.g., listening to an important discussion), the model can indicate that for optimal acceptance the user should be able to use only one single headshake (indicating only two levels of good or bad evaluation of the settings of the hearing device). In another context (e.g., when the user is walking in the forest), the model can indicate that for optimal acceptance the user should be able to express evaluation with more precision, and for example, use several headshakes (indicating more levels of evaluation of the settings of the hearing device). Together with the PhD student for research project 5, you will closely collaborate on integrating the proposed determinants of acceptance (research project 5) with the parameterized model for user interactions (research project 4).

Interaction Design for Hearing Aid Personalization

Scientific challenge

This research project will focus on the human-technology interaction component of hearing aid personalization. Firstly, the tuning process should be as unobtrusive as possible to the HA wearer, since the tuning process should not interfere with daily tasks, such as ongoing conversations. Secondly, the tuning process should be covert in the sense that it should not be noticeable to conversation partners and other bystanders. Finally, there are privacy, security, and ethical issues relating to the in-situ recording of acoustic data. We wish to avoid storing recorded data for offline training. The main research challenges are to identify (social) psychological, ethical, legal, and practical issues of in-situ hearing device tuning and translate these issues to an appropriate interaction protocol. To formulate the interaction protocol, we need to understand why users choose personalized algorithms, and under which circumstances personalization might not be desirable. Based on earlier research on acceptance of technology, satisfaction, and usability, the current research will identify (social) psychological, ethical, legal, and practical issues of in-situ hearing device tuning and translate these issues to an appropriate interaction protocol. Different from earlier research on these topics, for understanding acceptance of a hearing device, it is crucial to investigate determinants of more covert acceptance. That is, preferably, users remain unaware of many characteristics of this technology. For example, users preferably remain only peripherally aware of many interface elements, and even when interacting with the hearing aid is necessary (e.g., for requesting personalization), that interaction remains as covert as possible (to the user and her/his surroundings). Additionally, as described in research project 4, research projects 4 & 5 closely collaborate on integrating the proposed determinants of acceptance (research project 4) with the parameterized interaction model for user interactions (research project 5) allowing not only to adapt the parameters of the interaction model based on the modeling of covert and overt determinants of acceptance and trust (identified and tested in research project 5), both also allowing creation (research project 4) of a model that incorporates parameters of characteristics of the interaction itself (e.g., interaction modalities and resolution, interaction context, etc). Thereby, this research opens new domains of research into technology acceptance, satisfaction with, and usability of more covert technologies.