EmoHarmonix: Innovating emotional audio insights with harmonic-synthesis deep learning architectures

Abstract

Emotional audio analysis holds significant potential in applications such as human-computer interaction, mental health monitoring, and customer service. However, current methods often fail to capture the intricate harmonic structures within audio signals, resulting in suboptimal emotion classification accuracy. Addressing these challenges, this study proposes novel harmonic-synthesis deep learning architectures that leverage the interplay between chords and musical harmonies for a more nuanced understanding of emotional content in audio. The methodology includes the introduction of these architectures, the creation of a meticulously curated emotional audio dataset encompassing diverse emotional states, and advanced feature extraction techniques that exploit harmonic properties. The harmonic-synthesis based feature extraction algorithm involves several steps: preprocessing, short-time Fourier transform (STFT), harmonic detection, harmonic feature extraction, temporal dynamics modeling with LSTM, feature aggregation, and normalization. Extensive evaluations show that the proposed models achieve significantly higher classification accuracy compared to traditional methods, with the harmonic-synthesis CNN reaching up to 93.7% accuracy on the RAVDESS dataset. The results indicate that the harmonic-synthesis approach effectively captures subtle emotional cues, thus offering more robust and precise emotion detection systems. By providing open-source implementations, this research fosters further advancements in this field. Additionally, the harmonic-synthesis architectures were benchmarked against existing state-of-the-art models, consistently outperforming them across various metrics. These findings underscore the potential of harmonic-synthesis techniques to significantly advance the field of emotional audio analysis, providing a solid foundation for future research and real-world applications.