Computational Modelling of Variance in Musical Harmony

European tonal art music, unlike other forms of art, enjoys rigorous formalization and a rich vocabulary of descriptions that have a relatively high precision and expressive power. With music theory, we aim to explain and offer generalizations about the concepts and processes of music. For example, the simultaneous sounding of multiple musical notes that form a coherent entity in the mind of a listener is commonly referred to as harmony. These entities, the basic components of harmony, are structures that are generally referred to as chords, which are sonorities of (commonly, three or more) notes. Notwithstanding the rigorous formalization of music, this dissertation argues that people often arrive at diverging harmonic analyses of the same musical piece, which results in harmonic variance: differing, yet useful harmonic analyses of the same musical piece. Reasons for diverging analyses are differences in application of music theory to ambiguous musical passages and fundamental and inherent human perceptual and cultural differences that stem from enculturation differences. Disagreement in harmony analyses is problematic for creating ground-truth datasets for current computational approaches to harmony analysis, such as automatic chord estimation. This dissertation proposes different approaches to investigate, incorporate, model, exploit and analyze variance in musical harmony found between human annotators and output of algorithms. In Chapter 2, this dissertation introduces a new dataset with multiple reference annotations and several statistical methods to investigate the degree of agreement between expert harmonic annotators. Having multiple reference annotations means that either the variance needs to be resolved in a smart way through a data fusion algorithm, as is proposed in Chapter 3, or that the paradigm of automatic chord estimation needs to be expanded to allow for multiple subjective reference annotations, as is proposed in Chapter 4. The inherent ambiguity of harmonic analysis and transcription is also problematic for evaluating automatic chord estimation systems. For example, it raises the problem of how to evaluate and compare the output of algorithms on the same piece of music. In the final chapter (Chapter 5) of this dissertation we investigate the algorithmic disagreement between automatic chord estimation systems evaluated on different musical performances of Schubert’s Winterreise. This dissertation demonstrates that the next leap forward for computational harmony analysis is to not stick to the conventional aim of describing immanent or idealized harmonic structures, but instead move towards modeling esthesic, or perceived musical structures. Since music is inseparable from the occasion and purpose for which it is produced, the computational modeling of harmony should also take into account occasion and purpose through the modelling of the inherent variance found in human harmonic transcriptions.