Creating a visual language for the diplacusis piano

In previous posts I have discussed the construction of a “diplacusis piano”, a digital instrument that reproduces accurately what I actually hear. Diplacusis is a phenomenon in which you hear two different pitches, one in each ear. In my case, the left ear is mostly in tune, whereas the right ear is mostly out of tune, by fairly random amounts.

The problem with composing for the resulting instrument is twofold: firstly, because of my hearing loss I cannot hear the (quiet) sounds it produces very well; secondly, what I do hear I hear with diplacusis, so diplacusis on diplacusis!

How then to compose for this instrument, given that I have only a poor idea of what a person with normal hearing would hear? My solution is to develop a visual language based on the spectrograms of each note. I have been steadily learning about the character of each spectrogram as I go.

Here are some stills of most of the keyboard. The image quality has been reduced for speed of upload, but they are clear enough for you to be able to see how they vary. It’s really intriguing. My idea now is to start to connect together the various overtones to begin to create some kind of “harmony”. You’ll see that I have put gridlines on each image to help with this.

These are static images (generated with Pierre Couprie’s wonderful EAnalysis software). In the live performance, I will work with spectrograms that continuously evolve over time. This, I hope, will act both as a kind of score but also, for listeners who have even less hearing than myself, a visual version of the music that can be enjoyed without necessarily hearing everything.

So, here is a selection of the keyboard, just to give you an idea:

And here are just two notes for comparison at higher quality. You can see how different they are in terms of both structure and behaviour over time. This gives me a starting point for composition.

C4 (middle C)

Building the “Diplacusis Piano”, Part 3/3: Making Music!

In the last two posts (here and here) I have described the process of building a digital “piano” that reproduces my diplacusis. Having constructed the instrument with the help of Professor Craig Vear, I have begun to muse on the creative possibilities that this has revealed.

It is immediately clear that this is not really a piano at all, despite having piano sounds as its raw material. If I play a common chord, or attempt to play some classical piano music, all one hears is an out-of-tune piano. It’s a bit like a honky-tonk but worse – some kind of abandoned instrument. Interestingly, the brain filters out the “rubbish” from the signal and quickly the out-of-tuneness recedes into a normal piano.

So, to avoid sounding like I’m just trying to write piano music for a bad instrument, I must find a new way of thinking about composing for this diplacusis piano. This echoes my experience with diplacusis and hearing loss generally. I need to find new ways of listening if I am to appreciate and enjoy music now. My aim is to create something beautiful, despite the supposed limitations imposed by my condition.

Craig was keen to describe how each note, each adjusted sample, made a different sonic journey lasting 10 seconds. What he could hear was a fascinating mixture of rhythmical beats, emerging harmonics, clusters of partials, percussive noise, all evolving over time. Every single note has its own character, which he was able to describe to me in some detail, waving his arms expressively as he did so. So this is not a piano, but rather an 88-note composition with a total duration of just under 15 minutes!

The problem is, of course, that I cannot hear them! To me, each sample lasts about 3 seconds, and I do not trust what I hear even within that time frame. So, how can I possibly write music for this instrument if I cannot hear it properly?

Once again, new digital technologies come to my aid. Firstly, there are my wonderful GNResound Linx Quattro hearing aids. During the building of the instrument, I removed the hearing aids, so as to capture as accurately as possible my diplacusis. Now, by reinserting them, I can gain a much better impression of the sounds of the instrument. I can hear them for longer and understand the complex shifting interactions between the higher partials. However, the hearing aids alone are insufficient, especially in the lower registers. Even with my unvented mould, which prevents sound escaping from my right ear, the low end response is not enough.

As we worked on the instrument, we used a spectrogram to understand what was happening in each sample. This was fascinating, because it conveyed rich information about each note’s “story”, showing the strange rhythmic pulsations that arise from beats, the emergence and withdrawal of various overtones, the intensity of different registers, and so on.

So, my way of composing is becoming clear: I must familiarise myself with the story that each of my 88 mini compositions tells. Then I can string these together in ways which create a convincing musical narrative. There may be many such narratives – that remains to be seen – but each will have its own unique and engaging storyline that listeners can perceive.

To help them in this, I plan to add a video component to the performance, showing the spectrograms as they change, any musical descriptions (in text) or notations that are relevant, and perhaps a more imaginative interpretative layer. Multiple windows on a single screen, conveying the story of the piece.

This will help people in the Aural Diversity concert (where this will be premiered) whose hearing diverges from my own. They will be able to experience the composition in several ways at once. My performance will not resemble a traditional piano recital much. The keys on the keyboard are merely triggers for sonic navigations to begin. But it will hopefully prove engaging as I convey the emotional nature of the discoveries described in these posts and combine that with an informative and stimulating visual display.

Building the “Diplacusis Piano”, Part 2/3: In the studio

In the previous post I described the background to this project to construct a digital piano that renders my diplacusis audible to others. This post describes my studio session with Craig Vear, during which we assembled the entire instrument.

We worked in the Courtyard Studio at De Montfort University, which was the very first space I constructed when I started up the Music Technology programme back in 1997. Craig Vear is a former student of mine who is now a Professor. I’ve known him from the days of the BA Performing Arts (Music) degree at Leicester Polytechnic, where I started my academic career in 1986. It seems that past investments are repaying me handsomely! Here’s Craig in the studio, attempting to describe to me how one of the notes unfolds:

First we created middle C (C4) using Bosendorfer samples. This was something I had already done in my previous attempt, but the difference this time is that Craig’s ears were able to hear the interesting journey the difference tones take as the edited and filtered sample unfolds. This is the first clue about the creative possibilities that will subsequently emerge.

We matched the extent of my hearing loss in the right channel, in particular, and panned the stereo channels hard left and hard right. We introduced some filters to take out the lower frequencies as appropriate (it gets much more extreme in the lower registers) and some high ones too, using my audiogram as a guide. Finally, we detuned the samples. In most cases this was an adjustment only to the right channel, but sometimes it also entailed adjusting the left. Detuning meant converting frequency information in Hertz into cents (i.e. hundredths of a semitone). It’s a bit hard to make out in this photo, but the two high screens show an online hertz/cents converter on the left and my original diplacusis chart on the right. The desktop screens show the samples on the left and the filters and tuning information on the right.

I had already decided that none of the sounds will rise above piano (i.e. soft). This is because my hyperacusis also means that I find any loud sounds distressing nowadays. Having tried to play a conventional piano recently, I realised that the mechanical sound of hammers hitting strings is too painful for me, regardless of the diplacusis. So this will be a soft and gentle instrument.

So, to give an idea what this sounds like, here is the original sample plus its “diplacusis” version:

Untreated C4
Diplacusis-adjusted C4

We repeated this process across the entire 88-note range of the piano, following the findings described in the previous post. Here are some more C-diplacusis notes, to give an idea of the sheer range and variety of sounds that resulted:

C6 (N.B. – this is unaffected by diplacusis)

The final step in the building process is to create an instrument in Logic (my sequencer of choice) using the ESX24 sampler. This maps the various samples across the whole instrument. In the range that I had specified using my singing method, we made individual samples for each note. In the other ranges we transposed samples up or down across a minor 3rd.

Building the “Diplacusis Piano”, Part 1/3: Background


In a previous post I described my struggles with diplacusis and my intention to build a “piano” that could reproduce the sounds that I actually hear for the benefit (?) of others. This series of posts will document the progress I have made so far and the exciting compositional possibilities that are opening up as a result.

Diplacusis is a disturbing phenomenon in which the two ears hear a given musical note at two different pitches. It is yet one more from the smorgasbord of symptoms associated with Ménière’s Disease (see this post for a detailed account of my Ménière’s experiences), alongside vertigo, hearing loss, tinnitus and aural fullness.

I decided to try to build a musical instrument that would convey to others what this sounds like. I wanted this to offer me a creative opportunity to make some beautiful music. What it is in fact providing is not just that, but a whole new direction for my composition.

This post is a detailed account of the first steps in building this instrument. It is necessarily a digital instrument: there is no way this could be done using traditional technologies. I have been greatly helped by my GNResound Linx Quattro hearing aids and by my friend, the composer and Professor Craig Vear, who provided not just technical fluency in the studio and an otologically “normal” pair of ears, but also the ability to describe each sound to me as it emerged from this new instrument.

Starting Points

I decided to start with a piano simply because that is the instrument I used to play back in the days when I regularly made music. Piano sounds also have a pleasing decay which I instinctively felt would work well with this phenomenon. Nobody wants to listen to sustained diplacusis!

In my previous scientific study of my own diplacusis, I mapped the differences in pitch across my own singing range by laboriously stopping the good ear and singing the pitch I heard in Hertz, then comparing it with the correct pitch. This gave me a verified chart from F#2 (~92Hz) to C4 (~261Hz). To understand what comes next, you need to see my audiogram:

Andrew Hugill’s audiogram (July 2017)

This one is a little bit out of date, but my hearing has not changed much since then. Observe that (as is usual in audiology) the right and left ears are reversed in the image. You will also notice that audiology takes no interest in frequencies below 125Hz or above 8kHz. This is because audiology is mainly interested in speech and, frustratingly, takes little account of music.

Anyway, you will see quite clearly that my right ear (in red) is way below my left ear. This is what severe hearing loss looks like. My left ear has normal hearing (above 10dB) in the region between 1500 Hz and 4000 Hz. This is my salvation in speech situations. But there is quite a lot of hearing loss around that. Nevertheless, my pitch perception in that ear is tolerable.

One other thing to notice is that the lower frequencies show a marked decline in both ears. This is typical of Ménière’s Disease, where the bass disappears first. By contrast, in age-related hearing loss (presbycusis) the high frequencies deteriorate first, which is why so many hearing aids concentrate on the high end.

First efforts

Now you can see why the next step in preparing for the instrument was so daunting and has taken me many months of struggle to figure out. I could no longer rely on either my audiogram or my singing voice to help me understand my own pitch perception, because the rest of the piano keyboard is simply out of range. To make matters worse, every time I tried it was like working in a hall of endlessly reflecting mirrors. I would listen to my diplacusis with my diplacusis… it was very uncomfortable and very tiring.

So with considerable effort, I worked on trying to understand my own hearing by feeling my way with trial and error. Gradually a number of key features emerged:

  1. There is an octave between F#5 (~698Hz) and F#6 (~1397Hz) where there is no diplacusis at all. In other words, I hear a piano just like a normal piano, as anyone else would, albeit with greatly reduced hearing in one ear.
  2. In the range above that, the diplacusis gradually reappears, getting worse the higher up you go. However, since the piano sounds pretty metallic in that register anyway the effect is not as disturbing as you might expect.
  3. The range from C4 (~261Hz) down to F2 (~87Hz) is affected by random amounts of diplacusis as per the chart from the earlier study.
  4. Below E2 (~82Hz) this random diplacusis effect continues, but now a new phenomenon enters, presumably resulting from the general loss in low frequency hearing. The fundamental frequencies of each note and then the first and second partials, gradually disappear, leaving a thudding sound and a collection of higher overtone frequencies. This complex spectrum is then subject to precisely the same diplacusis that affects the higher register, resulting in a perceptible shift in spectrum but no discernible change in pitch.
  5. And this is, I think, a novel finding: every diplacusis induced detuning is flat! This seems to contradict the received wisdom that diplacusis notes are sharp. I need to do more research into this.

Given the difficulties of translating the above into any kind of instrument, I eventually had to admit defeat and seek help. This is where Craig Vear enters the picture and the account of our building session yesterday will be the subject of my next post.