From tim at rootdown.net  Tue Apr 12 14:05:14 2022
From: tim at rootdown.net (Tim Grant)
Date: Tue, 12 Apr 2022 15:05:14 -0600
Subject: [Stk] MIDI event timing relative to audio stream
Message-ID: <CAPkkH4WZZHyY2CeuuoLWaVHvYwR-mBnWMtD5piSOFnLWKZPUcw@mail.gmail.com>

Hello!

I've recently started prototyping a synthesizer project using RtAudio and
RtMidi. I'm doing the initial development in Windows and I plan to
eventually port the project to something like a Raspberry Pi if the
prototype goes well.

I have the audio output and MIDI input working, and I'm starting to think
about triggering the synth voices with MIDI notes. I would like to keep the
jitter between the notes and the triggered sounds low, but I'm not quite
sure how to achieve this with these two APIs.

A diagram might be helpful here:
MIDI      -----N----------------------N-------------------------->
Input          |                      |
               |                      |
Audio     |----*-----|----P-----|-----*----|-----P----|---------->
Stream       Frame 1    Frame 2    Frame 3    Frame 4    Frame 5

My thinking is that I need to know at what point in the current audio frame
an MIDI note arrives. From the diagram, assume a note (N) arrives at the
moment corresponding to the 5th sample in Frame 1; knowing this, I can
begin playback at the 5th sample on the next frame (P). This way, the
latency is held constant at 1 frame and the amount of jitter is minimized.

So with that long-winded introduction out of the way, my question is:

Is it possible to recover the timing of MIDI events relative to the audio
stream? I can't see how to do this using the delta time provided with the
MIDI note and the stream time provided by RtAudio, but my hope is that I'm
missing something.

Thanks!

Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://cm-mail.stanford.edu/pipermail/stk/attachments/20220412/37f951f9/attachment.html>

From arthur.benilov at gmail.com  Tue Apr 12 15:38:34 2022
From: arthur.benilov at gmail.com (Arthur Benilov)
Date: Tue, 12 Apr 2022 23:38:34 +0100
Subject: [Stk] MIDI event timing relative to audio stream
In-Reply-To: <CAPkkH4WZZHyY2CeuuoLWaVHvYwR-mBnWMtD5piSOFnLWKZPUcw@mail.gmail.com>
References: <CAPkkH4WZZHyY2CeuuoLWaVHvYwR-mBnWMtD5piSOFnLWKZPUcw@mail.gmail.com>
Message-ID: <CAEUJLjA8C9_F3=c+tqGKxbsawJBxeUomCq4Bq7_cmyB7hE7=-A@mail.gmail.com>

Hi Tim,
You can timestamp the MIDI events relative to the next audio frame using
the system steady clock. The very first MIDI event received will have delta
time zero, all following events will have delta time measured relative to
the first event. By relating these to the system clock and the system time
recorded at the audio callback you can estimate for a particular MIDI event
its time within the next audio frame. Also you don't need to be
sample-perfect, all commercial software synths I know will discretize the
MIDI events processing within an audio frame to 16 or 32
samples @44.1-48kHz. You may also let the jitter be and just decrease the
audio frame size to minimize it (Raspberry Pi will be able to do 256
samples per frame @44.1kHz with its onboard audio), it may not be that
critical for an instrument you are developing.

Cheers,

Arthur.


On Tue, Apr 12, 2022 at 10:06 PM Tim Grant <tim at rootdown.net> wrote:

> Hello!
>
> I've recently started prototyping a synthesizer project using RtAudio and
> RtMidi. I'm doing the initial development in Windows and I plan to
> eventually port the project to something like a Raspberry Pi if the
> prototype goes well.
>
> I have the audio output and MIDI input working, and I'm starting to think
> about triggering the synth voices with MIDI notes. I would like to keep the
> jitter between the notes and the triggered sounds low, but I'm not quite
> sure how to achieve this with these two APIs.
>
> A diagram might be helpful here:
> MIDI      -----N----------------------N-------------------------->
> Input          |                      |
>                |                      |
> Audio     |----*-----|----P-----|-----*----|-----P----|---------->
> Stream       Frame 1    Frame 2    Frame 3    Frame 4    Frame 5
>
> My thinking is that I need to know at what point in the current audio
> frame an MIDI note arrives. From the diagram, assume a note (N) arrives at
> the moment corresponding to the 5th sample in Frame 1; knowing this, I can
> begin playback at the 5th sample on the next frame (P). This way, the
> latency is held constant at 1 frame and the amount of jitter is minimized.
>
> So with that long-winded introduction out of the way, my question is:
>
> Is it possible to recover the timing of MIDI events relative to the audio
> stream? I can't see how to do this using the delta time provided with the
> MIDI note and the stream time provided by RtAudio, but my hope is that I'm
> missing something.
>
> Thanks!
>
> Tim
> _______________________________________________
> Stk mailing list
> Stk at ccrma.stanford.edu
> https://cm-mail.stanford.edu/mailman/listinfo/stk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://cm-mail.stanford.edu/pipermail/stk/attachments/20220412/92c89979/attachment.html>

From tim at rootdown.net  Tue Apr 12 16:20:17 2022
From: tim at rootdown.net (Tim Grant)
Date: Tue, 12 Apr 2022 17:20:17 -0600
Subject: [Stk] MIDI event timing relative to audio stream
In-Reply-To: <CAEUJLjA8C9_F3=c+tqGKxbsawJBxeUomCq4Bq7_cmyB7hE7=-A@mail.gmail.com>
References: <CAPkkH4WZZHyY2CeuuoLWaVHvYwR-mBnWMtD5piSOFnLWKZPUcw@mail.gmail.com>
	<CAEUJLjA8C9_F3=c+tqGKxbsawJBxeUomCq4Bq7_cmyB7hE7=-A@mail.gmail.com>
Message-ID: <CAPkkH4U_KkZJOvy=XoyYfhRBZVg-XZQiHRGugs-hJsj=86QKBw@mail.gmail.com>

Thanks, Arthur.

Not all commercial synths discretize events to the frame boundaries. For
example, it's possible to trigger Native Instruments Battery with sample
accuracy when sequenced via a DAW, and even when being triggered via
external MIDI it does so with sub-frame accuracy within the limits of the
host MIDI driver. I have no illusions of achieving sample accuracy, but if
I can achieve sub-frame accuracy I will be happy. My intended application
is percussion-oriented so low jitter is a very desirable property for me. I
understand that RtMidi has been designed for ease of use, and not
necessarily for the highest performance.

I'm measuring the delta between MIDI events in the callback using a
high_resolution_clock, and the delta times differ from those reported by
the callback somewhat significantly, sometimes > 1ms. I suppose this is due
to the low precision of the timestamps with the Windows MM API? Or is the
timestamp reliable, and there is a variable time between the MIDI event and
the callback invocation? I haven't measured the jitter in the audio
callback yet. If the time between audio callbacks is reasonably stable then
the system clock might be a reliable-enough reference.

Also, perhaps this is a bug, but the delta time for the first event is not
zero, but some (often large) value like 4291232.2280 seconds. If that's
unexpected then I can file an issue on GitHub.

Thanks,

Tim

On Tue, Apr 12, 2022 at 4:38 PM Arthur Benilov <arthur.benilov at gmail.com>
wrote:

> Hi Tim,
> You can timestamp the MIDI events relative to the next audio frame using
> the system steady clock. The very first MIDI event received will have delta
> time zero, all following events will have delta time measured relative to
> the first event. By relating these to the system clock and the system time
> recorded at the audio callback you can estimate for a particular MIDI event
> its time within the next audio frame. Also you don't need to be
> sample-perfect, all commercial software synths I know will discretize the
> MIDI events processing within an audio frame to 16 or 32
> samples @44.1-48kHz. You may also let the jitter be and just decrease the
> audio frame size to minimize it (Raspberry Pi will be able to do 256
> samples per frame @44.1kHz with its onboard audio), it may not be that
> critical for an instrument you are developing.
>
> Cheers,
>
> Arthur.
>
>
> On Tue, Apr 12, 2022 at 10:06 PM Tim Grant <tim at rootdown.net> wrote:
>
>> Hello!
>>
>> I've recently started prototyping a synthesizer project using RtAudio and
>> RtMidi. I'm doing the initial development in Windows and I plan to
>> eventually port the project to something like a Raspberry Pi if the
>> prototype goes well.
>>
>> I have the audio output and MIDI input working, and I'm starting to think
>> about triggering the synth voices with MIDI notes. I would like to keep the
>> jitter between the notes and the triggered sounds low, but I'm not quite
>> sure how to achieve this with these two APIs.
>>
>> A diagram might be helpful here:
>> MIDI      -----N----------------------N-------------------------->
>> Input          |                      |
>>                |                      |
>> Audio     |----*-----|----P-----|-----*----|-----P----|---------->
>> Stream       Frame 1    Frame 2    Frame 3    Frame 4    Frame 5
>>
>> My thinking is that I need to know at what point in the current audio
>> frame an MIDI note arrives. From the diagram, assume a note (N) arrives at
>> the moment corresponding to the 5th sample in Frame 1; knowing this, I can
>> begin playback at the 5th sample on the next frame (P). This way, the
>> latency is held constant at 1 frame and the amount of jitter is minimized.
>>
>> So with that long-winded introduction out of the way, my question is:
>>
>> Is it possible to recover the timing of MIDI events relative to the audio
>> stream? I can't see how to do this using the delta time provided with the
>> MIDI note and the stream time provided by RtAudio, but my hope is that I'm
>> missing something.
>>
>> Thanks!
>>
>> Tim
>> _______________________________________________
>> Stk mailing list
>> Stk at ccrma.stanford.edu
>> https://cm-mail.stanford.edu/mailman/listinfo/stk
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://cm-mail.stanford.edu/pipermail/stk/attachments/20220412/cc4f8ff8/attachment-0001.html>