• Multi-source Raw Audio Mixer
  • Example Usage
  • How does it work?
• Test documentation

Multi-source Raw Audio Mixer

A component that mixes raw audio data from an unknown number of sources, that can change at any time. Audio data from each source is buffered until a minimum threshold amount, before it is included in the mix. The mixing operation is a simple addition. Values are not scaled down.

Example Usage

Mixing up to 3 sources of audio (sometimes a source is active, sometimes it isn't):

Graphline(
    MIXER = RawAudioMixer( sample_rate=8000,
                           channels=1,
                           format="S16_LE",
                           readThreshold=1.0,
                           bufferingLimit=2.0,
                           readInterval=0.1),
                         ),
    A = pipeline( SometimesOn_RawAudioSource(), Entuple(prefix="A") ),
    B = pipeline( SometimesOn_RawAudioSource(), Entuple(prefix="B") ),
    C = pipeline( SometimesOn_RawAudioSource(), Entuple(prefix="C") ),

    OUTPUT = RawSoundOutput( sample_rate=8000,
                             channels=1
                             format="S16_LE",
                           ),
           linkages = {
               (A, "outbox") : (MIXER, "inbox"),
               (B, "outbox") : (MIXER, "inbox"),
               (C, "outbox") : (MIXER, "inbox"),

               (MIXER, "outbox") : (OUTPUT, "inbox"),
           },
         ).run()

Each source is buffered for 1 second before it is output. If more than 2 seconds of audio are buffered, then samples are dropped.

How does it work?

Send (id, raw-audio) tuples to RawAudioMixer's inbox. Where 'id' is any value that uniquely distinguishes each source of audio.

RawAudioMixer buffers each source of audio, and mixes them together additively, outputting the resulting stream of audio data.

Constructor arguments:

• sample_rate, channels, format The format of audio to be mixed. The only format understood at the moment is "S16_LE"
• readThreshold number of seconds of audio that will be buffered before RawAudioMixer starts mixing it into its output.
• bufferingLimit maximum number of seconds of audio that will be buffered. If more piles up then some audio will be lost.
• readInterval number of seconds between each time RawAudioMixer outputs a chunk of audio data.

RawAudioMixer buffers each source of audio separately. If the amount of audio in any buffer exceeds the 'buffering limit' then the oldest samples buffered will be lost.

When one or more buffered sources reaches the 'read threshold' then they are mixed together and output. How often audio is output is determined by setting the 'read Interval'.

Mixing is done additively and is not scaled down (ie. it is a sum() function, not an average() ). Therefore, ensure that the sum of the sources being mixed does not exceed the range of values that samples can take.

Why the buffering, thresholds, and read intervals? It is done this way so that RawAudioMixer can mix without needing to know what sources of audio there are, and whether they are running or stopped. It also enables RawAudioMixer to cope with audio data arriving from different sources at different times.

You may introduce new audio sources at any time - simply send audio data tagged with a new, unique identifier.

You may stop an audio source at any time too - simply stop sending audio data. The existing buffered data will be output, until there is not left.

If there is not enough audio in any of the buffers (or perhaps there are no sources of audio) then RawAudioMixer will not output anything, not even 'silence'.

If a shutdownMicroprocess or producerFinished message is received on this component's "control" inbox this component will cease reading in data from any audio sources. If it is currently outputting audio from any of its buffers, it will continue to do so until these are empty. The component will then forward on the shutdown message it was sent, out of its "signal" outbox and immediately terminate.

TODO:

• Needs a timeout mechanism to discard very old data (otherwise this is effectively a memory leak!)
  • If an audio source sends less than the readThreshold amount of audio data, then stops; then this data never gets flushed out.

Test documentation

Tests passed:

• Multiple sources of audio will be mixed
• If there is no input (no incoming audio data), there will be no output (no outgoing audio data)
• A single source of audio will pass through