A Flutter plugin for music sequencing

flutter_sequencer

this flutter plugin lets you set up sampler instruments and create multi-track sequences of notes that play on those instruments. you can specify a loop range for a sequence and schedule volume automations.

it uses the core sampler engine from audiokit on both android and ios, which lets you create an instrument by loading some samples and specifying what notes they are. if you play a note you don’t have a sample for, it will pitch shift your other samples to fill in the gaps. it also supports playing sf2 (soundfont) files on both platforms, and on ios, you can load any audiounit instrument.

the example app is a drum machine. in theory, though, you could make a whole sample-based daw with this plugin. you could also use it for game sound effects, or even to generate a dynamic game soundtrack.

how to use

create the sequence

final sequence = sequence(tempo: 120.0, endbeat: 8.0);

you need to set the tempo and the end beat when you create the sequence.

create instruments

final instruments = [
  sf2instrument(path: "assets/sf2/tr-808.sf2", isasset: true),
  sfzinstrument(path: "assets/sfz/splendidgrandpiano.sfz", isasset: true),
  samplerinstrument(
    id: "80's fm bass",
    sampledescriptors: [
      sampledescriptor(filename: "assets/wav/d3.wav", isasset: true, notenumber: 62),
      sampledescriptor(filename: "assets/wav/f3.wav", isasset: true, notenumber: 65),
      sampledescriptor(filename: "assets/wav/g#3.wav", isasset: true, notenumber: 68),
    ]
  ),
];

an instrument can be used to create one or more tracks.
there are four instruments:

1. samplerinstrument, to load a list of sampledescriptors in the audiokitsampler manually without an sfz file
   • on ios and android, it will be played by the audiokit sampler
   • a sampledescriptor can point to a .wav or a .wv (wavpack) file. i recommend using .wv when
     possible, since it supports lossless compression. it’s easy to convert audio files to wavpack
     format with ffmpeg.
   • the only things you need to specify about a sample are its path, whether or not it’s an asset,
     and what note number it should correspond to.
   • optionally, you can set a note range or a velocity range for each sample. you can create
     “velocity layers” like this, where lower velocities trigger different samples than higher ones.
2. sfzinstrument, to load a .sfz sfz file.
   • this will also be played by the audiokit sampler.
   • only a few sfz opcodes are supported. look at sfz_parser.dart to see which ones are
     acknowledged.
   • this is not a full-fledged sfz player. sfz just provides a convenient format to load samples
     into the sampler.
3. sf2instrument, to load a .sf2 soundfont file.
   • on ios, it will be played by the built-in apple midi synth audiounit
   • on android, it will be played by tinysoundfont
4. audiounitinstrument, to load an audiounit
   • this will only work on ios

for an sf2 or sfz instrument, pass isasset: true to load a path in the flutter assets directory.
you should use assets for “factory preset” sounds. to load user-provided or downloaded sounds
from the filesystem, pass isasset: false.

optional: keep engine running

globalstate().setkeepenginerunning(true);

this will keep the audio engine running even when all sequences are paused. set this to true if you
need to trigger sounds when the sequence is paused. don’t do it otherwise, since it will increase
energy usage.

create tracks

sequence.createtracks(instruments).then((tracks) {
  setstate(() {
    this.tracks = tracks;
    ...
  });
});

createtracks returns future<list<track>>. you probably want to store the value it completes with in
your widget’s state.

schedule events on the tracks

track.addnote(notenumber: 60, velocity: 0.7, startbeat: 0.0, durationbeats: 2.0);

this will add middle c (midi note number 60) to the sequence, starting from beat 0, and stopping
after 2 beats.

track.addvolumechange(volume: 0.75, beat: 2.0);

this will schedule a volume change. it can be used to do volume automation. note that track volume
is on a linear scale, not a logarithmic scale. you may want to use a logarithmic scale and convert
to linear.

control playback

sequence.play();

sequence.pause();

sequence.stop();

start, pause, or stop the sequence.

sequence.setbeat(double beat);

set the playback position in the sequence, in beats.

sequence.setendbeat(double beat);

set the length of the sequence in beats.

sequence.settempo(120.0);

set the tempo in beats per minute.

sequence.setloop(double loopstartbeat, double loopendbeat);

enable looping.

sequence.unsetloop();

disable looping.

get real-time information about the sequence

you can use singletickerproviderstatemixin to make these calls on every frame.

sequence.getposition(); // double

gets the playback position of the sequence, in beats.

sequence.getisplaying(); // bool

gets whether or not the sequence is playing. it may have stopped if it reached the end.

track.getvolume();

gets the volume of a track. a volumeevent may have changed it during playback.

real-time playback

track.startnotenow(notenumber: 60, velocity: 0.75);

send a midi note on message to the track immediately.

track.stopnotenow(notenumber: 60);

send a midi note off message to the track immediately.

track.changevolumenow(volume: 0.5);

change the track’s volume immediately. note that this is linear gain, not logarithmic.

how it works

the android and ios backends start their respective audio engines. the ios one adds an audiounit
for each track to an avaudioengine and connects it to a mixer audiounit. the android one has to
do all of the rendering manually. both of them share a “basescheduler”, which can be found under the
ios directory.

the backend has no concept of the sequence, the position in a sequence, the loop state, or even time
as measured in seconds or beats. it just maintains a map of tracks, and it triggers events on those
tracks when the appropriate number of frames have been rendered by the audio engine. the
basescheduler has a buffer for each track that holds its scheduled events. the buffer is supposed to
be thread-safe for one reader and one writer and real-time safe (i.e. it will not allocate memory,
so it can be used on the audio render thread.)

the sequence lives on the dart front end. a sequence has tracks. each track is backed by a buffer on
the backend. when you add a note or a volume change to the track, it schedules an event on the
buffer at the appropriate frame, based on the tempo and sample rate.

the buffer might not be big enough to hold all the events. also, when looping is enabled, events
will occur indefinitely, so the buffer will never be big enough. to deal with this, the frontend
will periodically “top off” each track’s buffer.

development instructions

note that the android build uses several third party libraries. the gradle build will download them
from github into the android/third_party directory.

the ios build depends on the audiokit library. it will be downloaded by cocoapods.

to build the c++ tests on mac os, go into the cpp_test directory, and run

cmake .
make

then, to run the tests,

./build/sequencer_test

i haven’t tried it on windows or linux, but it should work without too many changes.

to do

prs are welcome! if you use this plugin in your project, please consider contributing by fixing a
bug or by tackling one of these to-do items.

difficulty: easy

• (important) enable setting the audiokit sampler envelope parameters
• change position_frame_t to 64-bit integer to avoid overflow errors
• make constants configurable (top_off_period_ms and lead_frames)
• support federated plugins

difficulty: medium

• start using dart null safety
• support tempo automation
• support pitch bends and custom tunings
• midi out instrument
  • create an instrument that doesn’t make any sounds on its own and just sends midi
    events to a designated midi output port.
• refactoring
  • globalstate, sequence, and track should have a clear separation of responsibilities
  • engine, sequencer, mixer, scheduler do too
  • “track index” and “track id” are used interchangeably, “track id” should be used everywhere
  • make the names and organization of the different files (plugin/engine/scheduler) consistent
• support macos
  • most of the ios code should be able to be reused
• add more c++ tests
  • for basescheduler, specifically.
• add dart tests
  • test loop edge cases
• ensure there are no memory leaks or concurrency issues

difficulty: hard

• generic c++ instrument
  • create an instrument that can be used on ios and android. the audiokit sampler works this way,
    but this library uses audiokit’s audiounit wrapper for it. the hard part is creating an
    generic audiounit wrapper for a c++ iinstrument.
• full sfz support
  • this would probably involve the sfizz library
    • it may be difficult to build sfizz and its dependencies for android and ios. ideally it
      would have a more modular build – this project doesn’t need its support for au/lv/vst,
      audio output, and audio file decoding.
• support windows
  • some of the code in the android directory might be able to be reused
• record audio output
• support react native?
  • could use dart2js

difficulty: very hard

• (important) audio graph
  • create a graph of tracks and effects where any output can be connected to any input.
  • at that point, the most of the audio engine can be cross-platform. on ios, it could be
    wrapped in one audiounit, and external audiounit instruments and effects could be connected to it
    via input and output buses.
  • labsound seems like the library to use
    • it’s based on webaudio, so there could also be a web backend
  • can be used to add some key effects like reverb, delay, and compression
  • supports audio input, writing output to file, and other daw stuff
• support web