Page 1 of 3

The Design Thread

PostPosted: Fri Jan 25, 2008 7:05 pm
by Consul
The idea here is to create a companion thread to "The Ideas Collection" that can hopefully help us nail down a final architecture for this new engine.

I'm still a beginner coder, but I do know a thing or two about music. :) One of the big issues I always have with large projects is knowing what to start with. A project like this is definitely large. Even with a solid design, knowing what first piece of code to write can be difficult. So at this point, perhaps deciding on a solid design would be a good first step.

I really don't want to step on any toes here, Christian and Benno, so please feel free to reign me in if you feel I'm overstepping my bounds.

Re: The Design Thread

PostPosted: Thu Jan 31, 2008 5:53 am
by Consul
Okay, I've been doing some more thinking. Here's an idea and some drawings of how this modular system might go together.

First, an overview of the whole thing:


The idea here is that there is a global (this would be "global" on a per-patch basis) section that handles the things all notes need (MIDI messages, etc.). This is also the process that the child voice threads will be spawning off of. The mixer probably doesn't really need any explanation, but this is where global effects routing would happen.

Here's a close-up of an example of the global section:


The idea here is that this is the section that can send data to all of the voices currently playing, whatever they may be. So, there's a global scripting engine that can talk to all voices. This is where one would handle things like key-switching, key-latching, and the like. Of course, the structure I show is an example, as a modular system would allow this to be flexible.

Here's a close-up of a voice:


Again, a modular structure allows pretty much anything to happen. Here, I have a script controlling something about a sample oscillator (loop points, perhaps?), and a script controlling the filter envelope in some fashion (the sky's the limit here- sustain level based upon note number, maybe?). These scripts are local only to that voice. Indeed, I don't see why you couldn't use scripts for envelopes. That would be nice for my split-frequency wavetable resynthesis idea I mentioned in another thread.

Of course, nothing says the scripts can only operate on control signals:


It might eat CPU to use scripting for audio, but it would allow for a lot of playing around and prototyping.

Basically, this all comes down to the idea of "script as object" in patch building, at both a global and local level. Of course, when to pick global and when to pick local scripting and control would be up to the patch programmer, as many functions could overlap.

These are my ideas anyway. Take them for what they're worth.

Re: The Design Thread

PostPosted: Wed Feb 27, 2008 3:45 pm
by dahnielson
I have been thinking about sampler design lately while taking a closer look at Kontakt, SFZ and hardware designs.

The basic signal paths in most samplers (or rather for a single voice) are relatively common:


Building blocks
* A sample player, the source and play back the sampler in one way or another. Typically it has several parameters and at least one for the pitch. Control how something is played back.
* A filter bank that manipulates the audio stream and change timbre. Parameters depend on what filters that are available and inserted. Shapes the timbre and sound.
* An amplifier that control the output volume and stereo panning. The parameters are quite obvious.

The sample player being used can be thought of as a realization of a generalized source. Traditionally a sampler have just had one type of playback mechanism while a next generation design may have several: playback from RAM, playback from disk stream, rubber band playback, pure synthesis, etc.

Exactly what kind of filters that is available in the filter block is another design choice. The common and basic approach is to have a defined set of band pass and eq filters, the E-MU Proteus 2k has a flexible Z-plane filter that replace several types of filters at once, while Kontakt let you insert a number of filters. In a flexible next generation design only the available hardware (CPU and RAM) sets the limit for the number and kinds of filters inserted in the filter bank.

The only part that is relatively straight forward compared to the other blocks is the amplifier. No apparent need to be able to swap it out for other realizations unlike the source and filter blocks. It does what it does.

Parameterization and modulation
In most samplers it usually just comes down to what parameters that are available and how (or if) they can be manipulated. A really basic approach is to just having constant values, another (and common) is to have associated EGs and LFOs for some or all of the controls (SFZ for instance define a EG and LFO for every building block). I think my E-MU Proteus 2k is one of the most modular, take at a look at the basic programming chapter in the manual for example. E-MU makes the point that it's not a sample player but a synth that happen to use samples instead of oscillators and take a "patch" approach (but is instead limited by the number of available modules that can be patched together at once).

So we have a generalized signal path, but as you can see there several realizations of it. The actual implementation of each building block, and parts in the filter case of the filter bank, can be chosen out of a pool of ready implementations (and new ones can be added). The number of modulators that can be patched to parameters of the building blocks will be even greater.

So far we have just concentrated on the signal path for a single voice: the stuff that will make some kind of noise in the end. Another important block is the mapper who is responsible for mapping input events to parameter and modulator settings in addition to selecting and loading sample to be played back for each voice. It need to be really flexible and encompass a lot of needs that can be easily expressed. It is the equivalent of the dimension system in the DLS and GIG formats while the SFZ format has a smart set of "opcodes" with mapping possibilities similar to Kontakt (afaik).

GUI and scripting
The scripting in Kontakt is usually heralded as its core power. But for the most part that isn't true: the power comes from a great deal of flexibility and powerful building blocks so that few instrument need to take full advantage of the scripting. The exception being the line of basses from Scarbee that really use the possibility of the custom performance view GUI and manipulation/transformation of input events to make them really playable and realistic. To sum it up, in the case of Kontakt, scripting have three uses: creating custom GUIs, generating/filter/transforming input events and setting/changing parameters based on GUI and event callbacks. A script works on an instrument level and can be thought of as glorified programmable MIDI processor, MIDI event in MIDI event out, but with read/write access to internal parameters. When it comes to GUI there's always the question about separation using MVC/MVP patterns.

Engine definition and instrument loader
As a first step the loader define the engine: what modules it consist of and how they are put together. Then it reads the user provided instrument file it was designed for and fill in the mapper, set the appropriate parameters and load any scripts. Such flexibility makes it relatively easy to emulate the mapper and signal path of other samplers to play back formats like AKAI programs or SFZ instruments. An engine definition is created emulating what was available in the original sampler and then an instrument file loader need to be written for the format to populate the engine with instruments.

Re: The Design Thread

PostPosted: Wed Feb 27, 2008 4:29 pm
by dahnielson
Classes of operators
* signal shapers, high frequency operators that sits directly in the audio signal path and have both inputs (both audio signal and controllers) and outputs operating on a per sample basis.
* modulators, low frequency operators that get patched in as controllers to the signal shapers and whose outputs operate on a per sample basis while the inputs usually operates on a per event basis (but nothing stops us from patching together several modulators serially if that is a design desire and if it makes sense).
* event processors, like the mapper and script engine, that operates on a per event basis and so do their in- and outputs.

Re: The Design Thread

PostPosted: Wed Feb 27, 2008 7:00 pm
by dahnielson
Platform independent extensions
There have also been talks about some sort of DSP language. I see it fit into this model as the extension language to write additional filters (or any insert or send effect -- I didn't touch anything outside the voice audio signal path in my discussion above) and modulators in. A simplistic approach would be to just let the model provide a set of pre-defined filters and modulators that can be patched together, or provide them as plugins built for each platform. While the idea here is to have a platform independent language and in addition to having a standard set even let instruments provide their own. A bit like OpenGL shader language and Nvidia Cg but for audio. As already mentioned in the idea thread, this can be accomplished by defining a subset of C/C++ (or create a whole new front end to GCC supporting our own syntax) that is guaranteed to be available in terms of libraries and use GCC to build a dynamic library object that gets cached and loaded.

Simply put: A plugin that is distributed in source with the instrument and compiled by the sampler when the instrument is loaded (unless already cached).

Engine definitions and instruments
In this model you would be able to create separate engine definitions, patching together the stuff that need to be available, and then load instruments from other samplers (or your own format) into the defined engine. Of course a instrument fully native to the model could include a custom engine definition (and optional custom extensions to go with the definition) baked into the instrument.

Re: The Design Thread

PostPosted: Wed Feb 27, 2008 7:12 pm
by Consul
Heck, make the DSP language include enough functionality, design the modules to load as dynamic library objects, and you could make the entire sampler out of those building blocks. You could make anything else you want, for that matter. How about some Bezier curve oscillators?

I don't know about you, but I like that idea. CPUs are so powerful these days anyway. I'd rather have the power and flexibility.

Re: The Design Thread

PostPosted: Wed Feb 27, 2008 7:37 pm
by dahnielson
Consul wrote:Heck, make the DSP language include enough functionality, design the modules to load as dynamic library objects, and you could make the entire sampler out of those building blocks. You could make anything else you want, for that matter. How about some Bezier curve oscillators?

I don't know about you, but I like that idea. CPUs are so powerful these days anyway. I'd rather have the power and flexibility.

Yes, the general idea I had is that the modules are entry points for extensions developed in the DSP language. But filters and modulators are the most interesting because a "filter" can be any type of DSP operator and a modulator any type of low frequency operator. All the other bits (sources, amp, mapper, script engine) have a more finite set of implementations and don't vary that much, some of them are clearly not operating in DSP territory or only partly doing so (like the sources, mapper, script engine).

Of course implementing as much as possible in the DSP language, eating you own dog chow so to speak (the food not the breed)...

Agree that the general approach could be "everything is a plugin" and "here's a engine definition how they are patched together" where many plugins (a.k.a. extensions) are shipped with the sampler as standard (no need to reinvent the wheel).

Re: The Design Thread

PostPosted: Wed Feb 27, 2008 7:49 pm
by Consul
The other thing to think about is, if a framework could be made wherein modules can be coded and loaded (hey, a rhyme!) and then used, it would be much easier to attract other programming talent, who would see that they don't need to know the internal specifics of the engine in order to make cool new modules, like filters or what have you. If I needed to dive into the complete source code tree just to play around with a new filter design, I'd be less likely to get started.

Not to trumpet Reaper again, but the Jesusonic DSP language is ultimately what I found most attractive about it. Unfortunately, I can't afford to use Windows for music production anymore. When I get to be a better coder, something I'd like to do is make a JS-like language as a JACK client. Maybe FAUST could work in this manner.

Re: The Design Thread

PostPosted: Wed Feb 27, 2008 9:13 pm
by dahnielson
I just want to point out that I've been using "engine definition" as a label for a certain concept but not suggested how it would be implemented but that it can just as well be expressed in our hypothetically DSP language as an initialization callback. The same goes for the "scripting" part that can be event processors expressed in the DSP language.

(For the moment I've just been trying to think about stuff on a more abstract level, otherwise it easy fall into the trap of having every instrument builder build their own sampler from scratch and we're just end up with another generic DSP library.)

Re: The Design Thread

PostPosted: Wed Feb 27, 2008 11:26 pm
by Consul
That's a very good point, actually. My thinking was, if we build the framework, from there, we can build a useful instrument that people can then use directly. But, the underlying framework is exposed if people want to try coding their own modules.

The first question I have is, where would we start? This is a big project, and it needs planning. Top-down, or bottom-up? I can put together a design doc skeleton that we can fill in as we go. Or will this whole idea be rejected out of hand? I don't know. I personally don't see the point in just making another sampler with a fixed architecture. LinuxSampler as it is now already works great for that. I really want something I can dive into.