I had a conversation with a friend of mine on AIM about the challenges of passing signals around in an application, and he had some interesting ideas coming from a networking perspective (packet-based signals). I don't know if it'll be useful, but I present it here anyway, with his permission, cut and paste from the conversation with editing here and there to cut out the non-related stuff.
-----
01:13:50 PM) Darren: As for me, I'm researching "tokens" in relation to programming. The CLAM audio DSP guys referred to "passing tokens around" in terms of how their system works, and the stuff I'm finding on tokens in programming doesn't fit that.
(01:14:13 PM) cavehamst: huh, that could mean anything
(01:14:43 PM) cavehamst: passing pointers to data structures, some sort of queuing system, anything
(01:14:57 PM) Darren: Yeah, I thought so.
(01:16:46 PM) Darren: Another fellow and myself are brainstorming a synthesis and sampling program that will allow for the routing of control and audio signals, treat control signals as audio signals (that bit needs a bit of explaining), and allow for the creation of custom GUIs by sound and synth designers.
(01:17:25 PM) cavehamst: hmm, interesting
(01:17:56 PM) Darren: My main contribution is the "treating control signals as audio signals" part.
(01:18:52 PM) Darren: My thought is, if control signals can be routed in the same way as audio, and can be processed by DSP blocks, then a few simple DSP functions could be combined to create more complex functions.
(01:19:53 PM) Darren: But it would do this for control signals, so things like LFOs, envelopes, and lots of different types of parameter modulation could be created from building blocks.
(01:23:59 PM) cavehamst: so, you are going to wrap the audio packets with control data or?
(01:24:34 PM) Darren: Oh, for God's sake! Why didn't I think of that?!
(01:25:13 PM) Darren: Well, the thing is, they aren't packets. We're running the system in real-time as a clock ticks at the sample rate.
(01:26:05 PM) Darren: But the control signals will look just like audio signals, though with different periodic behavior.
(01:26:23 PM) cavehamst: well, at the base level, it's just a stream of numbers. you could packetize that, 'tokenize' if you would, and pass it through kind of like TCP/IP data with headers
(01:26:53 PM) cavehamst: provided that the audio stream is moving slow enough that your processor has the overhead to handle chunks, which, it should, I would think
(01:27:11 PM) cavehamst: that's the method employed in cell phone handsets, chunking out packets of data back and forth
(01:27:22 PM) Darren: There's one important reason that we can't: latency. When you play a note on the keyboard, you have to hear it start right away. Packetizing the data would generate too much delay from initial note-on.
(01:27:28 PM) cavehamst: of course, the core reason for doing that on a cell network is to handle radios dropping data, but hey
(01:28:31 PM) cavehamst: well, the packets could be really small, chop them up every millisecond or so. i can't discern anything under about 25ms as being slower or faster then 1ms
(01:28:52 PM) Darren: At 44.1khz, the packets would have to be 128 sample large (a sample would be a 32-bit float).
(01:30:18 PM) cavehamst: eh, just an idea. you have to tag the data somewhere, maybe by the control signal referencing an index pointer in the audio stream as it goes through
(01:30:50 PM) Darren: Well, the idea is, the control stream would hook up to a parameter that will change as the stream changes.
(01:31:04 PM) Darren: The parameter could be, for example, the cutoff of a filter.
(01:31:12 PM) cavehamst: how often would the control signal need to change? could you just preface a control signal in front of a train of audio with the implication of, 'do this until next control frame'?
(01:31:28 PM) Darren: And the filter cutoff would go up or down as the control signal does.
(01:32:00 PM) Darren: That's a thought. Typically, you can run control signals at about 1/10th the rate of audio, and not notice it.
(01:32:29 PM) Darren: I want to avoid that, though, as there are certain advantage to running control signals at audio rates.
(01:32:43 PM) Darren: Audio-rate frequency and amplitude modulation being among them.
(01:33:14 PM) Darren: That's essentially a whole other form of sound synthesis available as a side effect.
(01:33:16 PM) cavehamst: well, really small control signals tacked onto small audio trains, kind of like packetizing them, shouldn't hurt throughput much
(01:34:02 PM) cavehamst: heh, have something like a baudot tokenizer to strip out the control signal automagically
(01:34:09 PM) Darren: The destinations are also different. Audio signals will go to the input of a function, control signals will go to a parameter of the function.
(01:34:35 PM) Darren: But I guess that can be dealt with once the control signal is stripped out.
(01:34:37 PM) cavehamst: you'd need some way of syncing the two
(01:34:54 PM) cavehamst: if they dont arrive right at the same time and are processed in parallel
(01:35:00 PM) Darren: Well, yeah, the master clock would be the sync source, at the sample rate.
(01:35:06 PM) cavehamst: ok
(01:35:34 PM) Darren: Hrm...
(01:35:43 PM) Darren: You are making me think, though, and that's good.
(01:36:28 PM) cavehamst: well, my one piece of advice for tokenizing is this: the best way to handle asynchronous data is to have the packets be prefaced by a special flag, and end with a different special flag. that way, the pre-processor doesn't have to calculate packets, it just finds whole ones quickly and shuffles them off to a handler
(01:36:46 PM) cavehamst: i dealt with a serial protocol once that had to be calculated as you go
(01:37:22 PM) cavehamst: if the 8th bit was then, then expect x more bytes, and the nth byte told you how many more bytes to eat, and then there was a checksum, and, gah
(01:38:03 PM) cavehamst: you ended up having to tokenize and deal with the data in realtime as it came in, instead of being able to take advantage of threads and parallelize the incoming data, and process the data packets in parallel
(01:38:08 PM) cavehamst: anyway
(01:38:24 PM) cavehamst: i've spent too much of my time dealing with asynchronous protocols
(01:38:49 PM) Darren: I don't think a signal packet method is the way to go, but it's always good to listen to other ideas.
(01:39:25 PM) cavehamst: hehe
(01:39:34 PM) Darren: But there are ideas in there that might find use.
(01:39:37 PM) cavehamst: i'm full of ideas, whether any of them work, that's open for debate
(01:39:44 PM) Darren: Especially the idea of multiplexing signals.
(01:41:21 PM) Darren: The other thought is splitting audio data into power-of-two sized packets makes them automatically FFT-ready.
(01:42:19 PM) Darren: But the convolution library we were going to use allows for variable-frame FFT, which is a great boon to real-time synthesis.
(01:42:33 PM) Darren: variable frame size, I mean
(01:43:13 PM) Darren: Any DSP block that needs to do FFT will likely handle its own windowing.
-----
And that's it in a nutshell.