Friday, July 11, 2008

VST.NET Plugin Persistence

VST.NET provides a simple way for a VST Plugin to persist its internal state. This post discusses how to implement persistence in a VST.NET Framework Plugin.

Before you start to implement persistence into your plugin make sure you need it. The Host will in general save all parameter data instead when no persistence support is detected in the Plugin. So when all you need are your Plugin Parameters you don't have to do anything.

But as soon as you support Programs; the ability to have multiple (editable) presets with Parameter settings then you have to implement persistence. When your plugin contains custom configuration data that is not expressed in Parameters (such as mapping items in a Midi Note Mapper), implementing persistence will be your only way to allow the user to save and load those settings inside the Host.

Another reason to implement persistence (even if you don't support Programs) is when you like to be backwards compatible with an earlier version of your Plugin. When the new version of your Plugin should be able to read in data of older versions, you will need to handle that manually.

Persistence is implemented through implementing an interface, like any other feature in the VST.NET Framework. The IVstPluginPersistence interface contains methods for reading and writing chunks of data and determining the version of the plugin that wrote the data (called during reading).

How and where you wish to implement the IVstPluginPersistence interface in your Plugin code design is up to you. Some options are discussed in this post: VST.NET Plugin Structure.

An important concept of VST persistence is that it has two modes: Persist a single (current) Program or all Programs. But you'll not see any method parameters to distiguish between the two modes in VST.NET. You'll only have to worry about implementing one set of persistence methods.

The ReadPrograms method reads multiple Programs from a Stream and stores them in the collection passed into the method. The WritePrograms method also takes a collection of Programs you have to serialize into the Stream. It doesn't really matter where those Programs came from (current or all). Its good practice to serialize a Program count into the stream so you know -when reading it back in- how many Programs to expect.

The CanLoadChunk method returns a boolean value to indicate if the data can be read. The method accepts a structure that contains details on the Plugin that wrote the data and its version.

Its important that you write out data in the exact same sequence as you expect to read it back in. One way to ensure this, is to make Reader and Writer classes in your plugin. These classes will be constructed on the Stream and have public methods to de-/serialize an object, say a Program or a Parameter.

Having these classes in your plugin architecture will also help moving to a new version while still supporting backwards compatibility.

So thats it. Implement three methods, expose the IVstPluginPersistence interface through the PluginInterfaceManager(Base) and you have persistence in your plugin.

Have any questions or suggestions? Leave them at VST.NET at the Discussion list.

Sunday, June 29, 2008

VST.NET Plugin Structure

VST.NET allows you to write VST Plugins in managed code (VB.NET/C#). This post discusses the basic principles and structure of a typical Plugin implementation. It will attempt to show you how to write a plugin using the VST.NET Framework using code snippets.

Please read the introduction that discusses the main components of VST.NET.

Life cycle of a Plugin

When the Host loads the renamed Interop assembly and calls the exported main method, the Interop loads the managed Plugin assembly (with the .net postfix). The assembly is scanned for a public type that implements the IVstPluginCommandStub and that class is instantiated. The Framework contains a default implementation for this interface that can be reused by a Plugin developer. The name of the derived stub class can be any valid class name you'd like.

public class MyPluginCommandStub : Plugin.StdPluginCommandStub,
Core.Plugin.IVstPluginCommandStub
{
protected override IVstPlugin
CreatePluginInstance()
{
return new MyTestPlugin();
}
}

The Plugin stub class is a public class and derives from the standard Framework implementation. The Plugin stub class also overrides the abstract CreatePluginInstance method to return a new instance of the Plugin root class (that implements IVstPlugin).

After the Plugin root class has been created it will receive a call to the Open method and from that moment on can expect VST calls on its interface implementations.

When the Plugin should be unloaded, its Dispose method is called.

The following picture displays the chain through which every Host to Plugin call travels.

Detailed information about the startup and tear down sequences can be found here.

Plugin Capability Discovery

Part of the Plugin creation sequence is communicating the plugin capabilities to the host. The VST.NET Framework employs a paradigm where groups of related VST Plugin methods are grouped into interfaces. When a Plugin implements an interface it communicates that capability implicitly.

The Plugin root class also implements the IExtensible interface (besides IVstPlugin). And it is through this interface that the Plugin is queried for its implemented interfaces. This explicit querying for interfaces allows a Plugin to dynamically determine the interfaces it supports.

The following figure displays the call sequence for a Host to Plugin call and how the StdPluginCommandStub forwards the call to an interface implementation.

To help with implementing the IExtensible interface the Framework provides a PluginInterfaceManagerBase class. The plugin either derives its root class or a seperate class from this base class and overrides the virtual creation methods; one for each interface. Only the methods for the supported interfaces have to overriden and implemented.

internal class MyPlugin : Plugin.PluginInterfaceManagerBase, IVstPlugin
{
protected override IVstPluginAudioProcessor CreateAudioProcessor(IVstPluginAudioProcessor instance)
{
if(instance == null) return new MyPluginAudioProcessor(this);

return instance; // return thread-safe instance
}

// IVstPlugin members....
}

All CreateXxxx methods contain an instance parameter. This parameter is null when the interface is queried for the first time. The CreateXxxx method should return a reference to the interface implementation. In the example a seperate class implements the Audio Processor and receives a reference to the Plugin root object in its constructor (this). The interface manager will store the interface reference and use that to satisfy subsequent requests. When an interface request for the same interface is received on a different thread it was created on (when CreateXxx was called with instance == null), the CreateXxxx method is called again specifying the original interface reference. It is now up to the CreateXxxx method implementation to return a thread-safe implementation for the interface. The original interface reference is passed to the CreateXxxx method in order to allow the implemenation to wrap it in a thread-safe wrapper class.

You, as a Plugin developer, are free to choose where you implement the Vst interfaces. The example used a seperate class but the IVstPluginAudioProcessor could also have been implemented on the Plugin root class itself instead. The implementation for the CreateAudioProcessor method would simply return this as an interface reference.

You can use the IExtensible interface internally to query for interface implementation classes as well (not just for interfaces). With the example code in place you could do this:

// myPlugin is a reference to the Plugin root class instance.
MyPluginAudioProcessor audioProcessor = myPlugin.GetInstance<MyPluginAudioProcessor>();

This makes navigating around the classes that implement the Vst interfaces a lot easier.

Vst Call Execution flow

  1. The host calls one of the callback functions that were returned in the AEffect structure from the exported main function.
  2. The Interop layer marshals that call to the implementation of the IVstPluginCommandStub. This interface defines a method for each possible call the Host can place tot the Plugin.
  3. The StdPluginCommandStub Framework class queries for the specific Vst interface. Each method of the IVstPluginCommandStub interface is mapped to 'a method' (or construct) on a Vst Framework interface.
  4. The Interface Manager attempts to map the requested (interface) type to one of the possible interfaces.
  5. When the Interface Manager has matched the type it checks for a non-null reference and calls the appropriate CreateXxxx method if there is no reference. It returns the reference to the requested type.
  6. The StdPluginCommandStub checks if the Plugin supports the requested Vst interface (not null) and calls 'the method' sometimes returning the return value from that method.

The key to exposing Plugin capabilities to the Host is to implement the related Vst interface and overriding the appropriate CreateXxxx method on the Interface Manager. The VST.NET Framework will take care of the rest.

The following picture displays the objects that make up a typical Plugin.

The VST.NET Plugin Framework aims to provide an easy was to implement Vst Plugins with lots of flexibility how and where the Vst interfaces are implemented. Just remember that the Interface Manager is at the heart of how these interfaces implementations are discovered.

I hope this post has shed some light on the basic structure of a VST.NET plugin.

Also read about Plugin Programs and Parameters.

Saturday, May 24, 2008

VST.NET Programs and Parameters

This post discusses how Plugin Programs and Parameters are implemented in VST.NET.

A VST Plugin can support parameters. Information about these parameters is communicated back to the Host for it to display in a generic Plugin parameter dialog that is dynamically constructed by the Host based on the parameter (meta) information returned by the Plugin. Even when the Plugin provides its own editor UI the Host still needs to know about these Parameters, for instance to automate them.

A program is a named set of all parameters your Plugin supports. Multiple programs allow the user to make presets with parameter settings and allow him/her to switch quickly between them.

Parameter Meta Information

VST.NET devides a parameter into two parts. There is a VstParameterInfo instance for each parameter in the Plugin. This class contains the parameter meta information that describes the value range of the parameter and its labels. The parameter meta information is also communicated to the host when it calls GetParameterProperties on the Plugin. (The Framework fills the structure this method expects based on the IVstPluginParameters interface and the VstParameterInfo instances along with VstParameterCategory instances.) There is only one VstParameterInfo instance alive for each parameter in the Plugin. The number of VstParameterInfo instances is equal to the number of parameters the plugin supports, it has no relation to the number of programs the Plugin supports.

The other part of a parameter is where its real value is stored. The VstParameter class represents the value-part of a parameter. It references the VstParameterInfo to be able to get to its own meta data. The number of VstParameter instance is a multiplication of the number of parameters of the Plugin, times the number of programs. As said earlier, each program contains a full set of all the Plugin parameters. So each VstParameter in a single program references to a unique VstParameterInfo instance.

So we have multitple VstParameter instances (one in each program) referencing the same VstParameterInfo instance. One of those VstParameter instances has the 'active' or 'current' value for the parameter (the instance that lives in the active/current program). We've introduced another class to manage those VstParameter instance and represent the active/currentvalue. The VstParameterManager is part of VstParameterInfo and is hooked onto each VstParameter instance representing that parameter (definition). The VstParameterManager is (usually) referenced by the Plugin Component (for instance an oscilator) that works with the parameter or the component subscribes to its events.

The following picture shows the multiple (Vst)Paramater instances that all reference the same (Vst)ParameterInfo meta data instance. A (Vst)ParameterManager that keeps tabs on all the (Vst)Parameter instances and communicates the active/current parameter value to the Plugin Component.

Parameter Creation

The idea is that all Plugin (sub) components create the VstParameterInfo instances for their own parameters. The Plugin should 'add these up' to present a complete list to the host through the IVstPluginParameters interface.

The following figure shows the object interaction between a Plugin Component and the VstParameterInfo and VstParameterManager instances. The Plugin Component creates a VstParameterInfo instance for each parameter it requires and populate (setting) its properties. Typically the Plugin Component maintains a reference to the VstPropertyManager it also creates for a specific parameter to be able to work with the active/current parameter value. Note that the VstPropertyManager is part of the parameter meta information on VstParameterInfo.

When the Plugin Component is done with the VstParameterInfo construction for parameters it publishes these instance internally to allow the Plugin to collect all the parameter defintions of all its sub components (oscilators, filters, etc.).

Because the Plugin Component creates both, it has a chance to instantiate a derived type with custom functionality. To register extra parameter meta data, derive a class from VstParameterInfo and add extra properties and to modify the behavior of the VstParameterManager you can also derive a custom class and assign that instance to the ParameterManager property on VstParameterInfo.

The following figure shows the object interaction that takes place when the programs create their (Vst)Parameter instances. Ideally there is a central class that contains the knowledge of all parameters in the Plugin and has a fill method to populate a collection with all VstParameter instances for all parameters.

The VstParameter instance is always constructed on an instance of VstParameterInfo. The VstParameter object will initialize its value with the default value listed in the meta data and allow the VstParameterManager to subscribe to the newly created VstParameter instance.

Program Activation

The following figure shows a situation were 2 programs contain n parameters. For the first parameter (Parameter 1) the (Vst)ParameterInfo and (Vst)ParameterManager objects are displayed. Of course every (Vst)Parameter in a (Vst)Program has its own (Vst)ParameterInfo and (Vst)ParameterManager instance.

But what happens when the user changes programs? The whole set of parameter values should be switched from one program to another. VST.NET defines the IActivatable interface that allows an object to be activated or deactivated. The VstParameter class implements this interface (as does the VstParameterCollection). So when one VstProgram is switched to another VstProgram, all the VstParameter instances are deactivated on the first and activated on the second. The VstParameterManager for each parameter gets notified of this deactivation and activation and is able to grab the new active/current value and notify the Plugin Component.

The following figure displays the object interaction for (de)activation of a (Vst)Program.

The notifacation to the PluginComponent is not shown here but an event is raised each time the active/current parameter value changes on the (Vst)ParameterManager.

Wrapping up

I hope this post has explained to you how the Programs and Parameters work in VST.NET. The seperation between running values of parameters and parameter (meta) definition is a key concept to understand what is going on. The parameter manager takes a lot of work out of your hands and keeps track for you on the active/current parameter value your Plugin Component can work with directly.

Refer to the Jacobi.Vst.Framework.TestPlugin source code for an example of this implementation.

More on VST.NET here.

Tuesday, May 06, 2008

VST.NET

I have recently started an open source project that aims to bring Virtual Studio Technology (VST) - an audio and midi processing Plugin interface standard owned by Steinberg - to the .NET world.

The native VST interface consists of a C++ SDK on top of a C procedural interface. The actual interchange of information between Host (for instance a sequencer application like Cubase) and the Plugin takes place through function pointers using opcodes. An opcode is basically a message identifier that states the action to be taken, either by the Host or the Plugin.

A Plugin is allowed to process digital audio (samples) and midi information and also has the option to parameterize its operation and to provide a custom UI. Most of these features are optional. Plugin capabilities can be queried by the Host as can host capabilities be queried by the Plugin.

On top of this plain C interface definition of function pointers and opcodes an (somewhat) object oriented C++ layer is provided in the Steinberg VST SDK. But this SDK does hardly anything to soften (or structure) the overwhelming number of methods.

VST.NET does nothing with the C++ SDK classes that Steinberg provides. It interfaces at the lowest level with the C function pointers and opcodes and translates those to and from managed code (C# in this case). But besides just providing the marshaling between managed and unmanaged worlds, VST.NET also adds a Framework in an attempt to structure and clarify the posibilities of the VST interface. I have written about this idea in a previous post.

Interop

So how do you interop bewteen C and managed code? Well, Microsoft was kind enough to put managed code features in C++ as well. This means that you can write fully managed code in C++ or mix between native unmanaged C++ and managed C++ in one assembly. It is this ability that lies at the heart of the Interop solution.

The Interop assembly in VST.NET is a C++ mixed assembly. On one side it interfaces with the native C interface using function pointers and opcodes while on the other side it forwards those calls to a managed object after it has converted the native structs to managed structs.

There are two objects involved in marshaling a call from the Host to a Plugin. The PluginCommandProxy is managed a C++ class that has a reference to the PluginCommandStub, a managed implementation of the .NET interface: IVstPluginCommandStub. It is the proxy that takes the calls from the Host and converts the parameters and calls the managed Stub. The IVstPluginCommandStub interface just lists all methods the Plugin could implement (in a versioned hierarchy).

There are also two objects involved in marshaling a call from the Plugin to the Host. The HostCommandStub implements the managed .NET interface IVstHostCommandStub that lists all the methods a Host can implement (also in a versioned hierarchy). The implementation of this interface is passed to the managed Plugin in (at a very early stage) in order to be able to call the Host. All conversion (marshaling) of method parameters takes place in the stub. Wether or not the Plugin implements a proxy is up to the plugin developer.

The Core

Basically all C datatypes (mostly structs and enums) defined in the VST interface have a managed counterpart living in the Core assembly. This assembly is a C# .NET assembly and that is its sole purpose. The Interop assembly references the Core assembly to be able to marshal to (and sometime from) the managed types. The Core assembly also contains the IVstPluginCommandStub and IVstHostCommandStub interface definitions.

It is posible to create a managed Plugin interfacing at the Core level of VST.NET. The plugin would have to implement the IVstPluginCommandStub interface in a public class that will get called by the Interop assembly. But this interface does nothing to structure, group or partition the method available to the Plugin.

When trying to adopt an existing managed code audio or midi processing logic to a VST Plugin, interfacing at the Core level might be the best thing to do. This would produce the least overhead and you would only have to connect the IVstPluginCommandStub methods you need (like I said: a lot of Plugin features are optional) to your existing code.

The Framework

Basically the Framework provides the managed plugin developer with a set of clearly defined interfaces that represent the total set of features available to the Plugin. All plugins must implement the IVstPlugin interface for it contains the capability 'mechanism' to communicate to the Host what features are implemented (more on this later) along with some basic product information.

So if a Plugin wants to process digital audio samples it implements the IVstPluginAudioProcessor interface. If it provides its own GUI it implements IVstPluginEditor. If it supports parameters and programs it implements the IVstPluginParameters and IVstPluginPrograms interfaces (etc.). The Framework defines an interface for each group of functionality the Plugin can implement.

The Plugin root object - that implements IVstPlugin - also implements the IExtensibleObject interface. This interface allows you to do a 'query interface' on the object. Through this mechanism the Plugin can publish what interfaces it supports. So IExtensibleObject.GetInstance<IVstPluginAudioProcessor>() queries for the audio processor interface. If the Plugin returns null it doesn't support audio processing. The reason to introduce this mechanism lies in the desire to be able to dynamically determine the capabilities of a plugin (there is a class of Plugin that requires that feature) but also to communicate these capabilities in coherent interface definitions.

Loading the Managed Plugin

There is one part missing from this story. How does the managed Plugin get loaded and how does the Interop assembly acquire a reference to the implementation of the IVstPluginCommandStub interface.

The Interop assembly exports a 'main' method. This is part of the VST interface definition. The exported main method is called by the Host after the dll has been loaded. The host passes its own function pointer to the main and expects a structure that defines the plugin it is dealing with (this structure holds several function pointers to the Plugin). So the Host thinks the our Interop assembly contains the actual Plugin. But our managed Plugin is located in another asssembly. The Interop main function uses a Loader (located in Core) to find and load the managed plugin assembly and create an instance of the public class that implements the IVstPluginCommandStub interface. There is a naming convention that requires the managed plugin assembly to have a .net postfix to the assembly name in order for the Interop assembly - also renamed - to find it.

Both the Core and Framework assemblies should either be located in the same loaction as the renamed Interop and managed plugin assembly or they can be installed into the GAC which makes sense if you have a number of managed plugins installed on one machine.

Wrapping up

Well, currently I'm still developing VST.NET so some details mentioned here might change. I hope you have some idea how VST.NET works and how you can use it. You can always leave questions at the VST.NET site at codeplex or email me at obiwanjacobi at hotmail dot com.

http://www.codeplex.com/vstnet

Tuesday, December 04, 2007

Midi Device Schema Language - part II

After more than a year of doing other things I picked up the Midi Device Schema Language project again. This post describes the results I got so far.

A short recap: I previously wrote about how I used Xml Schema to describe the content and structure of a Midi System Exclusive Message (SysEx). By describing the structure of the SysEx message allows any Midi application to interpret the content of that message in a logical way. The language describes the logical content of the message using data types that represent the physical structure. This means that for instance a patch name that is the SysEx message is one logical element in the Device Schema (perhaps called PatchName) and is declared with a data type that indicates how many bytes are part of the text string. Or a logical field can be a Boolean that is taken from a specific bit-position in a data byte and multiple logical Boolean's can draw from the same data byte.

The following schema describes test data that I use for unit testing. The Schema declares a simple (Data) type that defines a ProductName of 5 characters long. A complex (Record) type defines the SysEx message from start (F0) to end (F7).

<xs:schema id="TestSchema" xmlns="..." xmlns:m="..." >
<xs:import namespace="urn:midi-device-definition:midiTypes" schemaLocation="MidiTypes.xsd" />
<xs:complexType name="deviceSchemaTest">
<xs:sequence>
<xs:element name="SOX" type="m:midiSysEx" fixed="240" />
<xs:element name="SysExData" type="m:midiSysExData" />
<xs:element name="Bit0" type="m:midiBit0" />
<xs:element name="Bit1" type="m:midiBit1" />
<xs:element name="Bit2" type="m:midiBit2" />
<xs:element name="Bit3" type="m:midiBit3" />
<xs:element name="Bit4" type="m:midiBit4" />
<xs:element name="Bit5" type="m:midiBit5" />
<xs:element name="Bit6" type="m:midiBit6" />
<xs:element name="LowNibble" type="m:midiLSN" />
<xs:element name="HiNibble" type="m:midiMSN" />
<xs:element name="Name" type="ProductName" />
<xs:element name="EOX" type="m:midiSysEx" fixed="247" />
</xs:sequence>
</xs:complexType>
<xs:simpleType name="ProductName">
<xs:restriction base="m:midiString">
<xs:length value="5" />
</xs:restriction>
</xs:simpleType>
</xs:schema>

The fields inside the SysEx message range from single bit values to a multiple bytes value.
The SOX field has a fixed value F0 (240 decimal) and marks the start of the SysEx message.
The SysExData field is a 7-bit SysEx data byte. The data of a SysEx message cannot have its highest bit set (bit 7) as specified in the Midi specifications.
The Bit0 to Bit6 fields all represent one bit of one data byte. Notice Bit7 is not in there.
Then the Low and High Nibble fields each represent 4 bits of data. The Most Significant Nibble (MSN) data type only has 3 bits. The highest bit (bit 7) is always cleared.
Then the Name field contains the product name text string of 5 characters.
Finally the EOX has a fixed value F7 (247) and marks the end of the SysEx message.

When this Schema is used to read this binary test SysEx message (hex):
F0 41 70 0F 4D 49 44 49 31 F7
the following Xml is generated*:

<deviceSchemaTest>
<SOX>240</SOX>
<SysExData>65</SysExData>
<Bit0>False</Bit0>
<Bit1>False</Bit1>
<Bit2>False</Bit2>
<Bit3>False</Bit3>
<Bit4>True</Bit4>
<Bit5>True</Bit5>
<Bit6>True</Bit6>
<LowNibble>15</LowNibbel>
<HiNibble>0</HiNibble>
<Name>MIDI1</Name>
<EOX>247</EOX>
</deviceSchemaTest>

*) Note that the framework does not generate this Xml. It is an application specific interpretation of the logical data the framework publishes. Technically an implementation of the IMidiLogicalWriter interface decides what happens with/to this logical data. The framework calls data typed methods on this interface to publish logical data.

As you compare the binary input and the logical output (Xml in this case) you clearly see that the framework is capable of mapping one physical byte to multiple logical fields (Bit0-Bit6, Low-HiNibble) and mapping multiple physical bytes to one logical field (ProductName).

The knowledge of how to map physical bytes to logical fields (and visa versa) is contained in a Converter. There is a Converter for each Data Type in the language. But we did not have to write a Converter for the ProductName we declared ourself. Because it derives from midiString it uses that Converter (StringConverter). But if we wanted to, we could have registered a custom Converter for that data type, allowing you to handle that case in a special way. The StringConverter looks for a length constraint to know how many characters to read or write to the physical data stream (midiString does not define the length constraint).

To read a binary SysEx message you have to specify the Record Type (deviceSchemaTest) from a Device Schema that you will be reading*. Then the framework will analyze the Fields for the Record Type and the Data Types used by those fields. For each field a Converter is matched and then reading of the binary stream can begin. Each Converter reads the bytes from the binary stream as it sees fit. The BitConverter (Bit0-Bit6) uses BitFlags to indicate which bit it wants to read. The stream position is only advanced to a new data byte when the requested bit(s) are no longer available in the Carry - a one byte data cache.

*) One of the problems that I still need to resolve is how to automatically determine the Schema from a binary data stream. One way to solve this might be to look for specific markers inside the SysEx message content. For example Roland specifies a Command ID in their SysEx messages to indicate what type of message you are dealing with. Other manufacturers may not leave any unique clues to determine the message type.

Thursday, October 04, 2007

Midi Speech Controller Application

I am currently working on the first version of my Midi Speech Controller Application. I am doing the final programming and testing now while some of my (professional) colleagues are helping me with the UI design (thanks guys!). The application is written in C# using the .NET 3.0 framework. It uses WPF (User Interface) and the Speech API. While waiting on the UI design, I'm also looking into building an installer using the Windows Installer Xml (WIX). Unfortunately I am unable to find a redistributable for the Microsoft Speech technologies. I may end up advising the users of my application to install the Speech SDK…

The application runs in two modes: Run-mode and Edit-mode. In Edit-mode the user can define a Preset with one or more Patches. A Patch is collection of Midi Commands that are executed in order when the Patch is activated. A patch can be activated by speaking its text phrase out loud (into a microphone that is connected to the PC) –or- if the speech engine is not installed, by selecting the patch from a list and activating it manually through a button click.

In Run-mode the application can be made to listen by clicking on the Start button. All text phrases for all Patches for the current selected Preset are registered with the speech recognition engine. When the user speaks out a phrase for a Patch, the speech is recognized and displayed to the user, the Patch is located and its Midi Commands are executed.

Preset
The Preset represents one set of Patches that can be active at one time. There can be many Presets, but only one is active. Presets (with their Patches and Midi Commands) can be save to- and loaded from disk. A Preset also contains two options for audio feedback. The options for successful or unsuccessful speech recognition are: None, Beep and Speak. So when a text phrase for a Patch is successfully recognized the computer can say the Patch name back to you – or just beep – or do nothing. When the Speak option is selected for the unsuccessful recognition option, "Not Recognized" is spoken.

Patch
The Patch represents a set of Midi Commands that will all be executed when the Patch is activated. The Patch has a (human readable) name and a (pronounceable) text phrase associated with it. Both can be set to any text, although it is recommended to keep the text phrase as short as possible.

Midi Command
A Midi Command contains the definition of a valid Midi message that is sent to a specific Midi Port (and channel) on the PC. A Program Change, a Control Change or a System Exclusive message can be defined. The data for the System Exclusive message has to be hand-typed into a text field (for now), including the Start and End markers (F0 and F7 resp.).

The application is targeted at users that literally have their hands full or just need a simple solution to output a large number of Midi messages at the click of a button (when the speech engine is not installed). People that do live performances can benefit from this application. Imagine speaking the title of the song you're about to play and all you equipment being initialized for that song.

I came about the idea for this application when I was playing my (electronic) V-Drums and wanted to change kits. With no hands and feet to spare, speech seemed the only option.

Tuesday, September 04, 2007

Redesigning the Steinberg VST SDK

This post proposes a new structure for the Steinberg VST SDK.

Steinberg's Virtual Studio Technology (VST) represents a plugin interface that allow third-party plugins to run inside a host, for instance Cubase. There are a lot of hosts that support the VST interface and there are a lot of plugin's these days.

Steinberg provides a C++ SDK in two parts. The Interface (files) declare the common ground between the host and the plugin. Then there are a couple of C++ classes that are built on top of these interfaces and allow a custom plugin to derive from an SDK-provide base class. We're not going to talk about the GUI classes here.

The Interface cannot be changed for obvious reasons. The structure of the base classes provided by the SDK is very simplistic. They just defined all functionality on one base class -assuming a plugin would consist of one root class were all features are implemented. Also, there are some methods on the base class that should be called in the constructor and nowhere else.

The VST communication between host and plug-in is basically done through callback procedures (much like a window procedure) although the plugin provides a structure with some additional information at start-up.

While I was looking into how to interop the VST plug-in Interface (version 2.4) with the managed .NET framework I also started to sketch out a new structure for the managed VST interface(s) (yes, I know there is a managed VST implementation called noise).

There were several design principles that I used:
  • Allow the plug-in design to be more flexible
    Don't force everything to be in one (base) class.
  • Group related functionality together
    Use the single responsibility principle.
  • Communicate capabilities (canDo's) via implemented interfaces
    Define interfaces for canDo's and their associated methods. Other capabilities are communicated through a property.

The Plug-in
The functionality of the plug-in is devided in several interfaces. These interfaces are implemented by the plug-in and called by the interop framework.

IVstPlugin
This interface exposes basic plugin functionality without regard to the actual purpose of the plugin (synth, effect or midi processor).

  • Open / Close
    Opens and Closes the Plug-in. Basically to allocate and clean-up resources (memory).
  • Suspend / Resume
    Suspends and Resumes plug-in processing.
  • Parameters
    A structured Parameter (collection) exposing the plug-in's parameters.

IVstAudioProcessor (IVstAudioPrecissionProcessor)
The audio processor interface is implemented by the plug-in that has Audio capabilities. The precission processor receives audio channels that represent doubles in stead of singles (float).

  • BlockSize / SampleRate
    These properties are set by the host (indirectly) and used in audio processing.
  • Process
    The Process method takes an IAudioChannel collection for both input channels and output channels.

IVstMidiProcessor
The Midi processor interface is implemented by the plug-in that processes incoming Midi events. Outgoing Midi events are implemented at the host throught this interface.

  • Process
    The process method takes a VstEventsCollection that contain VstMidiEvent instances.

IVstPluginBuilder
The plug-in builder interface exposes the different aspects if the plug-in. It is up to the implementation of the builder (and the plug-in) to decide how these request are resolved.

  • GetPlugIn
    Returns the IVstPlugIn implementation (not null).
  • GetAudioProcessor / GetMidiProcessor
    Returns the IVstAudioProcessor and the IVstMidiProcessor implementations (or null).
  • GetPlugInEditor
    Returns the custom editor (interface not discussed) for the plug-in (or null).

The Host
The host functionality is seperated into several interfaces (as well). These interfaces are implemented by the interop framework and can be called by the plug-in.

IVstHost
The main interface representing the VST host.

  • Product / Vendor / Version
    Information about the host implementation.
  • Sequencer
    Provides access to an implementation of the IVstHostSequencer interface.
  • Shell
    Provides access to an implementation of the IVstHostShell interface.

IVstHostSequencer
The sequencer interface provides access to the host's sequencing functionality.

  • Capabilities / Input- & OutputLatency
    Provides information on the capabilities of the host.
  • GetTime
    Retrieves the current time (song pointer).
  • UpdatePlugInIO
    Indicates to the host that the plug-in has changed it's IO channels.

IVstHostShell
The shell interface provides access to the host's UI.

  • BaseDirectory
    Installation directory of the host.
  • Culture
    Host language information.
  • EditParameter
    Launches the edit parameter screen.
  • OpenFileSelector
    Launches the open file dialog.
  • SizeWindow / UpdateDisplay
    Resizes and updates the host's main window.

Feedback please
I know not all features of the VST SDK (version 2.4) are covered here but the general idea should be explained. I would really like to hear your opinion on this way of structuring the VST functionality.

Thursday, May 31, 2007

Online Book for audio (programming) content

Basically its just a bookmark list pointing to the audio related information sources on the internet. For now I've just 'indexed' (part of) Jeff Glatt's site and midi.org. Also started on some samples and libraries.

Get it here.

If you know any good links, post them here.

Saturday, May 19, 2007

Release of Midi Mapper and SysEx Utility.

The initial versions of the Midi Mapper and the Midi SysEx Utility are available for download.

Both applications are .NET 2.0 (WinForms) applications and have no other dependencies.

Contact me for bug reports, questions or feature requests at obiwanjacobi at hotmail dot com.

Have fun.

2007-05-22: updated the binaries.

Friday, May 04, 2007

Raising Events or Exceptions?

This post discusses the issues involved in raising events in a .NET class for a multi threaded application and a problem with the widely accepted solution.

When you write your own class that supports events, you also have to write the code that raises these events. A typical way to structure that code is in a protected virtual On[EventName] method and most implementations look something like this:

public event EventHandler Changed;

protected virtual void OnChanged()
{
if (Changed != null)
Changed(this, EventArgs.Empty);
}

But there is a race condition in this code. In a multi threaded application the last handler could be removed exactly between the null-check and the actual call to raise the event. In that specific case a NullReferenceException will be thrown because Changed is null.

One solution to this problem is to copy the (potential) last reference to the subscribed event handler into a seperate local variable. Something like this:

protected virtual void OnChanged()
{
EventHandler handler = Changed;
if (handler != null)
{
handler(this, EventArgs.Empty);
}
}

You capture that (potential) last event handler and test if it is not null. If that check passes, you raise the event. Your code works and no NullReferenceExceptions will be thrown. This is the typical solution to the problem you see suggested by many as the only right way to do it. But I see some problems with it.


The proposed solution does not take into account how this sequence of events impacts the event subscriber. To the event subscriber it is the worst code you can have! Why? Because to the subscriber an event handler that has just been removed from an event can still be called. You don't expect that as a subscriber and you shouldn't have to. This behavior of the event provider is counter-intuitive and can lead to hard (or imposible) to find bugs.

I'm not sure what the correct solution should be. I just wanted to point out the side effect of the solution that 'everybody' holds for the correct solution. For inspiration you should check out this blog post by Roy Osherove. He discusses defensive techniques for raising events.

Maybe something as simple as putting a try-catch block in the OnChanged method is good enough? Make sure you publish those exceptions, though. You'd still want to know about them.

Thursday, May 03, 2007

Robust P/Invoke for the Windows MIDI API

In my spare time I'm writing a MIDI program. I found that working with the Windows midiXxxx (multi media) API using P/Invoke can be a little complex. In this post I will present a solution to robustly interface with the midiXxxx API.

When opening a midi in or out port (device) you can pass a delegate that will be called when an event takes place on the port. Events such as receiving midi and opening and closing a port. This delegate instance should be kept alive (not Gargabe Collected) during the time the port is open and the midi function can call back on it.

Most solutions you see is that the instance of the callback delegate is stored as a member in the port class itself. This ties the lifetime of the delegate to the lifetime of the port class. This seems like a reasonable solution, right?

But what if the client of your port class doesn't play by the rules? Suppose the application that uses your port class forgets to close (or dispose) the port class. As soon as the client releases all references to the port class it will be elligable for garbage collection and so is the callback delegate. When the midiXxx API now tries to callback on the delegate you provide during the Open call, you'll get an exception that says someone was trying to call a garbage collected delegate.

Microsoft has added a new feature in the .NET 2.0 framework called SafeHandles. A safe handle is like an object wrapper around a native handle and is guarenteed to finalize (normal classes are not). The SafeHandle also provides protection against some security attacks (recycling of handles). The native midi handle that is returned from a midi-open call is usually stored as Int32 or IntPtr inside the port class. If a client forgets to call the close method on your port class the port stays open and cannot be used again. All the (managed) code that has access to the (native) port is collected eventually and your app starts to misbehave.

So lets walk through the solution for both these problems: keeping the delegate alive and ensuring the midi handles are closed.

My solution for keeping the delegate alive is simple. We cache one instance in a static variable. Static variables pretty much live untill the AppDomain dies. This introduces a new problem though: Multiple instances of the port class are now called back on the same delegate. Luckily we can supply 'user data' that is passed with each call to the callback function. We pass in the this reference of the port class itself.

Lets get to the code. First we declare the P/Invoke methods for the MIDI out port.

static class NativeMethods
{
public const int CallbackFunction = 0x30000;

public delegate void MidiOutCallback(IntPtr midiHandle, int message,
IntPtr userData, int param1, int param2);

[DllImport("winmm.dll")]
public static extern int midiOutClose(IntPtr midiHandle);

[DllImport("winmm.dll")]
public static extern int midiOutOpen(out MidiSafeHandle midiHandle, int deviceId, MidiOutCallback callback, IntPtr userData, int flags);
}

Notice that I specify the MidiSafeHandle type to be returned from the midiOutOpen function. The P/Invoke plumbing knows about SafeHandles and will create one. The MidiSafeHandle is derived from the Microsoft.Win32.SafeHandles.SafeHandleZeroOrMinusOneIsInvalid base class. You should have seperate SafeHandle implementations for the different flavors of midi handles.

class MidiSafeHandle : SafeHandleZeroOrMinusOneIsInvalid
{
public MidiSafeHandle()
: base(true)
{ }

protected override bool ReleaseHandle()
{
bool success = (NativeMethods.midiOutClose(this.handle) == 0);
handle = IntPtr.Zero;
return success;

}
}

Notice that the native midi handle is closed in the override of the ReleaseHandle method.
Next lets look at the MidiOutPort class. The first part is shown below:

class MidiOutPort : IDisposable
{
private GCHandle _gcHandle;
private MidiSafeHandle _midiHandle;

public MidiOutPort()
{
_gcHandle = GCHandle.Alloc(this, GCHandleType.Weak);
}

public void Open(int device)
{
int result = NativeMethods.midiOutOpen(
out _midiHandle, device, _callback, GCHandle.ToIntPtr(_gcHandle),
NativeMethods.CallbackFunction);
}

public void Close()
{
if (_midiHandle != null)
{
_midiHandle.Close();
}
}


The constructor allocates a GCHandle that contains a weak reference to our MidiOurPort instance. This GCHandle is used to pass a reference to our instance to the native midiOutOpen method that calls our delegate with it. A weak reference will not keep the MidiOutPort instance alive, so we do not interfere with garbage collection and the lifetime of the object.

The Open method opens the midi out port and returns a valid MidiSafeHandle on success. Notice we pass in the static delegate reference (_callback) and a reference to our own instance (_gcHandle). Opening the port will call the callback with an MOM_OPEN message.

The Close method just closes the MidiSafeHandle which in turn will use the midiOutClose API to actually close the device. Closing the port will call the callback with an MOM_CLOSE message.

private void OnMessage(int message, int param1, int param2)
{
Console.WriteLine(String.Format("OnMessage Message:{0} Param1:{1} Param2:{2}",
message, param1, param2), "MidiOutPort");
}

private static NativeMethods.MidiOutCallback _callback = OnCallback;

private static void OnCallback(IntPtr midiInHandle, int message, IntPtr userData, int param1, int param2)
{
try
{
GCHandle handle = GCHandle.FromIntPtr(userData);

if (handle.Target != null)
{
((MidiOutPort)handle.Target).OnMessage(message, param1, param2);
}
}
catch (Exception e)
{
// TODO: handle exception
Console.WriteLine(e);
}
}
}

We see the static _callback member being initialized with a delegate to the static OnCallback method. This method will receive all callbacks for all open midi (out) ports. It uses the userData to retrieve a reference to our GCHandle we created in the constructor of our class. Then it examines the Target property that should contain the reference to our MidiOutPort class unless it has been collected. If its not collected it calls the OnMessage (instance) method with the message data (message, param1 and param2). Now we're back in our MidiOutPort instance and we can do the message processing.

Notice that the MidiOutPort class does not implement a finalizer! It does implement IDisposable (not shown in code) but relies on the MidiSafeHandle to clean up the unmanaged resources.

To try our code simply instantiate the MidiOutPort class and call Open and Close. Build this into a console application so you can see the Console.Write calls.

MidiOutPort midiPort = new MidiOutPort();
midiPort.Open(0);
midiPort.Close();

Now comment out the call to the Close method. You will see that upon exiting the process, the MidiSafeHandle does its job and closes the handle, which causes a callback, which might or might not reach the original MidiOutPort instance (depending if the instance is already garbage collected).

This solutions is the most stable solution to this problem I managed to come up with so far. The solution is applicable to any P/Invoke interop using callbacks, I just used the Midi Out Port as a concrete example.

If you have any thoughts, suggestions or problems trying it out for yourself, please leave a comment.

Wednesday, May 02, 2007

Design Patterns [Links]

I collected some links to design patterns (mostly fowler and dofactory).
I hope you find them usefull.

http://www.martinfowler.com/eaaDev/index.html
http://www.dofactory.com/Patterns/Patterns.aspx
http://developer.yahoo.com/ypatterns/ (UI design patterns)
http://msdn2.microsoft.com/en-us/library/aa137892.aspx (Guides)

GUI Patterns
http://www.martinfowler.com/eaaDev/OrganizingPresentations.html

Separated Presentation
http://www.martinfowler.com/eaaDev/SeparatedPresentation.html
Passive View
http://www.martinfowler.com/eaaDev/PassiveScreen.html
Supervising Controller
http://www.martinfowler.com/eaaDev/SupervisingPresenter.html
Presentation Model
http://www.martinfowler.com/eaaDev/PresentationModel.html
Flow View Synchronization
http://www.martinfowler.com/eaaDev/FlowSynchronization.html
Observer View Synchronization
http://www.martinfowler.com/eaaDev/MediatedSynchronization.html
Event Aggregator
http://www.martinfowler.com/eaaDev/EventAggregator.html
Presentation Chooser
http://www.martinfowler.com/eaaDev/PresentationChooser.html

MVC / MVP [obsolete?]
http://www.martinfowler.com/eaaDev/uiArchs.html


Data

Data Patterns
http://msdn2.microsoft.com/en-us/library/ms998446.aspx

Notification
http://www.martinfowler.com/eaaDev/Notification.html
Range
http://www.martinfowler.com/eaaDev/Range.html
Special Case
http://martinfowler.com/eaaCatalog/specialCase.html
Null Object
?
Parallel Model
http://www.martinfowler.com/eaaDev/AgreementDispatcher.html


Events
http://www.martinfowler.com/eaaDev/EventNarrative.html

Domain Event
http://www.martinfowler.com/eaaDev/DomainEvent.html
Event Collaboration
http://www.martinfowler.com/eaaDev/EventCollaboration.html
Event Sourcing
http://www.martinfowler.com/eaaDev/EventSourcing.html
Agreement Dispatcher
http://www.martinfowler.com/eaaDev/AgreementDispatcher.html
Retroactive Event
http://www.martinfowler.com/eaaDev/RetroactiveEvent.html


Temporal patterns
http://www.martinfowler.com/eaaDev/timeNarrative.html

Audit Log
http://www.martinfowler.com/eaaDev/AuditLog.html
Time Point
http://www.martinfowler.com/eaaDev/TimePoint.html
Effectivity
http://www.martinfowler.com/eaaDev/Effectivity.html
Temporal Object
http://www.martinfowler.com/eaaDev/TemporalObject.html
Temporal Property
http://www.martinfowler.com/eaaDev/TemporalProperty.html
Snapshot
http://www.martinfowler.com/eaaDev/Snapshot.html

Creational Patterns

Abstract Factory
http://www.dofactory.com/Patterns/PatternAbstract.aspx
Builder
http://www.dofactory.com/Patterns/PatternBuilder.aspx
Factory Method
http://www.dofactory.com/Patterns/PatternFactory.aspx
Prototype
http://www.dofactory.com/Patterns/PatternPrototype.aspx
Singleton
http://www.dofactory.com/Patterns/PatternSingleton.aspx

Structural Patterns

Adapter
http://www.dofactory.com/Patterns/PatternAdapter.aspx
Bridge
http://www.dofactory.com/Patterns/PatternBridge.aspx
Composite
http://www.dofactory.com/Patterns/PatternComposite.aspx
Decorator
http://www.dofactory.com/Patterns/PatternDecorator.aspx
Façade
http://www.dofactory.com/Patterns/PatternFacade.aspx
Flyweight
http://www.dofactory.com/Patterns/PatternFlyweight.aspx
Proxy
http://www.dofactory.com/Patterns/PatternProxy.aspx

Behavioral Patterns

Chain of Responsibility
http://www.dofactory.com/Patterns/PatternChain.aspx
Command
http://www.dofactory.com/Patterns/PatternCommand.aspx
Interpreter
http://www.dofactory.com/Patterns/PatternInterpreter.aspx
Iterator
http://www.dofactory.com/Patterns/PatternIterator.aspx
Mediator
http://www.dofactory.com/Patterns/PatternMediator.aspx
Memento
http://www.dofactory.com/Patterns/PatternMemento.aspx
Observer
http://www.dofactory.com/Patterns/PatternObserver.aspx
State
http://www.dofactory.com/Patterns/PatternState.aspx
Strategy
http://www.dofactory.com/Patterns/PatternStrategy.aspx
Template Method
http://www.dofactory.com/Patterns/PatternTemplate.aspx
Visitor
http://www.dofactory.com/Patterns/PatternVisitor.aspx

Dependency Injection
http://martinfowler.com/articles/injection.html

Sunday, March 25, 2007

My First DSL: Object Model Designer

To learn and test out the DSL tools that come with the Visual Studio 2005 SDK, I've created a simple DSL to quickly create object models. The goal for this DSL was to have a 'class designer' that focused on object structure and not distract you with details.

You can create a new project using the Object Model Designer project template (its in C# Projects). The following image shows the project template for the Object Model Designer.



Each time you change your model, you have to regenerate the code (that represents that model) using the right-most button on the Solution Explorer or by selecting "Run Custom Tool" from the context menu of the MyModel.Designer.tt. This will generate the MyModel.Designer.cs file containing code for all classes in the model. The MyModel.cs is a demo implementation of how you would extend your model in code. All classes are generated as partial so you can regenerate your model anytime and might get compile errors for custom code that no longer 'connects' to the model code.

A MyModel.Designer.tt file is needed for every model (instance) you create in the project because the model is referenced in the .tt file. The actual .tt script is included from ObjectModelDesigner.tt.

You can download the installer for the Object Model Designer
here.

Saturday, February 10, 2007

Mini Pattern: The Guard Method

This post proposes a standard way to implement and call method parameter validation logic.

Instead of writing out the method parameter validation code in each method over and over again we will centralize it in a static helper class. This static helper class will be reusable over different projects and will speed up writing method parameter validation code.

This is the way most of us write method parameter validation code:

public void MyPublicMethod(string name)
{
if(String.IsNullOrEmpty(name))
{
throw new ArgumentNullException("name");
}

// method impl...

}

Instead we now call out static helper class named Throw:

public void MyPublicMethod(string name)
{
Throw.IfArgumentNullOrEmpty(name, "name");

// method impl...
}

With a central place to write parameter validation code you can make it a little more elaborate than you would if you'd had to write it in each method seperately. Notice the implementation make a distinction between is null and is empty.

public static void IfArgumentNullOrEmpty(
string argument, string argumentName)
{
if (String.IsNullOrEmpty(argument))
{
IfArgumentNull(argument, argumentName);

throw new ArgumentException("Argument cannot be empty.", argumentName);
}
}

There is one drawback in using this method of performing method parameter validation. FxCop does not recognize that you are performing validation on these parameters and will insist that you do. I hope that the FxCop (Code Analysis) team at Microsoft will come up with a code attribute we can use to decorate our "method parameter validation" methods.

You'll notice this way of implementing validation logic does not work well with The Throw Method. I personally think that method parameter validation exceptions require less or no context information other than perhaps the parameter value itself.

Mini Pattern: The Throw Method

This post proposes a standard way to throw exceptions from a class and fill the Exception.Data dictionary with the class' context information.

.NET 2.0 introduced a Data dictionary on the Exception class. This dictionary is ideal for holding context information at the place were the Exception is thrown. In order to be able to reuse the logic across all methods of the class and still use the single line syntax "throw new Exception();" a private Throw method is added to the class.

private void Throw(Exception exception)
{
exception.Data.Add("Name", this.Name);
exception.Data.Add("ID", this.ID);
exception.Data.Add("CreationDate", this.CreationDate);
exception.Data.Add("UserName", this.UserName);

throw exception;
}

The Throw method takes the state of the class and stores it in the Data dictionary of the Exception about to be thrown. This data represents (some of) the context in which the exception occurred.

Use the Throw method in other methods of your class whenever throwing an exception is appropriate.

public void MyPublicMethod(string name)
{
if(String.IsNullOrEmpty(name))
{
Throw(new ArgumentNullException("name"));
}

// method impl.
}

Make sure that you write out the extra information contained in the Exception.Data dictionary to the event log (or other log target) when you catch the exception. It should provide you with a little bit of runtime information to help you hunt down any bugs.

Wednesday, December 20, 2006

Two layers of disposability

This post discusses implementing the IDisposable interface.

Microsoft has a guideline on how to implement the IDisposable interface. In my opinion it's not a the whole picture of the solution. It suggests you implement a Finalizer, which in many cases you don't need at all.

In my experience I've encountered two scenarios where you would want to implement the IDisposable interface on your custom class:
  1. The class contains other instances of classes that implement IDisposable.
  2. The class owns unmanaged resources (COM references and Win32 handles).

Both of these scenarios can apply to your custom class ofcourse.

My point is that in scenario 1 you don't need the Finalizer and therefor you should not implement it. Do follow the virtual method Dispose(bool) pattern at all times though.

The following code snippet shows a basic base class implementation (notice the absense of a Finalizer):

public abstract class DisposableBase : IDisposable
{
protected virtual void Dispose(bool disposing)
{
// implement here...
}

public void Dispose()
{
Dispose(true);
}
}

I personally prefer to implement a protected IsDisposed property and a protected ThrowIfDisposed helper method at this level too but thats not relevant to our discussion.

The 2nd scenario would require the finalizer to make sure the unmanaged resources are released properly in all situations. The following code snippet does just that:

public abstract class UnmanagedDisposableBase : DisposableBase
{
protected override Dispose(bool disposing)
{
base.Dispose(disposing);

// only suppress calling finalizers when called from Dispose()
if(disposing) GC.SuppressFinalizer(this);
}

~UnmanagedDisposableBase
{
Dispose(false);

}

}

Note that error handling has been left out. The code only intents to illustrate my point.

Any derived class (either from DisposableBase or UnmanagedDisposableBase) would simply override the Dispose(bool) method -and call its base!- to dispose its internal resources (following the suggested Microsoft pattern).

Thursday, December 14, 2006

Using Xml Schema for describing Midi System Exclusive message content [repost]

[repost from http://bloggingabout.net/blogs/marc/archive/2006/09/25/Using-Xml-Schema-for-describing-Midi-System-Exclusive-message-content.aspx]

In my spare time I'm writing a Midi Console application. This application can manage all the settings of all the midi devices in a studio (Samplers, Sound modules, Drum machines etc.). The onboard user interface of most midi devices are poor (at best) and having an application that allows you to manage settings for multiple devices in a consise way would improve productivity of the mucisian.
Most midi devices support what's called System Exclusive messages. The content of these messages are not standardized by the
MMA but can be freely used by any manufacturer for its own purposes. A typical way to get to all the settings of a midi device is through using these System Exclusive messages.

The way applications dealt with these device specific messages in the past, was to write specific drivers for each device or -at best- have some sort of reference table where each settings was located in the System Exclusive message. This meant that any application targeting System Exclusive message would support a fixed set of midi devices. If your device is not on the list, you could not use the application.

After studying the different binary content layout of System Exclusive message for several manufacturers, it occurred to me that they could be described in a meta language which could than be used to handle the interpretation and compilation of these device specific messages. I decided to use (or abuse most would say) Xml Schema (xsd) to describe the content of these messages.

I'm currently writing a Midi Device Schema Specifications document that describes how one would use Xml Schema for Midi System Exclusive messages. For those who are interested: I post new versions to this thread in the mididev newsgroup and the latest version of the specifications can be found here.

Any feedback is most welcome on any aspect of the specifications or the solution in general.

[BizTalk] How to enrich a message [repost]

[repost from http://bloggingabout.net/blogs/marc/archive/2006/05/30/12265.aspx]

While working on my first BizTalk 2006 project I came across the need to enrich messages as they were routed through BizTalk. I knew that a Map would be the logical approach because most fields could be copied through directly. Based on some other properties a database lookup should be performed and the resulting data structure is to be added to the destination message. The structure of this data is complex; an hierarchy with multiple, repeating elements.

After trying all sorts of things covering Database Lookup, and all sorts of Scripting functoids configurations and searching the net, I finally came across this post. http://www.infusionblogs.com/blogs/syd/archive/2006/05/17/480.aspx. It nearly describes the problem I tried to solve and sure enough, within 5 minutes I had a working solution.

I made a custom .NET assembly with a class that has a public method to lookup the extra information based on the source message properties. This method takes several (in my case) string parameters and returns a string that contains the Xml sub structure thats to be included in the destination message. Note that the string returned is the OuterXml of the DocumentElement of the XmlDocument. If you return the OuterXml of the XmlDocument you'll include a <?xml?> PI and the map doesn't like that (with good reason).

public class DataLookup
{
public string Lookup(string param1, string param2)
{
XmlDocument xmlDoc = new XmlDocument();
// load xml document...
return xmlDoc.DocumentElement.OuterXml;
}
}
Then I created a map in VS.NET, selected the source and destination schemas and placed two Scripting Functoids on the map-surface. The first scripting functoid has its inputs connected to the required input properties from the source schema and the output is connected to the second scripting functoid and is configured to call out to my .NET assembly. The second scripting functoid is configured as an Xslt call template (code is shown below) and its output connects to the root element in the destination schema where the additional information has to go.

So, during a transformation, the source properties are fed into my custom class' method, a database lookup is performed and an xml document is created which is returned as a string. This Xml as string is then passed into the xslt named template and injected into the destination message at the specified root element.

Source Properties => => Destination Xml

It seems a bit strange that you have to convert Xml into a string (to be convered into Xml again by the Xslt template) in an xml technology-based BizTalk map. But hey, it works...


Singleton Generics [repost]

[repost from http://bloggingabout.net/blogs/marc/archive/2006/02/13/11034.aspx]

The Singleton pattern is probably the most famous pattern of all. Usually it is implemented as a behaviour of a specific class. But why not let the developer decide how to manage instance lifetimes? The new .NET 2.0 Generics feature gives us just the tools for creating these object lifetime classes.

public static class StaticInstance
where T : new()
{
private static T _instance;
private static object _lock = new object();

public static T Current
{
get
{
if (_instance == null)
{
lock (_lock)
{
if (_instance == null)
{
_instance = new T();
}
}
}

return _instance;
}
}
}

This code manages one instance of Type T in a (AppDomain) static variable, your typical Singleton implementation. Any class can be used as a Singleton now, just call StaticInstance.Current to access the instance of your type 'MyClass'. Beware though that being a Singleton instance has concurrency issues in that multiple threads could access that one instance of your class at the same time.

In an ASP.NET context you often have the need to have "static" information available but private to the current request. Well, simply write another instance manager class such as this one:

public static class HttpContextInstance
where T : new()
{
private static string _typeName = typeof(T).FullName;

public static T Current
{
get
{
Debug.Assert(HttpContext.Current != null);

T instance = (T)HttpContext.Current.Items[_typeName];

if (instance == null)
{
instance = new T();
HttpContext.Current.Items[_typeName] = instance;
}

return instance;
}
}

public static void Dispose()
{
IDisposable instance = HttpContext.Current.Items[_typeName] as IDisposable;

if (instance != null)
{
instance.Dispose();
}

HttpContext.Current.Items[_typeName] = null;
}
}

The instance is stored in the Items collection of the HttpContext, thus making the instance private to just the current web request. I've also included a Dispose method to dispose of the instance's resources when the request is done (global.asax) and clear the slot in the HttpContext items collection. You could think of other implementations for storing instances in the Thread Local Storage, the logical CallContext or any other place that might be convienient to you.

Have fun,
Marc Jacobi


[UPDATE 14-feb-06]

I like to point out some of the problems that you may encounter using this approach. The following issues should be taken into account:

  1. A Type specified for T must be able to cope with the concurrency consequences of the instance class implementation. For the StaticInstance example this means that it should syncronize access to its member variables.
  2. The Type (T) must have a public default constructor and your team could use that default constructor to create their own instances. For some types this is not a real big issue for others it can introduce hard-to-track-down bugs. If your Type (T) is not designed to be instantiated more than once implement your own Current property and remove your (default) constructor(s).
  3. All team members should "know" what Type (T) is accessed by which instance class. If one member uses StaticInstance.Current and another uses HttpContextInstance.Current you'll have 2 instances living two different lifetimes. This is a weakness that can be overcome and we will discuss next.

Because C# (generics) does not support the typedef keyword (C++: allows defining a new type using other types declaratively) the only way to simplify and hardwire a generics type is to derive from it. So if you use the following code template for instance class implementations you can fix issue 3 by deriving a new type.

public class StaticInstance
where T : new()
{
private StaticInstance()
{}

public static T Current
{
get
{
// [your implementation here]
}
}
}

Now, say you use a static instance of MyClass in your application you can derive a new type to hardwire the T parameter. This also gives you one point of definition for the MyClass-singleton and makes it easy to transparently change the instance class backing up the singleton.

public class MyClassSingleton : StaticInstance
{}

I hope this update gives you a better overview of the consequenses of using this approach.
Keep the questions and suggestions coming.

Greetings,
Marc Jacobi

DataSet Manager [repost]

[repost from http://bloggingabout.net/blogs/marc/archive/2005/11/30/10406.aspx]

Why are OR-mappers cool? I dont know? My experience with them has been limited and the time I did use them the support for common constructs was very poor (at best): I don't think its a good idea for a OR-mapper to caugh up a SQL select query for each instance that needs to go in a collection. The framework should be able to retrieve any hierarchy of data in one round trip (not counting the sceanrio where you have to decide how big the data packet may get versus how many roundtrips are optimal). Also I believe the OR-Mapper problem domain is two fold: 1) you need a good data access layer that supports all the features the required for a "good" OR-mapper and 2) you need the OR-mapping itself, map relational data to objects and back, which is a relatively simple problem to solve.

So I started thinking about the data access layer. I'm a big fan of typed DataSets. If you know your VS.NET you can whip up a typed DataSet within 5 minutes. My assumtion is that the data access layer works with (typed) DataSets. Also I propose to put your application data model in one type DataSet. For instance, You put the complete datamodel of Northwind into one typed "Nortwind" DataSet. If you have a really big solution you could work with subsets for each sub-system.

Now I want to be able to express my data queries in the same 'entities' defined in my application typed DataSet (Northwind). Why would an application programmer have to express his data requests in a different "language" than his application data model? Now we need an entry point that will handle our data requests and knows about our application data model. Enter the DataSet Manager.

The way the DataSet Manager retrieves data is by using the structure defined in the (typed) DataSet. It needs a copy of this structure as its data model. For now we assume that this application model reflects the physical data model in the database. A "Real" mapper would allow a certain abstraction here, allowing your application business entities to be subsets (or combined) database entities. The DataSet manager would have (overloaded) methods to fecth data for a (running) typed application DataSet. For instance "Fill up the (Northwind.)Empoyees table", "Fetch all orders handled by this Employee", make changes to the dataset and save.

In the following code examples we assume we have a typed DataSet named Northwind with the complete Northwind database schema.

// create the DataSetManager and initialize the DataModel
DataSetManager mgr = new DataSetManager();
mgr.CreateDataModel(new Northwind(), "Initial Catalog=Northwind");

This code creates a new DataSetManager and initializes the instance with the application data model to use and a connection string to the physical database. Inside the DataSetManager the schema of the (typed) DataSet is analysed and the DataSetManager creates DataAdapters (in this prototype) for each DataTable. The DataSetManager is ready.

// load all Employees data into an empty dataset
Northwind dataSet = new
Northwind();
mgr.Fill(dataSet, dataSet.Employees);

Notice the second Northwind DataSet instance. The first was passed to the DataSetManager as a schema, this one is used (by the application programmer) for actually storing data. We ask the DataSetManager to fill up the Employees table and pass it the running DataSet and a reference to the table definition. Because we use typed DataSets both are contained in one instance (thats what makes a DataSet typed). All methods of the DataSetManager take a DataSet as a first parameter. This allows for seperation of data holder and data definition. The DataSetManager will build a "Select * from Employees" query for this method call and writes the results back to the (running) DataSet.

But wait a minute. If EnforceConstraints is set to true this won't work. The lack of data in the other tables the Employee table has a DataRelation with will cause the contraints to fail. Not quite so. The DataSetManager knows about the schema and therefor knows about these relations too. It examines the content of the passed dataset and dissables these contraints that 'point' to empty tables. If you pass in a dataset with its EnforceConstraints set to false, the DataSetManager does nothing.

// find all Orders for the first Employee
Northwind.EmployeesRow employee = dataSet.Employees[0];
mgr.Expand(dataSet, employee, DataSetManager.FindRelation(dataSet.Employees, dataSet.Orders));

We retrieve a reference to an Employee and ask the DataSetManager to expand for this Employee instance (DataRow) using the relation between Employees and Orders. We use a helper method to find the DataRelation between these tables. Again the order data concerning the employee is placed in the passed dataset.

// change some data
employee.Notes = String.Format("Handled {0} orders.", dataSet.Orders.Count);
// update all changes made to the dataset
mgr.Update(dataSet);

Now we change some data in the dataset (on the employee) and ask the DataSetManager to persist the changes back to the database. Using the standard DataSet.GetChanges method and using a DataAdapter the empoyee is updated in the database.

These are the method (overloads) the DataSetManager supports:
public DataSet DataModel{get;}
public void CreateDataModel(DataSet dataModel, string
connectionString)
public int
Fill(DataSet dataSet)
public int
Fill(DataSet dataSet, DataTable table)
public int
Fill(DataSet dataSet, DataRelation relation)
public int
Expand(DataSet dataSet, DataTable table)
public int
Expand(DataSet dataSet, DataRow row)
public int
Expand(DataSet dataSet, DataRelation relation)
public int
Expand(DataSet dataSet, DataRow row, DataRelation relation)
public int
Update(DataSet dataSet)
public int Update(DataTable table)

This prototype was written using .NET 1.1 and therefor no generics are used. But in a future version this would certainly be an option for added type safety. One thing thats missing from this prototype is where-clauses. This is one of the problems i'm still wresteling with. How would you express filter criteria using application entities? I've considered Query by example but abandoned that path. The problem with QbE is that you would introduce yet another instance of the application typed DataSet used for holding the filter criteria. And the other problem is that complex filters are difficult to express using QbE. The only other option would be to define yet another proprietary object model for expressing filter criteria.

Also notice that this is no OR-mapper (yet). Its just a really intuitive way to work with your application data. The OR-bit would map back and forth between your application typed DataSet and your (domain) objects. The real power of OR-mappers is not in the mapping part but in the data access layer.

So I would really like to hear your comments, suggestions and objections and if you want to see the DataSetManager internals drop me a line at obiwanjacobi@nospam_hotmail.com (remove the nospam_ ;-).

First Contact

This is the first post on my new blog.

My Previous Blog: http://bloggingabout.net/blogs/marc/