There is a sample on MSDN that shows how to write and install custom transport agent (it can be found here: http://msdn.microsoft.com/en-us/library/aa579185%28v=exchg.140%29.aspx ).  The agent itself is very simple, but it has been pointed out that if you use the VB.Net sample code, then the installation command is incorrect.  If you use the suggested command:

Install-TransportAgent -Name "MyCustomAgent" -TransportAgentFactory "MyAgents.MyAgentFactory" -AssemblyPath "C:\myagents\MyAgent.dll"

You will receive this error:

The TransportAgentFactory type "MyAgents.MyAgentFactory" doesn't exist. The TransportAgentFactory type must be the Microsoft .NET class type of the transport agent factory.
Parameter name: TransportAgentFactory
    + CategoryInfo          : InvalidArgument: (:) [Install-TransportAgent], ArgumentException
    + FullyQualifiedErrorId : 72602B5B,Microsoft.Exchange.Management.AgentTasks.InstallTransportAgent

The reason for this is that when you use VB.Net, something seems to happen to the namespace in that it has to be doubled-up in the Install-TransportAgent call.  I don't currently have an explanation for this, but the correct command for the sample agent in VB.Net is:

Install-TransportAgent -Name "MyCustomAgent" -TransportAgentFactory "MyAgents.MyAgents.MyAgentFactory" -AssemblyPath "C:\myagents\MyAgent.dll"

The same applies to any other VB.Net transport agent, so if you are having problems installing, please try the above.