Let's take the scenario of a WCF client and a WSE service using MTOM either with or without security, here's some of the settings that help make this possible.  I'm going to focus mostly on code, but almost all or all of these are available via config.
 
First off, the proxy created for WCF will be missing pretty much all of the necessary configuration that would be present if this was a WCF service.  This is because WSE doesn't support WS-Policy so the wsdl it generates doesn't say it uses MTOM and a specific security policy like a WCF wsdl.  WCF will generate 2 bindings for a WSE service, a BasicHttpBinding for SOAP 1.1 and a custom binding for SOAP 1.2.
 
Without security, it's easiest to just modify the generated bindings.  For SOAP 1.1 just changed messageEncoding from "Text" to "Mtom".  For SOAP 1.2 replace the textMessageEncoding element with an mtomMessageEncoding element.
 
In code, SOAP 1.1 is just as easy:
 
binding.MessageEncoding = WSMessageEncoding.Mtom;
 
For SOAP 1.2, it's still conceptually simple but more code is required.  The easiest approach is to search through the Elements collection on the binding for an element of type TextMessageEncodingBindingElement, and then replace it in the collection with an MtomMessageEncodingBindingElement.
 
Turning security on adds a couple more twists.  The first one is WSE only supports WS-Addressing headers in the 8/2004 namespace but the default for WCF is 8/2005.  Next to do signing only, for example, the setting for this isn't on the binding it's on the contract description of the proxy's endpoint, so the code would look like this:
 
proxy.Endpoint.Contract.ProtectionLevel = System.Net.Security.ProtectionLevel.Sign;
 
For the security case it's probably best to just start from scratch.  Here's some code:
 
Binding GetNewBindingForSecurity(MessageVersion messageVersion)
{
   // Create and configure security for binding
   MessageSecurityVersion securityVersion =
MessageSecurityVersion.WSSecurity10WSTrustFebruary2005WSSecureConversationFebruary2005WSSecurityPolicy11BasicSecurityProfile10;
   SecurityBindingElement securityBE =
  SecurityBindingElement.CreateMutualCertificateBindingElement(securityVersion, true);
   securityBE.DefaultAlgorithmSuite = SecurityAlgorithmSuite.Default;
   securityBE.IncludeTimestamp = true;
   securityBE.MessageSecurityVersion = securityVersion;
   securityBE.SetKeyDerivation(false);
   securityBE.SecurityHeaderLayout = SecurityHeaderLayout.Lax;
 
   AsymmetricSecurityBindingElement asymmetricSecurityBE =
            securityBE as AsymmetricSecurityBindingElement;
   if (asymmetricSecurityBE != null)
   {
      asymmetricSecurityBE.MessageProtectionOrder =
                     MessageProtectionOrder.SignBeforeEncrypt;
   }
 
   // Create and configure encoding element
   MtomMessageEncodingBindingElement encodingBE = new MtomMessageEncodingBindingElement();
   //messageVersion is either MessageVersion.Soap11WSAddressingAugust2004 or MessageVersion.Soap12WSAddressingAugust2004 to interop with WSE
   encodingBE.MessageVersion = messageVersion;
 
   // Create and configure transport element
   HttpTransportBindingElement transportBE = new HttpTransportBindingElement();
   transportBE.MaxReceivedMessageSize = Int32.MaxValue;
   return new CustomBinding(securityBE, textBE, transportBE);
}
 
Three things are happening here.  Security is configured (MutualCertificate in this case but it could be any of the WSE turnkey security assertions) with the proper settings.  Mtom is configured with the proper message version, note the two values that will interop correctly.  Finally a transport is selected.  This could also be https, but then why are you using message level security, or possibly TCP.