There's examples of this in the documentation, but I've found it to be a useful and not necessarily intuitive technique.  Say for interop purposes the X509SecurityTokenManager needs to use RSA15 instead of RSAOAEP and the EncryptedKeyTokenManager should use Triple DES instead of AES.  Just add this to the config file:
 

<binarySecurityTokenManager>
   <add type="Microsoft.Web.Services3.Security.Tokens.X509SecurityTokenManager, Microsoft.Web.Services3, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" valueType="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-x509-token-profile-1.0#X509v3">
       <keyAlgorithm name="RSA15"/>
   </add>
</binarySecurityTokenManager>
<securityTokenManager>
   <add type="Microsoft.Web.Services3.Security.Tokens.EncryptedKeyTokenManager, Microsoft.Web.Services3, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" namespace="http://www.w3.org/2001/04/xmlenc#" localName="EncryptedKey">
       <keyAlgorithm name="TripleDES"/>
   </add>
</securityTokenManager>

Just put the type of the token manager that needs to be configured, and add the configuration.  The security tab in WSE Settings Tool will even fill in these strings automatically.