After switching from message security to transport security, I'm seeing a bunch of weird protocols being used in message exchanges (even when the credentials are still at the message level). What's going on? Also, why do I need to provide a certificate for the server? HTTPS doesn't make me do this.
WCF has several negotiation protocols that it can use to exchange credentials. When using message security, these protocols are typically implemented at the SOAP level running through the WS-Security family of standards. With transport-level security, the client and service have to securely exchange credentials on their own. The negotiation protocol that transport security uses is going to depend on the type of identifying token being exchanged. For example, the exchange of Windows credentials has a different protocol than the exchange of UserNamePassword credentials. The details of the individual negotiation protocols aren't relevant here because they all ultimately provide transport protection of the exchange (although it is simple to find the details of protocols like SPNego).
A certificate is needed when the protocol must authenticate the service before having a secure channel to send the token. Unless you have a scheme similar to that of SSL-enabled web sites, you need some out-of-band way of providing that certificate to the client.
Next time: Avoiding OneWay Deadlocks