When creating web app in SharePoint 2010, we can only choose to use Windows Authentication if we choose to use Classic mode. If we want to use any other authentication method, like FBA, we much choose to use Claim mode. (Although FBA is able to be selected in Classic mode in Beta 2, it would not be available in RTM).
With Claim mode authentication, no matter whether we use Windows Authentication, FBA or the authentication based on other trusted identity providers, the identity is finally converted to claim identity. So now if we want to use different authentication methods for a single web app, we don’t need to extend the web app to get multiple authentication providers. A single authentication provider with Claim mode authentication is all what we need.
The configuration of FBA in Claim mode is a little different from the way we use for MOSS 2007. The following are the steps.
$w = Get-SPWebApplication "http://<Url>/"
$w.UseClaimsAuthentication = True;