1. Configure SharePoint Central Admin 

• Login to SharePoint Central Admin
• Go to Application Management / Application Security / Authentication Providers and Change the Web Application to the one which needs to be configured for Forms Based Authentication
• Click zone / default, change authentication type to forms and enter ActiveDirectoryMemebershipProvider under membership provider name ( for example , "ADMembershipProvider") and save this change
  

2.Update the web.config of SharePoint Central admin site
 

under <configuration> node

<connectionStrings>
<add name="ADConnectionString" connectionString="LDAP://DynamicsAX.local/CN=Users,DC=DynamicsAX,DC=local />
</connectionStrings>

under <system.web> node

<membership defaultProvider="ADMembershipProvider">
<providers>
<add name="ADMembershipProvider"
type="System.Web.Security.ActiveDirectoryMembershipProvider,System.Web,Version=2.0.0.0,Culture=neutral,PublicKeyToken=b03f5f7f11d50a3a" connectionStringName="ADConnectionString"
connectionUsername="xxx" connectionPassword="yyy"
enableSearchMethods="true"
attributeMapUsername="sAMAccountName"/>
</providers>
</membership>

replace the connectionstring to the one matching your encvironment and replace"xxx" with domainname\username and "yyy" with password

3.Update the web.config of SharePoint Web application

•  Repeat step 2 for the web.config of the SharePoint webapplication to be configured for Forms Based Authentication
•  Change the authentication in web.cconfig to
  <authentication mode="Forms">
      <forms loginUrl="/_layouts/login.aspx"></forms>
  </authentication>

For more details
http://technet2.microsoft.com/Office/en-us/library/23b837d1-15d9-4621-aa0b-9ce3f1c7153e1033.mspx?mfr=true
http://msdn2.microsoft.com/en-us/library/ms998360.aspx