A] Forms Authentication

1. If using SQL 2005 SP2 make sure you have installed the sharepointrs.msi that includes the following fix http://support.microsoft.com/kb/939942/ else move onto step2.
2. Add the following entries to the Web.config file for the SharePoint web application you want to enable launching of Report Builder:
   <location path="_vti_bin/ReportBuilder/ReportBuilder.application">
     <system.web>
       <authorization>
         <allow users="*" />
       </authorization>
     </system.web>
   </location>
   <location path="_vti_bin/ReportBuilder/ReportBuilder.exe.manifest">
     <system.web>
       <authorization>
         <allow users="*" />
       </authorization>
     </system.web>
   </location>
   <location path="_vti_bin/ReportBuilder/ReportBuilder.chm.deploy">
     <system.web>
       <authorization>
         <allow users="*" />
       </authorization>
     </system.web>
   </location>
   <location path="_vti_bin/ReportBuilder/ReportBuilder.exe.deploy">
     <system.web>
       <authorization>
         <allow users="*" />
       </authorization>
     </system.web>
   </location> 
3. For non-English sites ensure you add the appropriate appropriate LCID and Culture entries as well to the Web.config file. For example enabling for launching in Japanese in addition to English (in Step 2) add the following:
   <location path="_vti_bin/ReportBuilder/1041/ReportBuilder.chm.deploy">
     <system.web>
       <authorization>
         <allow users="*" />
       </authorization>
     </system.web>
   </location>
   <location path="_vti_bin/ReportBuilder/ja/ReportBuilder.resources.dll">
     <system.web>
       <authorization>
         <allow users="*" />
       </authorization>
     </system.web>
   </location>
   For details on LCIDs and Cultures supported by the Reporting Services Add-in for SharePoint refer to "Supported Languages" section @ http://technet.microsoft.com/en-us/library/aa905871(SQL.90).aspx or the install folder under %programfiles%\Common Files\Microsoft Shared\web server extensions\12\ISAPI\ReportBuilder.
4. Enable anonymous access on the ReportBuilder folder via IIS manager. For more information, see "How to Enable Anonymous Access on the ReportBuilder Folder" section @ http://msdn.microsoft.com/en-us/library/ms365173(SQL.90).aspx

A known error you can run into if not setup correctly is "Cannot Start Application. Cannot continue. The application is improperly formatted. Contact the vendor for assistance."

B] Basic Authentication
*Note: These steps also apply to other authentication schemes used with SharePoint that result in a User Credentials dialog box shown to users at login time.*

1. If using SQL 2005 SP2 make sure you have installed the sharepointrs.msi that includes fix #50001712 from http://support.microsoft.com/kb/941450 else move onto step2.
2. If users encounter an HTTP 401 or 400 when launching ReportBuilder then you need to ensure users select the "Remember Password" checkbox when you are provided the user credentials dialog at login time. The root cause is that the login / authentication via the user credential dialog box results in SharePoint creating a WSS_KeepSessionAuthenticated cookie to ensure this user remains authenticated for the duration of their session. ReportBuilder is launched usingClick-once, which is not aware of nor supports using this WSS_KeepSessionAuthenticated cookie resulting in the error from SharePoint during launch of ReportBuilder.
   *Additionally, if the user is using an OS that supports UAC then ensure you invoke the browser with "Run As Administrator".*