Recently I came across the following error, while configuring BAM Alerts in BizTalk Server 2013:

Error configuring BAM Tools (BAMTools) Additional information: Error granting "Domain\user" permission to Microsoft Sql Notification Services bin directory. (Microsoft.BizTalk.Bam.CfgExtHelper.ToolsHelper)
Error getting Microsoft Sql Notification Services bin directory from SOFTWARE\Microsoft\Microsoft SQL Server\NotificationServices\InstalledVersions. (Microsoft.BizTalk.Bam.CfgExtHelper.DataProtector)
Error getting register key (Microsoft.BizTalk.Bam.CfgExtHelper.DataProtector) Overlapped I/O operation is in progress. (Microsoft.BizTalk.Bam.CfgExtHelper.DataProtector)

This error was resolved by adding sufficient permissions to the following folders:
C:\Program Files\Microsoft SQL Server\90\NotificationServices\9.0.242\bin
C:\Program Files (x86)\Microsoft SQL Server\90\NotificationServices\9.0.242\bin

Then we un-configured BAM Tools, BAM Alerts, BAM Portal and retried to configure BAM Alerts. This time the Alerts got configured successfully. We then tried to configure EDI\AS2 Runtime and got the following error:

Deploying Activity... Done.
ERROR: The BAM deployment failed. The alerts were not deployed. There was a failure while executing nscontrol.exe.
Error:"Microsoft Notification Services Control Utility 9.0.242.0c Microsoft Corp. All rights reserved.
Notification Services failed to get the metadata for the specified instance. Instance Name: BAMAlerts
SqlServerError: Source: .Net SqlClient Data Provider 
Number: 2  State: 0  Class: 20 
Server:  
Message: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server) 
Procedure:  
Line Number: 0
A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)"
(Microsoft.BizTalk.Configuration.EDIAS2.Utility)

Considering the error message, we verified the network connectivity and retired the configuration. We still got the same error.

Next, we enabled BAM tracing in bm.exe (as per the following document: http://msdn.microsoft.com/en-us/library/bb203854(v=bts.20).aspx)

There was not detailed error message apart from the fact that the following line of error was repeated:

Notification Services failed to get the metadata for the specified instance.Instance Name: BAMAlerts
We again un-configured BAM Tools, Alert and Portal and reconfigured again. This time we used a different value in “Prefix for Alert Database Name” while configuring BAM Alerts. We were able to configure all the artifacts successfully.

On further investigation, we found that, there were services created by the name of NS$BAMAlerts. BAMAlerts was the value that we were using for “Prefix for Alert Database Name”.
We cleared the NS$BAMAlerts using the following commands:

Net Stop <ServiceName> (This is to stop the service before deleting it)

Open a Notification Services command prompt and type the following

NSControl unregister -name <ServiceName> (This will clean up the registry, window service and performance counters)

NSControl delete -name <ServiceName> -Server <SQL_Server_Name> -force ( This will clean up the databases references, if any)

After deleting the NS$<BAM Services>, we again un-configured and reconfigured BizTalk. This time we were successful in configuring BizTalk.

Hope this helps.. !!!

Written By
Rasika Chaudhary

Reviewed By
Jainath V R

Microsoft GTSC India