So what a day, spent about half my weekend trying to install AX 2012 reporting services extension installed on my machine which had SQL Server 2008 R2 with SP2.
The trouble started when I found out that AX 2012 Reporting Server Extensions (RSE) cannot be installed on SSRS in SharePoint Integrated Mode.
The solution was simple to use reporting services configuration manager and recreate reporting databases in native mode.
However I wasn't aware I had this option till I spoke to my colleague (after I had already uninstalled SSRS from my VM - argh). I then installed Reporting Services however it was not patched with SP1 and SP2.
So AX installer recommended me to install Sql Server 2008 R2 CU3. However this did not work for me as I got a message saying that no instances or shared components found upgradeable.
I ran the Sql Server 2008 R2 SP1 and got the same error. I was lost and confused how to patch this component only. After a lot of other attempts I finally ran Sql Server 2008 R2 SP2 setup which immediately notified me that some components have not been applied with the patch and it dawned that I simply had to run this installer instead of trying the others.
<Update> Well even after this issue was resolved I hit upon another issue which roll backed the RSE installation. The AX setup log gave following message-
An error occurred adding user to the SQL Server Reporting Services DynamicsAXBrowser role in the folder DynamicsAX
After some search on the web found out that you need to delete unused entries from sysserversessions (Note-did not find this table in AOT but went to SQL Server) to resolve this issue.
This is the link that helped me understand the issue: (translate it from Russian to English)http://axforum.info/forums/showthread.php?t=40312
Hoping this will help someone in similar situation