Fabulous Adventures In Coding
Eric Lippert is a principal developer on the C# compiler team. Learn more about Eric.
I am super excited to announce that the Roslyn project code is now sufficiently coherent that we can start showing it to customers!
But I am getting ahead of myself somewhat. What is this "Roslyn" project?
Here's the deal. We've got these great premiere languages for .NET development, C# and Visual Basic. Obviously the compilers need to do considerable lexical, syntactic and semantic analysis of the code in order to first off, produce IL out the back end of the compiler, and second, produce all of that on-the-fly analysis needed for syntax colourization, IntelliSense, automatic refactorings, and so on. We do all this work to produce an analysis of the code, but we do not let you, the customer, take advantage of that analysis engine directly. Rather, you just get to see how the compiler and IDE teams make use of that analysis engine.
This is unfortunate. When I look around just Microsoft I see a dozen little C# and VB language lexers, parsers and semantic analyzers that different teams have written to meet their own needs. There have got to be many more out there in the wild. This is a problem we can solve once for you, and then let you make use of the analysis engine for your own purposes.
Furthermore, it became clear as we were designing the next versions of C# and VB that the existing compiler infrastructure built in unmanaged C++ was not going to meet our needs without a major overhaul at some point. We want a compiler infrastructure that supports both new interesting language features, and more dynamic ways to interact with your code as you're developing it.
To achieve all those goals we decided to join the C# and VB teams together, and then split the resulting team apart into two subteams: one concentrating on upcoming C# and VB features like async/await, and one concentrating on the long-term future of the compiler and tools infrastructure. The latter team is codenamed "Roslyn", and that's the team I've been working on for some time now.
We are now at the point where the Roslyn codebase is sufficiently fully-featured and coherent that it makes sense to start getting feedback from customers. You can download the Roslyn Community Technology Preview as of right now; it installs as an extension to Visual Studio 2010 SP1. We would love to get as much constructive feedback from you as possible; rather than leaving comments here, please leave comments on the Roslyn Forum. That way we'll be sure that our crack team of program managers sees the feedback and can aggregate it all appropriately. We're making a huge investment here and want to know that we're heading in the right direction to make something insanely great for you guys.
What we're looking for feedback on primarily is the set of code analysis APIs we're exposing to you guys. The APIs themselves as a set of classes with methods and so on, are now reasonably complete; we do not anticipate making major changes to the infrastructure for Symbols, SyntaxNodes and so on unless we hear loud feedback that we have gone with the wrong model. Does this set of APIs meet your needs? What parts do you like, and what would you like to be different?
I want to make sure that we're setting expectations appropriately here. This is a very early pre-beta-quality release of some extremely complicated software. Keep that in mind as you are using it. The lexical and syntactic analysis engines that underlie those APIs are pretty much complete. The semantic analysis engine that sits behind the semantic analysis APIs is right now nowhere near done; on the C# side we are still missing semantic analysis of major feature areas like query comprehensions, attributes, iterator blocks, optional arguments, dynamic, async/await and unsafe as well as a bunch of control flows; VB is in a similar state. Extension methods, method type inference, lambdas, and generics are working, so LINQ queries in the "method call" syntactic form should work.
Also, the performance of the entire analysis engine is not as high as we would like; we have done some performance tuning but there is more to come.
In short, you're getting the earliest possible build we could reasonably show to people to get good feedback; we are still a long way from being done. We are absolutely not announcing any dates or "ship vehicles" for a final release; even if I knew - which I don't - I couldn't tell you, so don't ask.
We are also as a part of this release shipping a preview of a new "interactive scripting" environment for C# that will allow you to play around with code in a more experimental and free-form way, like you can do in F# and other "Read-Eval-Print Loop" languages. Give it a spin and let us know what you think! (Sorry, no VB version of this feature is available with the CTP.)
Over the next few months I'll do some more posts describing the interesting technology we've built behind-the-scenes to make the C# and VB Roslyn code analysis engines work. For now, you can get more information on Roslyn at the following places:
Some helpful blog posts:
Is there a plan to reimplement the existing tools to use roslyn once it's done(such as intellisense) or is that too large of a customer impacting project/too far into the future to tell?
No; the plan is to re-implement the existing tools like IntelliSense to use Roslyn *before* it's done. There is no reason why we have to wait for the whole thing to be finished before we start using it internally! In fact, that would be a bad idea; by using it internally we are discovering very early what parts of the APIs work well and which ones need improvement. We have been re-implementing important tools like the "extract method" refactoring to use Roslyn for some time now; we are now at the point where we are satisfied that our APIs will meet our needs to build tools on top of Roslyn. The relevant question that motivates the CTP is are the APIs good enough for you to build the code analysis tools you want with them? -- Eric
Great work to all of you involved. I am actually at a conference at Microsoft today and got the chance to see Alex Turner demonstrate the capabilities of Roslyn as a part of his talk on C# today. Really impressive stuff and I am excited to see what direction Roslyn and the other related parts (especially meta programming, which Alex explained rather well why there is some hesitation over the subject) take C# in the future.
I can definitely see this having advantage for development groups being able to do things like build their own static analysis tools for enforcing coding standards specific to their team. I really can't wait to get my hands on the CTP and start prodding it and seeing what sort of things I can do with it.
Will there be some documentation on MSDN or discussion here about how C# works in "scripting"/top-level expression mode and how the incremental submissions in interactive mode trample on or succeed each other? I read the original request for comments here a few years ago and I'd appreciate a look into what has since transpired.
That would be a great blog topic. I do not understand this area very well myself; I've been working on the semantic analyzer mostly, not on the interactive code. It would help me to learn it if I wrote about it, so perhaps I shall do so. -- Eric
Additionally, will it be possible to ask for the expanded syntax tree of some code, where (for example) iterators are turned into classes, foreach loops into whiles and usings into try-finallys, or will these conversions continue to only happen from C# directly to IL? I have a theory it could make flow analysis more consistent, although if you're already supporting the special functionality of all these things (I didn't even mention hoisting locals for closures or the await state machine), I guess the extra logic for this is just a few more drops in the ocean.
We considered exposing the "lowered" forms but decided to not do so. The reason is that we decided to only expose the "syntax" trees and not the "semantic" trees. If you want semantic information about a syntax tree, just ask for an analysis of it. Most of the "lowerings" you talk about are lowerings on the semantic tree; only query comprehension rewrites are purely syntactic. -- Eric
As an addition to my previous questions, thank you all very much for doing this. I've been hoping for something like this to happen for years even before the word started going around back in 2008. You are pushing the state of the art forward for statically-typed languages and you are entrusting the community with the same smarts Microsoft has. Besides also making it open source, which I know you want to happen, I can think of no better sign of respect.
Could I use this as a replacement for the missing parts of the Microsoft.CSharp namespace, so I can more easily do dynamic code generation?
So, is my understanding that Roslyn code analysis features are not and won't be available outside Visual Studio correct? Or is it true only for this CTP?
The code analysis APIs are just a library; you can new up a Compilation, start plugging code and references into it, and query the compilation for facts about the analyzed code. Obviously the infratstructure that does stuff like adding smart tags to the VS editor requires VS. -- Eric
Is there any Roslyn API documentation online?
Full API documentation is not yet written; the complete set of APIs is enormous and still somewhat in flux, so it would be premature to spend a lot of resources on documenting them. I'm not sure what all is included in the CTP as far as documentation goes. -- Eric
Based on my reading of the overview material, it seems like Roslyn isn't just a code analysis tool (compiler front-end), it's also an IL emitter (compiler back-end). This means it should allow you to create your own language (possibly very much like C#) rather more easily than before, even if it does not allow you to extend the C# language itself. For example, you can use Roslyn to parse your C#-like program, invoke some macros (think of them as declarative refactorings) that you define to rewrite the syntax tree, and then use Roslyn to generate assemblies from your output -- or convert the rewritten syntax tree back into C# to be compiled by ordinary means.
Previously this sort of thing was performed only by complex IL manipulation tools like Cecil or PostSharp. Now you can write your own "PreSharp" in a day! At some point in the future, the C# compiler itself could even have extensibility points that you can hook into with Roslyn APIs.
Did I get any part of this wrong?
Though it might be possible to do all that with Roslyn, that would be using the tools "off label". They are not designed for those scenarios. The point of Roslyn is to enable analysis of C# and VB programs or program fragments, not to enable analysis of languages that are in some way similar to C# and VB.
I agree that it would be very nice to have an IL-emitting compiler back-end that took as its inputs a form "lower" than C# or VB parse trees, but "higher" level than IL. We're kicking around ideas in that space and discussing them with the Microsoft Research guys that work on CCI, but like I said before, these ideas are still too inchoate to discuss further. -- Eric
Gabe: It's curious that Anders has previously opposed tools like PostSharp on the grounds that IL weaving is messy. I wonder what his opinion would be of Roslyn-powered precompile-conversion, official extensibility points or not.
Ah, so this is not anything like LLVM, but it is exactly like Clang http://clang.llvm.org/ . From it's web page, Clang:
- Modular library based architecture
- Support diverse clients (refactoring, static analysis, code generation, etc)
Will the interactive window let me run statements in the context of a running code around a breakpoint while debugging? (Just like the watch window lets to do it with lots of limitations)
Artificial intelligence programming.
when will microsoft create in .net
It would be interesting to implement something like that gist.github.com/1306491 in roslyn
Will these APIs be exposed in Visual Studio itself in such a way that you can use them to write "custom" compiler error messages to cover code "business rules" - like "this interface should only be implemented in a certain assembly" or "when inheriting from 'class Foo<T> where T : Foo<T>', T must be the class itself" or "classes that inherit from this class must have this static method" or "classes annotated with this attribute must have a constructor with these parameters" or "the <T> in this method must be an interface/delegate/enum (since "where T : delegate" etc is not supported directly)"
I frequently find myself with code rules like this and no way to enforce them, which can lead to bugs if everyone on the team is not aware of all the contracts as well as the original programmer is. It'd be great if we could deputize the compiler to help enforce these things.
Your example rules can be easily verified using mere reflection. Just write a verification app and invoke it in postbuild event (or get more fancy and do an msbuild task).
@Stuart & @Martin:
If something more standard than a fully custom verification system is desired, You can use tools like NDepend. NDepend in particular is explictly designed to use custom rules. The standard ules are mere suggestions for you to edit or diable as you see fit. The real power is adding new rules like the ones you mention, or rules to enforce layering, or similar.