prakashb

Semantic Information – Part 1

Prakash B — Mon, 06 Feb 2012 23:40:15 GMT

NOTE: This blog post contains code samples that work with Microsoft “Roslyn” October 2011 CTP.

Roslyn provides a mechanism by which you can get semantic information that the compiler has about the source code. This blog post explains how you can get the semantic information for certain kinds of syntax nodes and what types of semantic information is provided by Roslyn.

Roslyn API has a method called GetSemanticModel on the Compilation object which takes the syntax tree for which semantic information is required and returns an instance of an object of type SemanticModel that has a bunch of members which provides answers for the semantic questions about specific syntax nodes in the tree. In this blog post we will be exploring in detail the results of the GetSemanticInfo method on the SemanticModel.

GetSemanticInfo takes an ExpressionSyntax node as an argument (C# has another overload which takes a ConstructorInitializerSyntax node as an argument) and returns an SemanticInfo object which contains semantic information such as type, symbols and diagnostics information pertaining to that specific node.

The following is the list of properties that are available on the SemanticInfo object:

CandidateReason: Specifies the reason in case of failure to resolve the node to a symbol.
CandidateSymbols: Contains the list of Symbols that were considered for the given node.
ConstantValue: Contains the value in case the node is an expression which is a compile time constant.
ConvertedType: Returns the type of the expression node if it underwent an implicit conversion.
ImplicitConversion: Returns information about the conversion if the expression node underwent an implicit conversion.
IsCompileTimeConstant: Returns true if the expression is a compile-time constant.
MethodGroup (C#) / MemberGroup (VB): List of symbols that were considered when a symbol resolves to a method group for C# and method or property group for VB.
Symbol: Symbol to which the node is resolved to.
Type: Type of the expression represented by the syntax node.

Let us consider the source code below for the purpose of illustration:

C#:

class Program
{
    static void Main(string[] args)
    {
        int i1 = 10;
        long i2 = i1 + (34 + 30);
    }
}

VB:

Module Program
    Sub Main(args As String())
        Dim i1 As Integer = 10
        Dim i2 As Long = i1 + (34 + 30)
    End Sub
End Module

In the above source code, if you would like to get the semantic information for the NumericLiteralExpression node “34”. The code using the Roslyn API will look like this:

C#:

class Program
{
    static void Main(string[] args)
    {
        string code = @"

class Program
{
    static void Main(string[] args)
    {
        int i1 = 10;
        long i2 = i1 + (34 + 30);
    }
}";

        var tree = SyntaxTree.ParseCompilationUnit(code);

        var comp = Compilation.Create(
                        "Test",
                        syntaxTrees: new[] { tree },
                        references: new[] { new AssemblyFileReference(typeof(object).Assembly.Location) }
                        );

        var model = comp.GetSemanticModel(tree);
        var node = tree.Root.FindToken(code.IndexOf("34")).Parent as ExpressionSyntax;
        var semanticInfo = model.GetSemanticInfo(node);
    }
}

VB:

Module Module1
    Sub Main()
        Dim code As String =
<a>

Module Program
     Sub Main(args As String())
         Dim i1 As Integer = 10
         Dim i2 As Long = i1 + (34 + 30)
     End Sub
End Module

</a>.Value

        Dim tree = SyntaxTree.ParseCompilationUnit(code)
        Dim comp = Compilation.Create(
            "Test",
            syntaxTrees:=New SyntaxTree() {tree},
            references:=New MetadataReference() {New AssemblyFileReference(GetType(Object).Assembly.Location)})
        Dim model = comp.GetSemanticModel(tree)
        Dim node = DirectCast(tree.Root.FindToken(code.IndexOf("34")).Parent, ExpressionSyntax)
        Dim semanticInfo = model.GetSemanticInfo(node)
    End Sub
End Module

Since the expression is an NumericLiteralExpression syntax node “34”, the semantic info returned would have the following information:

ConstantValue: 34
ConvertedType: Integer (No implicit conversion)
ImplicitConversion: Identity (No implicit conversion)
IsCompileTimeConstant: True
Type: Integer

The rest of the semantic info are not applicable for the NumericLiteralExpression syntax node.

Now lets get the semantic info for the AddExpression Syntax node “34 + 30” in the above code. You can get to the AddExpression node by replacing the code for declaring the “node” variable in the above sample Roslyn code with the one below:

C#:

        var node = tree.Root.FindToken(code.IndexOf("34")).Parent.Parent as ExpressionSyntax;

VB:

        Dim node = DirectCast(tree.Root.FindToken(code.IndexOf("34")).Parent.Parent, ExpressionSyntax)

The semantic info returned would have the following information:

ConstantValue: 64
ConvertedType: Integer(No implicit conversion)
ImplicitConversion: Identity (No implicit conversion)
IsCompileTimeConstant: True
Type: Integer

Since the compiler evaluates the expression “34 + 30” as a compile-time constant the ConstantValue returned in this case would be “64”.

Now lets get the semantic info for the expression “i1 + (34 + 30)”. You can get to this expression node by replacing the code for declaring the “node” variable in the above sample Roslyn code with the one below:

C#:

  var node = tree.Root.FindToken(code.IndexOf("34")).Parent.Parent.Parent.Parent as ExpressionSyntax;

VB:

  Dim node = DirectCast(tree.Root.FindToken(code.IndexOf("34")).Parent.Parent.Parent.Parent, ExpressionSyntax)

The semantic info returned would have the following information:

ConstantValue: null (C#) / Nothing (VB)
ConvertedType: Long (Implicit conversion to long)
ImplicitConversion: ImplicitNumeric (C#) / WideningNumeric (VB)
IsCompileTimeConstant: False
Type: Integer

Note that the type of this expression is integer but there is an implicit conversion to type long since the expression is assigned to a variable of type long. This is reflected in the information returned by the ConvertedType and the ImplicitConversion property in the semantic info.

Now lets get the semantic info for the IdentifierNameSyntax node “i1” which is part of the expression “i1 + (34 + 30)”. You can get to this IdentifierNameSyntax node by replacing the code for declaring the “node” variable in the above sample Roslyn code with the one below:

C#:

var node = tree.Root.FindToken(code.IndexOf("i1 + ")).Parent as ExpressionSyntax;

VB:

Dim node = DirectCast(tree.Root.FindToken(code.IndexOf("i1 + ")).Parent, ExpressionSyntax)

The semantic info returned would have the following information:

CandidateReason: None (Since the node successfully resolves to a symbol).
CandidateSymbols: Empty (Since the node successfully resolves to a symbol).
ConstantValue: null (C#) / Nothing (VB)
ConvertedType: Integer (No implicit conversion).
ImplicitConversion: Identity (No implicit conversion).
IsCompileTimeConstant: False.
MethodGroup (C#) / MemberGroup(VB): Empty.
Symbol: Local Symbol for variable “i1”
Type: Integer.

Note that the symbol returned contains the symbol of the local variable “i1” since the compiler successfully resolves the IdentifierNameSyntax node.

In this blog post we learned how to get the semantic information of your source code via the Roslyn API. We also explored the semantic information returned for certain kinds of nodes.

Next blog post will focus on some scenarios where the node does not successfully resolve to a symbol and the semantic information the Roslyn API provides for such scenarios.

Operating on specific kind of syntax nodes

Prakash B — Sat, 03 Dec 2011 01:13:00 GMT

NOTE: This blog post contains code samples that work with Microsoft “Roslyn” October 2011 CTP.

While working with Roslyn, you might often come across situations where you would like to analyze and/or process certain kinds of syntax elements in the source code. This blog post gives you a couple of ways you can go about doing it.

One approach is to call the DescendentNodes method on the SyntaxNode, this returns all the nodes that descend from the current syntax node. You can call this method on the root node of the tree and then use LINQ queries to filter them based on your requirements.

For Example: If you want to obtain a list of all syntax nodes corresponding to the “do” statement in C#, the code will look like:

C#:

    var tree = SyntaxTree.ParseCompilationUnit(code);
    var doStatements = tree.Root.DescendentNodes().
        Where(x => x.Kind == SyntaxKind.DoStatement);

Similarly, if you wish to obtain a list of syntax nodes corresponding to the “do” loop in Visual Basic which has the condition expression at either the top or the bottom of the loop, the code will look like:

VB:

    Dim tree = SyntaxTree.ParseCompilationUnit(code)
    Dim doStatements = tree.Root.DescendentNodes().
        Where(Function(x)
                 Return x.Kind = SyntaxKind.DoLoopBottomTestBlock OrElse
                      x.Kind = SyntaxKind.DoLoopTopTestBlock
              End Function)

You can then iterate over the list of nodes and the perform the analysis and/or process them.

Depending on the kind of processing you intend on doing on the syntax nodes, another option is to implement a SyntaxWalker or SyntaxRewriter and override the methods that visit the syntax nodes that you would like to process.

If you are interesting in performing updates to a node or rewriting syntax nodes then implementing SyntaxRewriter might be more appropriate.

The code sample below show the syntax rewriter implementation for updating “do” statement syntax in C#:

C#:

 class Rewriter:SyntaxRewriter
 {
     protected override SyntaxNode VisitDoStatement(DoStatementSyntax node)
     {
         // Operate on the current node and return the updated node
     }
 }

The code sample below show the syntax rewriter implementation for updating “do” loops in Visual Basic:

VB:

Public Class Rewriter
    Inherits SyntaxRewriter
    Protected Overrides Function VisitDoLoopBlock(node As DoLoopBlockSyntax) As SyntaxNode
        ' Operate on the current node and return the updated node
    End Function
End Class

You can then instantiate the writer and call the visit method with the root of the tree as parameter and get back the updated root node.

C#:

        var doRewriter = new Rewriter();
        var newRoot = doRewriter.Visit(tree.Root);

VB:

        Dim doRewriter = New Rewriter()
        Dim newRoot = doRewriter.Visit(tree.Root)

Using SyntaxRewriter you can either update the existing info on the node being processed, remove the node by returning a null or return an entirely new node that replaces the node being processed.

The code sample below illustrates how you can use SyntaxRewriter to transform a “do” statement in C# to a “while” statement by replacing the “do” node with a newly created “while” syntax node:

C#:

class Rewriter : SyntaxRewriter
{
    protected override SyntaxNode VisitDoStatement(DoStatementSyntax node)
    {
        // Operate on the current node and return the updated node
        node = (DoStatementSyntax)base.VisitDoStatement(node);

        // Get the different syntax nodes components of the Do Statement
        var doKeyword = node.DoKeyword;
        var doStatement = node.Statement;
        var whileKeyword = node.WhileKeyword;
        var condition = node.Condition;
        var openParen = node.OpenParenToken;
        var closeParen = node.CloseParenToken;
        var semicolon = node.SemicolonToken;

        // Preserve some level of trivia that was in the original Do keyword node.
        var newWhileKeyword = Syntax.Token(doKeyword.LeadingTrivia, 
            SyntaxKind.WhileKeyword, 
            whileKeyword.TrailingTrivia);

        // Preserve some level of trivia that was in the original Do keyword node and the 
        // original CloseParen token.
        var newCloseParenTrivias = closeParen.TrailingTrivia.ToList();
        newCloseParenTrivias.AddRange(doKeyword.TrailingTrivia.ToList());
        var newCloseParenTriviaList = Syntax.TriviaList(newCloseParenTrivias);
        var newCloseParen = Syntax.Token(closeParen.LeadingTrivia, 
            SyntaxKind.CloseParenToken, 
            newCloseParenTriviaList);

        var newTrailingTrivias = doStatement.GetTrailingTrivia().ToList();
        newTrailingTrivias.AddRange(semicolon.TrailingTrivia.ToList());
        var newWhileStatement = doStatement.WithTrailingTrivia(newTrailingTrivias);

        return Syntax.WhileStatement(newWhileKeyword, openParen, 
            condition, newCloseParen, newWhileStatement);
    }
}

Similarly, the code sample below illustrates how you can use SyntaxRewriter to transform a do-while loop which has the test condition on the top of the block in Visual Basic to the corresponding while loop:

VB:

Public Class Rewriter
    Inherits SyntaxRewriter
    Protected Overrides Function VisitDoLoopBlock(node As DoLoopBlockSyntax) As SyntaxNode
        node = DirectCast(MyBase.VisitDoLoopBlock(node), DoLoopBlockSyntax)
        Dim beginLoop = node.Begin
        Dim endLoop = node.End

        If (node.Kind = SyntaxKind.DoLoopTopTestBlock) AndAlso
            (beginLoop.WhileUntilClauseOpt.WhileOrUntilKeyword.Kind = SyntaxKind.WhileKeyword) Then

            Dim endKeyword = Syntax.Token(endLoop.LoopKeyword.LeadingTrivia,
                                          SyntaxKind.EndKeyword)

            Dim endWhileKeyword = Syntax.Token(Syntax.Whitespace(" "),
                                               SyntaxKind.WhileKeyword,
                                               endLoop.LoopKeyword.TrailingTrivia)

            Dim endWhile = Syntax.EndWhileStatement(endKeyword, endWhileKeyword)
            Dim beginWhileKeyword = Syntax.Token(beginLoop.DoKeyword.LeadingTrivia,
                                                 SyntaxKind.WhileKeyword,
                                                 beginLoop.DoKeyword.TrailingTrivia)

            Dim whileStatement = Syntax.WhileStatement(beginWhileKeyword,
                                                       beginLoop.WhileUntilClauseOpt.Condition)

            Return Syntax.WhileBlock(whileStatement,
                                     node.BeginTerminator,
                                     node.Statements,
                                     endWhile)
        End If

        Return node
    End Function
End Class

If you are interested in more syntax tree transformations similar to this, please do take a look at the Tree Transformation Samples that ships with the Roslyn CTP. You should be able to access the sample via the “Getting Started” link once you have installed the Roslyn CTP.

About me and the blog

Prakash B — Tue, 22 Nov 2011 01:13:54 GMT

ME:

Hello, I am Prakash Balasubramanian, a Software Development Engineer in Test in the Visual Studio team at Microsoft. For the last 5 years, I have been working on testing the new language features in the compilers for C# and Visual Basic. I joined the languages team at the beginning of C# 3.0 and LINQ, which were released as part of Visual Studio 2008. I continued to work on the C# Compiler team for the next release of C# 4.0 as part of Visual Studio 2010. Since then I moved on to a new project code-named Roslyn which involves rewriting the compilers for C# and VB in managed code. Nowadays I spend my time testing the new managed compilers for both C# and Visual Basic. You can read more about the Roslyn project here.

BLOG:

In this blog, I’ll discuss more about the Roslyn project and provide some tips on how to use some of the Roslyn APIs. I’ll try to cover some of the most frequently asked questions that we have been seeing in the Roslyn forum. Occasionally I might post on topics related to software testing as well.