Evolving the Reflection API

Evolving the Reflection API

Rate This
  • Comments 16

As many developers have noticed, the reflection APIs changed in the .NET API set for Windows Store apps. Much of .NET’s ability to offer a consistent programming model to so many platforms over the last ten years has been the result of great architectural thinking. The changes to reflection are here to prepare for the challenges over the next decade, again with a focus on great architecture. Richard Lander and Mircea Trofin, program managers respectively from the Common Language Runtime and .NET Core Framework teams, wrote this post. -- Brandon

In this post, we will look at changes to the reflection API for the .NET Framework 4.5 and .NET APIs for Windows Store apps (available in Windows 8). These changes adopt current API patterns, provide a basis for future innovation, and retain compatibility for existing .NET Framework profiles.

Reflecting on a popular .NET Framework API

The .NET Framework is one of the most extensive and productive development platforms available. As .NET developers, you create a broad variety of apps, including Windows Store apps, ASP.NET websites, and desktop apps. A big part of what enables you to be so productive across Microsoft app platforms is the consistency of the developer experience. For many developers, the language – be it C# or Visual Basic – is this common thread. For others, it is the Base Class Libraries (BCL). When you look more closely, you'll see that reflection is the basis for many language, BCL, and Visual Studio features.

Many .NET Framework tools and APIs rely on CLR metadata, accessed via reflection, to operate. Examples include static analysis tools, application containers, extensibility frameworks (like MEF), and serialization engines. The ability to query, invoke, and even mutate types and objects is a focal point for .NET developers in many scenarios.

In the .NET Framework 4, we support a rich but narrow set of views and operations over both static metadata and live objects. As we looked at expanding reflection scenarios, we realized that we first needed to evolve the reflection API to enable further innovation.

We saw the design effort to create .NET APIs for Windows Store apps as the avenue to establish the changes that we needed in reflection. We defined a set of updates to reflection that would enable more flexibility for future innovation. We were able to apply that design to .NET APIs for Windows Store apps and also to the Portable Class Library. At the same time, we were able to compatibly enable that design in the .NET Framework 4.5. As a result, we were able to evolve the reflection API consistently and compatibly across .NET Framework API subsets.

We took an in-depth look at the .NET APIs for Windows Store apps in an earlier .NET Team blog post. In this post, we saw that this new API subset is considerably smaller than the .NET Framework 4.5. We have documented the differences on MSDN. One difference that applies is Reflection.Emit, which is not available to Windows Store apps.

Reflection scenarios

Reflection can be described in terms of three big scenarios:

  • Runtime reflection
    • Primary scenario: You can request metadata information about loaded assemblies, types, and objects, instantiate types, and invoke methods
    • Status: Supported
  • Reflection-only loading for static analysis
    • Primary scenario: You can request metadata information about types and objects that are described in CLR assemblies, without execution side-effects
    • Status: Limited support
    • Alternatives today: CCI
  • Extensibility of reflection
    • Primary scenario: You can augment metadata in either of the two scenarios above
    • Status: Supported, but complicated

Today, we have reasonable support for runtime reflection, but not for the other two scenarios. Given that reflection is such a key API, richer support across all three scenarios would likely enable new classes of tools, component frameworks, and other technologies. Full support is inhibited by some limitations in the reflection API in the .NET Framework 4. Primarily, reflection innovation is constrained by a lack of separation of concepts within the API, particularly as it relates to types. The System.Type class is oriented (per its original design) around the runtime reflection scenario. This is problematic because System.Type is used to represent types across all scenarios. Instead, we would benefit from a broader representation of types, designed to support all three scenarios.

Splitting System.Type into two concepts

System.Type is the primary abstraction and entry point into the reflection model. It is used to describe two related but different concepts, reference and definition, and enables operations across both. This lack of separation of concepts is the primary motivation for changing the reflection API. For example, the following scenarios are either difficult or unsupported with the existing model:

  • Reading CLR metadata without execution side-effects
  • Loading types from alternate sources other than CLR metadata
  • Augmenting type representation (for example, changing shape, adding attributes)

In other parts of the product, we have first-class concepts of reference and definition. At a high level, a reference is a shallow representation of something, whereas a definition provides a rich representation. One needs to look no farther than assemblies, a higher level part of reflection, to see this. The System.Reflection.Assembly class represents assembly definitions, whereas the System.Reflection.AssemblyName class represents assembly references. The former exposes rich functionality, and the latter is just data that helps you get the definition should you want it. That’s exactly the model that we wanted to adopt for System.Type.

In order to achieve a similar split for the System.Type concept and class, we created a new System.Reflection.TypeInfo class and shrunk the meaning of the System.Type class. The TypeInfo class represents type definitions and the Type class represents type references. Given a Type object, you can get the name of the type as a string, without any requirement to load anything more. Alternatively, if you need rich information about a type, you can get a TypeInfo object from a Type object. Given a TypeInfo object, you can perform all the rich behavior that you expect with a type definition, such as getting lists of members, implemented interfaces, or the base type.

The value of Type and TypeInfo

The API changes in the .NET Framework 4.5 were made such that we could evolve the reflection API to deliver new scenarios and value. While we changed the shape of the API, we haven’t yet added the additional features that would deliver the value. This section provides a preview of what that value would look like in practice.

Suppose that we are using a static analysis tool that is implemented with the new reflection model. We are looking for all types in an app that derive from the UIControl class. We want to be able to run this tool on workstations on which the UIControl assembly (which contains the UIControl class) does not exist. In this example, let’s assume that we open an assembly that contains a class that derives from the UIControl class:

class MyClass : UIControl

In the .NET Framework 4 reflection model, the Type object (incorporating both reference and definition) that represents MyClass would create a Type object for the base class, which is UIControl. On machines that don't have the UIControl assembly, the request to construct the UIControl Type object would fail, and so too would the request to create a Type object for MyClass, as a result.

Here you see what the reference/definition split achieves. In the new model, MyClass is a TypeInfo (definition); however, BaseType is a Type (reference), and will contain only the information about UIControl that the (MyClass) assembly contains, without requiring finding its actual definition.

Type baseType = myClassTypeInfo.BaseType;

In other scenarios, you may need to obtain the definition of UIControl. In that case, you can use the extension method on the Type clas, GetTypeInfo, to get a TypeInfo for UIControl:

TypeInfo baseType = myClassTypeInfo.BaseType.GetTypeInfo();

Of course, in this case, the UIControl assembly would need to be available. In this new model, your code (not the reflection API) controls the assembly loading policy.

Once again, in the .NET Framework 4.5, the reflection API still eagerly loads the type definition for the base class. The reflection API implementation is largely oriented around the runtime reflection scenario, which has a bias towards loading base type definitions eagerly. At the point that we build support for the static analysis scenario described earlier, we will be able to deliver on full value of the reference/definition split, made possible by Type and TypeInfo.

Applying the System.Type split to the Base Class Libraries

Changing the meaning of Type and adding TypeInfo made it necessary to ensure consistency in the .NET Framework BCL. The .NET Framework has many APIs that return the Type class. For each API, we needed to decide whether a Type (reference) or a TypeInfo (definition) was appropriate. In practice, these choices were easy, since the API inherently either returned a reference or a definition. You either have access to rich data or you don't. We’ll look at a few examples that demonstrate the trend.

  • The Assembly.DefinedTypes property returns TypeInfo.
    • This API gets the types defined in that assembly.
  • The Type.BaseType property returns a Type.
    • This API returns a statement of what the base type is, not its shape.
    • The base type could be defined in another assembly, which would require an assembly load.
  • The Object.GetType method returns a Type.
    • This API returns a Type, since you only need a representation of a type, not its shape.
    • The type could be defined in another assembly, which would require an assembly load.
    • By returning a Type and not a TypeInfo, we also removed a dependency on the reflection subsystem from the core of the .NET Framework.
  • Language keywords, like C# typeof, return a Type.
    • Same rationale and behavior as Object.GetType.

Deeper dive into the reflection model update

So far, we’ve been looking at better abstraction in the reflection API, which is the reference/definition split that we made with Type and TypeInfo. We also made other changes, some of which contributed to the reference/definition split and others that satisfied other goals. Let's dive a little deeper into those changes.

Replacing runtime reflection-oriented APIs

In the .NET Framework 4.5 (and earlier releases), you can call Type.GetMethods() to get a list of methods that are exposed on a given type. Such a list of methods will include inherited methods. Our implementation of the GetMethods method has a particular policy for how it traverses the inheritance chain to get the complete list of methods, including loading assemblies for base types that are located in other assemblies. This approach can sometimes be problematic, since loading assemblies can have side-effects that change the execution of your program. The GetMethods method is an example of the heavy bias that the reflection API has to satisfying runtime reflection scenarios, and therefore, is not appropriate for reflection-only loading scenarios.

For the new model, we introduced the DeclaredMethods property that reports the members that are declared (as opposed to members that are available via inheritance) on a given type. There are several other properties, such as DeclaredMembers and DeclaredEvents that follow the same pattern.

The following example illustrates the difference in the behavior between Type/TypeInfo.GetMethods and TypeInfo.DeclaredMethods, using the .NET Framework 4.5.

class MyClass
{
public void SomeMethod() { }
} class Program { static void Main(string[] args) { var t = typeof(MyClass).GetTypeInfo(); Console.WriteLine("---all methods---"); foreach (MethodInfo m in t.GetMethods()) Console.WriteLine(m.Name); Console.WriteLine("======================="); Console.WriteLine("---declared methods only---"); foreach (MethodInfo m in t.DeclaredMethods) Console.WriteLine(m.Name); Console.ReadKey(); } }

The output is:

---all methods---
SomeMethod
ToString
Equals
GetHashCode
GetType
=======================
—declared methods only---
SomeMethod

You will notice that the GetMethods method retrieves all the public methods accessible on MyClass – including the ones defined on System.Object, like ToString, Equals, GetType and GetHashCode. DeclaredMethods returns all the declared methods (in this case, one method) on a given type, regardless of visibility, and including static methods.

Adopting current API patterns -- IEnumerable<T>

In the era of the async programming model, the reflection APIs stand out since many of them, such as MemberInfo[], return arrays. As you likely know, arrays need to be fully populated before they are returned from an API. This characteristic is bad for both working-set and responsiveness. In the .NET APIs for Windows Store apps, we have replaced all the array return types with IEnumerable<T> collections. Most of you will appreciate working with this friendlier API pattern, which will likely blend in better with the rest of your code.

We have not yet fully taken advantage of this model yet. In our internal implementation of these APIs, we are still using the arrays that were formerly part of the public API contract. In a later version of the product, we can change the implementation to lazy evaluation without needing an associated change to the public API.

Compatibility across .NET target frameworks

We designed the reflection API updates with a goal of compatibility with existing code. In particular, we wanted developers to be able to share code between the .NET Framework 4.5 and .NET APIs for Windows Store apps.

In .NET APIs for Windows Store apps, TypeInfo inherits from MemberInfo, while Type inherits from Object. Type definitions must inherit from MemberInfo to allow for nested types – types that are members of other types – in the same way that methods, properties, events, fields, or constructors are members of a type. You can see that this inheritance approach makes sense, particularly now that Type is very light-weight.

In the .NET Framework 4.5, TypeInfo inherits from Type, while Type is still a MemberInfo. In order to maintain compatibility with the .NET Framework 4, we could not change the base type of Type. We expect that future .NET Framework releases will maintain this same factoring (that is, Type will continue to be a MemberInfo) for backward compatibility.

However, if you are writing code that targets the .NET Framework 4.5, and you want to use the new reflection model, we encourage you to write that code as a Portable Class Library. Portable Class Library projects that target the .NET Framework 4.5 and .NET APIs for Windows Store apps follow the new model, as described above.

See the figure below for a visual illustration of the reflection type hierarchy in the .NET Framework 4.5 and .NET APIs for Windows Store apps.

Reflection type hierarchy in the .NET Framework 4.5 and .NET APIs for Windows Store apps

Figure: Reflection type hierarchy in the .NET Framework 4.5 and .NET APIs for Windows Store apps

Updating your code to use the new reflection model

Now that you have a fundamental understanding of the new model, let's look at the mechanics of the APIs. Basically, you need to know three things:

  1. The Type class exposes basic data about a type.
  2. The TypeInfo class exposes all the functionality for a type. It is also a proper superset of Type.
  3. The GetTypeInfo extension method enables you to get a TypeInfo object from a Type object.

The following sample code demonstrates the basic mechanics of Type and TypeInfo. It also provides examples of the data that you can get from Type and TypeInfo.

class Class1
{
    public void Type_TypeInfo_Demo()
    {
        //Get a Type
        Type type = typeof(Class1);
        //Gets the name of the type
        String typeName = type.FullName;
        //Gets the assembly-qualified type name
        String aqtn = type.AssemblyQualifiedName;

        //Get TypeInfo via the type
        //Note that .GetTypeInfo is an extension method
        TypeInfo typeInfo = type.GetTypeInfo();
        //Get the list of members
        IEnumerable<MemberInfo> members = typeInfo.DeclaredMembers;
        //You can do many other things with a TypeInfo
    }
}
  

We have received feedback that this change inserts another step – calling the GetTypeInfo extension method – and that it represents a migration hurdle for developers. This change is opt-in for the .NET Framework 4.5, for compatibility reasons. You do not have to use the GetTypeInfo method or the TypeInfo class if you are targeting the .NET Framework 4.5.

With .NET APIs for Windows Store apps, we had the opportunity to create a fully consistent API, which is why we chose to create a clean split between Type and TypeInfo. As a result, code that targets .NET APIs for Windows Store apps will need to use appropriate combinations of Type and TypeInfo classes and the GetTypeInfo extension method. The same is true for Portable Class Library code that targets both .NET APIs for Windows Store apps and the .NET Framework 4.5.

Writing code for the new reflection API – Windows Store and Portable Class Library

As we discussed above, you'll need to adopt the new reflection model if your code targets .NET APIs for Windows Store apps or you're creating a Portable Class Library project that targets both .NET APIs for Windows Store apps and the .NET Framework 4.5.

For example, you will notice that the Get* methods (for example, GetMethod) described earlier are not available, but are replaced by the Declared* properties (for example, DeclaredMethod). If the Get* methods are not present, the reflection binding constraints (BindingFlags options) are not available either. If you're writing new code, you'll need to follow the new model, and if you're porting code from another project, you'll need to update your code to the same model. We understand that these changes may result in non-trivial migration efforts in some cases; however, we hope that you can see the value that can be achieved with the type/typeinfo split.

While the APIs have changed, you may still need to access inherited APIs and filter results. There are a couple of patterns that you can use to accommodate those changes. We’ll look at those now.

We recommend that you write the code that provides the reflection objects that you need. You’ll actually get a clearer view of what the reflection sub-system does by seeing the code in your source file. Your implementation may also be more efficient than our implementation in the .NET Framework, since we accommodate several uncommon cases.

You'll need some code that is a proxy for GetMethods, but that is implemented in terms of the new reflection API. You might need a replacement for another Get* method, such as GetInterfaces; however, you should find that the GetMethods example equally applies. The most straightforward implementation for GetMethods follows. It walks the inheritance chain of a type, and requests the set of declared methods on each class in that chain.

public static IEnumerable<MethodInfo> GetMethods(this Type someType)
{
    var t = someType;
    while (t != null)
    {
        var ti = t.GetTypeInfo();
        foreach (var m in ti.DeclaredMethods)
            yield return m;
        t = ti.BaseType;
    }
}

Since binding flags are not provided in the new reflection API, you do not immediately have an obvious way to filter results, to public, private, static members, or to choose any of the other options offered by the BindingFlags enum. To accommodate this change, you can write pretty simple LINQ queries to filter on the results of the Declared* APIs, as you see in the following example:

IEnumerable<MethodInfo> methods = typeInfo.DeclaredMethods.Where(m => m.IsPublic);
  

We do offer another pattern as an option for porting code more efficiently. We created the GetRuntimeMethods extension method as a convenience API that provides the same semantics as the existing GetMethods API. Related extension methods have been created as an option for the other Get* methods, such as GetRuntimeProperties, as well. As the API names suggests, they are runtime reflection APIs, which will load all base types, even if they are located in other assemblies. These new extension methods do not support the BindingFlags enum, so the filtering approach suggested with LINQ above also applies.

Both of these suggested patterns are good choices for adopting the new reflection model. Note that if reflection support expands in the future to include reflection-only scenarios for static analysis, GetRuntime* methods would no longer be appropriate, should you want to take advantage of those new scenarios.

Writing code for the new reflection API – .NET Framework 4.5

If your code targets the .NET Framework 4.5, you can opt to use the new model, but you do not have to. The .NET Framework 4.5 API is a superset of old and new reflection models, so all the APIs that you’ve used before are available, plus the new ones.

Portable Class Library projects that target the .NET Framework 4, Silverlight, or Windows Phone 7.5 expose only the old model. In these cases, the new reflection APIs are not available.

Conclusion

In this post, we’ve discussed the improvements that we made to reflection APIs in .NET APIs for Windows Store apps, the .NET Framework 4.5, and Portable Class Library projects. These changes are intended to provide a solid basis for future innovation in reflection, while enabling compatibility for existing code.

For more information, porting guides, and utility extension methods, please see .NET for Windows Store apps overview in the Windows Dev Center.

--Rich and Mircea

Follow or talk to us on twitter -- http://twitter.com/dotnet.

Leave a Comment
  • Please add 3 and 7 and type the answer here:
  • Post
  • Making a breaking change like this across a *variant* of .NET seems like a really bad idea. You should either take the breaking change, or not take it. This whole concept of "well in this flavor of .NET, you do things this way..." defeats one of the major purposes of .NET - that code is portable.

    Also, I really don't agree with the idea that you expect developers to write their own extension methods if they want to get the functionality of methods that you removed. As you admit in this blog post, those methods you removed can be useful. Do they really harm anything leaving them in there?

  • There seems to be a variable name missing here:

    IEnumerable<members> = typeInfo.DeclaredMembers;

    Other than that I agree with MgSm88.

    Also please keep that CxO Bullshit about "delivering value" out of blog posts intended for developers.

  • @qqq -- code sample fixed. Thanks.

    Richard Lander [MSFT]

  • @MgSm88: we saw the .NET APIs for Windows Store subset as an important opportunity for evolving framework APIs. We knew that there was a tradeoff - and that not everyone would be happy :) - but felt that we were on the right side of it.

    Portable code is a very important goal to us, and we agree with your comment on “variants of .NET”. You’ll find that the new APIs play a significant role in this story. Some details are in this channel 9 video (channel9.msdn.com/.../NET-45-David-Kean-and-Marcea-Trofin-Portable-Libraries). The core of it is that the shape of APIs you see in .NET for Windows Store constitute the basis we intend to build portability from grounds up in the .NET ecosystem, moving forward.

    WRT removing methods, for those that were deemed quite useful, we actually provided the extension methods ourselves. For example, in the GetMethods case, we provided a GetRuntimeMethods extension method. These ship in the box, so you just need to use them, if you need the functionality. To your question about the impact of “leaving them in”, the simple answer is that whatever we pulled out contradicted one or more of our principles we presented in an earlier post (blogs.msdn.com/.../net-for-metro-style-apps.aspx)

    Mircea ("Mitch") Trofin [MSFT]

  • I totally follow the rationale of the change, especially since I have worked with the native metadata APIs, but I still find it very sad that there has to be so many versions of .NET each with small and not-so-small API differences. It's even more painful as documentation for this new .NET profile is really lacking (as is general WinRT, if that's what it's still called, documentation). For example, I haven't found any way to get documentation for the System.Type type from this .NET profile other than through Intellisense.

    What little documentation there is is also very hard to search for, again not only for this .NET profile but also for WinRT in general. This is especially a pain point since Microsoft sources themselves can't agree on whether it's "Metro", "Windows 8 Apps", "Windows Store Apps" or "Modern UI". As a result, we now have to try various permutations of ".NET for Modern UI Windows 8 Store Apps" to get any pertinent search results.

    Basically, I like the general direction of the changes from a design perspective, but the impact on the .NET ecosystem is really unpleasant and the documentation needs to be vastly improved.

  • @TrillianX -- The documentation issue is both known and noted. We want to get this fixed in one of our next doc refreshes. The search engine issue is annoying, but should sort itself out relatively quickly as the old permutations fall away in the search rankings.

  • I think it is a very good change, the only issue is that you can not change it across all versions, but it is understandable.

  • The breaking changes here is just another item to add to the very long list of reasons why .NET 4.5 should not have been an in-place upgrade.

  • @Vaccano -- There are no changes discussed in this post that break compatibility w/rt .NET 4 apps. The post clearly states in the first section that "we were able to compatibly enable that design in the .NET Framework 4.5".

    If you re-read the post, you'll find this text:

    "With .NET APIs for Windows Store apps, we had the opportunity to create a fully consistent API, which is why we chose to create a clean split between Type and TypeInfo. As a result, code that targets .NET APIs for Windows Store apps will need to use appropriate combinations of Type and TypeInfo classes and the GetTypeInfo extension method. "

    As the post states, we took the opportunity to create a fully consistent API with .NET APIs for Windows Store apps. You can consider that a "source break" with .NET 4 if you'd like, but not a binary break as your comment would suggest.

    Does that help?

    thanks -- rich

  • Because .net 4.5 does not support Windows XP and Windows Server 2003, So my team can not update to .net 4.5. Any improvements you done have no meanings to us.

  • Instead if IE<T>, which is the least rich interface possible, you could have returned a custom IList<T> derived type that still does lazy loading internally.

    What types of app do you have in mind for "Reflection-only loading for static analysis"? I can't come up with one that would depend on System.Reflection and have a problem with unavailable dependencies.

    I think the lazy loading changes largely could have been implemented transparently inside of System.Type. With the exception of array-based properties.

    But why would anyone only need a part of that array? Caller always either check the length (!= 0) or they iterate all members to find all matching stuff. Why would anyone stop (significantly!) earlier? When searching for the first match on average half of the collection would have been saved. And that is the most favorable example I can come up with.

    The hierarchy split across the project types/targets is awful. The different .NET targets are a mess even today. Portable libraries are really only portable by disallowing half of the framework! This is awful for developers. Never know where a member is available, in what versions and targets. Now we even change the type hierarchy of a core API.

    Is there someone centrally planning this stuff? Seems like there isn't.

  • @tobi: We picked IEnumerable<T> exactly because it is the least rich interface. Consider the DeclaredMethods property on TypeInfo. IList<T> would have suggested you could add to that list (you can't), and the index would have suggested you could rely on order (we deliberately do not want you to rely on that order).

    For your static analysis apps question, we want to enable building apps like Reflector, where you can load one assembly at a time and then the user may ask for the dependencies of an API. Suppose that our dll was compiled against .NET 4.5, but you don't have 4.5 yet and don't want to upgrade just yet. Ideally, you could see the list of APIs required by this dll without having to resolve them to implementation and without requiring the presence of 4.5 dlls (installed or otherwise).

    On transparently implementing lazy loading in Type; the main idea is control over when and where dependencies are loaded from. We opted for a design where the control is explicit rather than transparent. This is mostly based on feedback from compiler or other metadata tools (like the architecture explorer in VS).

    On the array question, suppose all I care about is finding types marked with some attribute. Arrays have eager semantics in .NET. This means that for every API I look for that attribute, I'd have to first get them all. That'd be quite inefficient, hence the preference for IEnumerable<T>

    For your last point, please refer to my earlier reply on this post.

  • Makes sense. Thanks.

  • I think it's a great change and I applaud that you are still willing to make major changes to core APIs. The new model is a lot more consistent and cleaner. Even though I wish to have .NET 4.5 and the Windows Store API to behave the same I understand that MS was unwilling to break compatibility and since it's an in-place upgrade it obviously can't break any existing code.

    I only have 2 wishes. First of all you should consider marking methods from the old model as obsolete in the next Framework version so that people are encouraged to change to the new model and second please avoid extensive use of extension methods. In this case it should have been easier to create a TypeInfo by passing a Type in the constructor or creating a factory method. Extension methods should be the exception, not the rule.

    Thanks!

  • Good. In my experience though even runtime reflection does not work in some common scenarios. Like how do you get all the events defined on a C# or VB class and raise one of them. E.g. I want to create a method which, once given a string containing the name of the event and another the name of the class, use reflection on the class to find the event and raise it passing the correct eventargs to it. It is thoroughly compilcated to achieve this simple goal it seems. C# and VB compilers do not store events the right way it seems. I am not sure. But the reflection methods do not work on events both for .NET 4 and for Silverlight. I presume for all .NET versions. I think the reason is that C# and VB compilers do not store events in the right way but use a private field. I am not sure. Please clarify.

Page 1 of 2 (16 items) 12