mfp's two cents

...on Dynamics AX Development!

July, 2011

  • mfp's two cents

    The solution to the element ID problem


    About 3 years ago heading into the design phase of AX 2012 I wrote the Solving the element ID problem post. As the post describes element IDs in AX have constituted an inherent problem in all past AX releases. The problem is so fundamental that when I met with the original Axapta architects (at the Damgaard Data 25 years reunion party) they considered it insolvable without a major rewrite. In AX 2012 we did solve the problem. This post outlines the solution.

    Design pillars

    Before I dive into the solution these are our design pillars:

    1. Backward compatible. We cannot introduce any inconsistencies in business data or meta data.
    2. Minimal impact on customizations. The uptake of required changes must be minimal and easily absorbed when upgrading to AX 2012.
    3. Completed by AX 2012 RTM. We do not desire a staged multi-release solution to this problem.

    The solution part A - 32 bit IDs

    First we needed to break the upper limit of 60.000 IDs. We decided to make IDs 32 bit (used to be 16 bit). The primary deciding factor for int32 (and not int64 or guid or str) was that there would be no X++ uptake cost of using an int32, as the int data type in X++ already is a 32 bit integer - in other words all X++ code (and table fields) handling/containing IDs would simply just continue to work untouched. In the SYS layer we have >500 fields containing IDs. Not having to convert these (e.g. to GUIDs), and provide upgrade scripts is highly desirable. 

    As the ID for table fields already were 32 bit (16 bit for the element ID and 16 bit for the array index) we decided to keep the field IDs as is. In other words the element ID for fields are still just 16 bit. Given these are scoped by the table, there are no upper limit concerns involved here.

    Besides a lot of kernel refactoring there was one significant X++ impact. The intrinsic function TypeID had to be deprecated. It returned a 32 bit integer where the lower 16 bits contained the Type (either Types::Enum or Types:UserType), and the upper 16 bits the element ID. This function was primarily used in relation to the Dialog framework. The solutionwas to change the dialog framework to use the name of the type instead of the bit mangled integer.

    The solution part B - Installation specific IDs

    To solve the uniqueness problem, we need to ensure that two (or more) solutions can be implemented independently, and installed on the same system without any ID conflicts. In AX 2009 the assignment of IDs happens at creation time (i.e. when the element is created in the AOT). In AX 2012 we are deferring the assignment of IDs to installation time. This means that the model file containing the elements does not contain any element IDs. The IDs are being assigned when the model is imported by AxUtil. The consequence of this is that the same class will have different IDs on different installations. We call it "Installation specific IDs".

    Caution: Deleting a model and reimporting it (or a newer version of it) will randomize IDs and thus cause data integrity issues. The right procedure when upgrading a model is to not delete the model, but just import on top of the existing model.

    Given the requirement to be backwards compatible we must ensure that elements released in AX 2009 (and AX 4) retained their IDs. To satisfy this requirement we introduced a new int property on all ID based elements: LegacyID, and assigned it the ID value from past releases. Now we have captured the ID the element used to have in a simple (and editable) property, and we can use it during import.

    As it is vital that an element retain its ID across releases, we need to find a way to support rename scenarios. Consider the scenario where a new AX 2012 class is renamed in AX7. As the class is new it will have an installation specific ID; once the AX7 model is being imported we need to retain the ID - but if the only match criteria is the class name, and the class has been renamed, then we cannot fulfill the requirement. To solve this we introduced a new guid property on all ID based elements and all root-elements: Origin. This property is set when an element is created, and remains static for the lifetime of the elements. It is the element's fingerprint.  Besides enabling invariant IDs across releases for renamed elements the Origin property has proven valuable in data export scenarios, and rename scenarios of newly fine grained meta data (like forms). This will be covered by another blog post soon.

    AxUtil will during import assign IDs based on these rules:

    1. If an element already exists with the same Origin, then replace the element and reuse its ID - else
    2. If an element already exists with the same Type, Name and ParentID, then replace the element and reuse its ID - else
    3. If the import element has a Legacy ID, and the LegacyId is available on the target system, then add the element setting ID = LegacyID - else
    4. Assign a new installation specific ID from a guaranteed free range that will not collide with any LegacyIDs (>60000 for fields, >1000000 for all other elements)

    Notice: IDs are no longer tied to layers, e.g. Layers no longer have an ID range. Nor do you need the Team Server (aka. ID Server) anymore when using a version control system.

    Data can be exported from one installation to another. It is vital that any ID on the source system is mapped to the corresponding ID on the target system. The data import feature has been improved so all columns containing element IDs will be converted automatically (given they are using the proper extended data type). However; we also needed to address eventual IDs in blobs (containers). Data import and model import has no insights into the structure of the blobs. The only solution to this is to replace all ID-based element references in blobs with name-based references. We have done this for:

    • All kernel classes returning a container - e.g. Query.pack(), QueryRun.pack(), Map.pack(), Set.pack() etc.
    • All X++ classes packing element IDs (typically Runbase derived classes). A best practice rule is in place to detect these.
    • All references in meta data - e.g. when a form data source refers to a table.

    Wrapping up

    To solve the ID problem we had to introduce two new id properties - LegacyID and Origin. This may sound like a step in the wrong direction; however, the nature of these new properties are less restricting than that of the element ID property. In fact the only restriction on the new properties is that Origin must be unique - given Origin is a guid that is easy to satisfy. This restriction is enforced at the database level.  All in all the changes outline above has been a significant investment to solve this inherent problem in the least intrusive way possible. Yet it has an impact. My next blog post will describe the impact.

    The most important message is: If you are a new Microsoft Dynamics AX 2012 customer or partner, you do not need to worry (or care) about element IDs.

  • mfp's two cents

    Uptaking installation specific IDs


    In Microsoft Dynamics AX 2012 we have solved the element id problem. The solution is outlined in this post. The solution consists of major changes both having an impact on existing AX 2009 and AX 4 solutions:

    1. Element IDs (like class ID and table ID) are now 32 bit.
    2. Element IDs are installation specific. The same class will have different IDs on different installations.

    When making IDs 32 bit we had to deprecate the TypeId() function. TypeId() took one argument the name of an Base Enum or Extended Data Type. And it returned an integer value, where the upper 16 bits were the ID, and the lower 16 bits denoted if the type was a Base Enum or an Extended Data Type. For example: TypeId(MyEDT) returned 0xC351 000B = (50001 << 16)+11. Naturally such a function wouldn't work with 32 bit IDs. It turned out, that we didn't need a replacement for the TypeId() function, as the name of Base Enums and Extended Data Types are unique across both types. The solution then simply become to update the framework methods that accepted the combined ID from TypeId() with a new name based implementation. And to make it better, it turned out that usage of TypeId() was restricted to very few frameworks. The table below shows a complete list of the usage patterns of TypeId() we found in the SYS layer, and how we updated them. When you are upgrading the compiler will detect all reference to TypeId() for you.

    Dynamics AX 4/2009              Dynamics AX 2012
    // Dialog related
    // When adding fields to a dialog, the types are now identified by Name – not ID.
    dialog.addField(typeid(MyEdt));             dialog.addField(extendedTypeStr(MyEdt)); 
    dialog.addField(typeid(MyEnum));            dialog.addField(enumStr(MyEdt)); 
    dialog.addFieldValue(typeid(MyEdt));        dialog.addFieldValue(extendedTypeStr(MyEdt)); 
    dialog.addFieldValue(typeid(MyEnum));       dialog.addFieldValue(enumStr(MyEdt)); 
    new DialogField(dialog, typeId(MyEdt));     new DialogField(dialog, extendedTypeStr(MyEdt)); 
    new DialogField(dialog, typeId(MyEnum));    new DialogField(dialog, enumStr(MyEnum)); 
    dialog.addField(types::string);             dialog.addField(identifierStr(str)); 
    dialog.addFieldValue(types::string);        dialog.addFieldValue(identifierStr(str)); 
    new DialogField(dialog, types::string);     new DialogField(dialog, identifierStr(str)); 
    // Dict related
    DictField.extendedTypeID() == typeid(MyEdt) DictField.typeID() == extendedTypeNum(MyEdt) 
    DictType.extendedTypeID() == typeid(MyEdt) == extendedTypeNum(MyEdt) 
    // Args related
    args.parmEnumType(typeId(myEnum))           args.parmEnumType(enumNum(myEnum)) 
    //Type related
    typeId2Type(typeId(myEdt))                  typeName2Type(extendedTypeStr(myEdt)) 
    // Other patterns with better alternatives
    // These alternatives can also be used in AX 4 / 2009
    typeId(myEdt) >> 16                         extendedTypeNum(MyEdt) 
    typeId(myEnum) >> 16                        enumNum(MyEnum) 
    typeId2ExtendedTypeId(typeId(myEdt))        extendedTypeNum(MyEdt) 

    As element IDs now are installation specific then we need to provide enough information for data export/import to convert the data correctly between the source and target system. There are 3 things to do:

    1. Do not store IDs in containers,
      If IDs are stored in a non-relational format (like a container), then data export/import cannot identify them, and the IDs will be imported as-is; which means they will resolve incorrectly on the target system. A best practice rule is available to identify IDs in containers.
    2. Use the right Extended Data Types on your fields (TableId, FieldId, ClassId, etc.)
      Data export will convert data using these Extended Data Types into a format that can be correctly resolved on the target system.
    3. For fields containing field IDs
    1. Set the RelatedTable property, OR
    2. Create a new field group with the field referencing the table and the field referencing the field.
  • For data export/import to be able to correctly convert a field ID, it is required that you specify which table the field belongs to. If the field always belongs to the same table, you can use the RelatedTable property to specify it (3.1), OR if the field ID is referencing a field in an arbitrary table, you need to link the table reference and the field reference to each other. You do this by creating a new field group containing the two fields (3.2)


  • mfp's two cents

    Seeing is believing - AX 2012 Creating an ISV Model


    This sneak preview shows how an ISV can create a solution in Microsoft Dynamics AX 2012, including:

    • How to create a new model in AX.
    • Why element IDs are not a concern anymore.
    • Protecting the solution using a licensing scheme.
    • Generating a signed model file for deployment.
    • How to generate licenses for customers.

    This post is also available on Channel 9.

    I've attached the small webApplication I used to demo the capabilites as a Visual Studio 2010 project.


  • mfp's two cents

    Microsoft Dynamics AX 2012 is RTM!


    Microsoft Dynamics AX 2012 is RTM. This means final build of the product has left the R&D team and will soon be general available. Today, you can already access MSDN documentation for Microsoft Dynamics AX 2012 for Developers



    My favorite features in AX 2012:

    I'm proud to have been a part of the extraordinary team building this extraordinary product, executing on a 3 year plan and delivering on time. Since you are reading my blog, I feel confident this product will have a positive impact on your professional life. Starting today!

  • mfp's two cents

    Microsoft Dynamics AX 2012 has achieved the “Compatible with Windows 7” designation


    Microsoft Dynamics AX 2012 has successfully passed compatibility testing and has achieved the “Compatible with Windows 7” designation for both 64-bit and 32-bit Windows 7. The Microsoft Windows Logo program defines a set of compatibility test cases covering areas of installation, security and stability. Microsoft Dynamics AX 2012 is now listed among the products that have been certified on the Microsoft Windows 7 compatibility center site.

  • mfp's two cents

    Microsoft Dynamics AX 2012 Virtual Launch Event


    Attend the virtual launch event to:

    • Hear from Microsoft executives
    • See product demos
    • Listen to customers and partners
    • Watch product tours, based on your business, industry, or role
    • Collaborate with your peers through feeds and community forums
    • Chat with product experts and sales associates

    Make sure to register before the event.

Page 1 of 1 (6 items)