Using Both Field and FieldRef in InfoPath List Loses Data

It appears that if you have both a Field and a FieldRef definition in your SharePoint list definition file that includes XML (InfoPath) field promotion, you can no longer update the field through the list; you must do it through the XML only.

I ran into this interesting bug that took quite some time to track down.  We have several site definitions that have a number of SharePoint custom lists that use a shared set of site columns.  One of the custom lists is a document library containing InfoPath XML files that are edited using Web-based forms.  The list definition includes fields that are "promoted" (synchronized with) the InfoPath XML file such as:

 <Field ID="{12345678-9ABC-DEF0-1234-56789ABCDEF0}"
       Type="Text" Name="Sample" StaticName="Sample"
       ColName="nvarchar11" DisplayName="Sample"
       Description="A Sample Field" Group="Samples"
       Required="FALSE" Hidden="FALSE" Customization=""
       ReadOnly="TRUE" RowOrdinal="0"
       Node="/my:myFields/my:Sample"
       SourceID="{A1234567-89AB-CDEF-0123-456789ABCDEF}" />

Note that the Sample field is defined from a site Column with the same ID as the Field.  The site column is defined in the feature identified by the SourceID GUID.  Also, the Node field defines where the value is promoted from (synchronized with) in the XML document.  For an extensive look at using XML with SharePoint lists, see Andrew May's 5-part blog on this topic: https://blogs.msdn.com/andrew_may/archive/2006/06/29/SharePointBeta2XMLDocumentParsers1.aspx.

A nice feature of SharePoint is that when you define a field this way, you can update the XML file and the column value is updated, or you can update the column value and the XML file will be updated.  Except in our case, it wasn't working correctly.  If the XML value were updated, the column value would be updated, but not the other way around.

We had code that would get a value from another system and update the column in the list.  The code would appear to work, without raising any errors, but neither the column value nor the value in XML would be updated.  No errors were logged anywhere that we could find.

After going around in circles trying different ways in code for updating the value, I finally thought to check the schema file for that particular list and compare it against another site's list definition that was working.  It turns out that the field we were trying to update had FieldRef in the <ContentTypes> section of the list definition.

 <FieldRef ID="{12345678-9ABC-DEF0-1234-56789ABCDEF0}"
          Name="Sample" DisplayName="Sample"
          Required="FALSE" Hidden="FALSE"
          Customization="" ReadOnly="TRUE"
          Node="/my:myFields/my:Sample" />

The other site where it was working only had a Field definition, not the FieldRef.  After commenting out the FieldRef and creating a new list from that definition, it started working.  I still have no explanation about why it was causing the problem, but it is working.

The last thing we had to do was to fix existing lists.  I wrote some code to open the list, copy the existing values from the column, delete the field, add the field back in using the site column definition and then restore the previous values to the column.