Configuring diagnostics.wadcfg to Capture Custom Log Files

Configuring diagnostics.wadcfg to Capture Custom Log Files

  • Comments 3

Windows Azure Diagnostics (WAD) has the ability to copy files to Azure blob storage.  The feature is meant for copying custom log files but you can use it to copy any file that WAD has permission to access.  To configure this feature via diagnostics.wadcfg add appropriate <DataSource> and <DirectoryConfiguration> elements to the XML.

Within <DirectoryConfiguration> there is the option to use <Absolute> or <LocalResource>.  The local resource approach is appropriate in most situations since absolute paths are harder to define which exist in both the DevFabric and Azure environments.  The MSDN documentation covering diagnostics.wadcfg shows the use of an absolute path to copy the logs from a %SystemRoot% location.

For custom logs use the local resource approach.  The configuration steps are:

  1. Define a <LocalStorage> element in ServiceDefinition.csdef.  Set the name attribute to any value desired.  Set the sizeInMB attribute value large enough to hold the anticipated log data.  Set cleanOnRoleRecycle to false for reasons discussed below.
  2. Add a <DirectoryConfiguration> element to diagnostics.wadcfg.  Set the container attribute value to the name of the Azure storage blob container you want WAD to use.  WAD automatically creates the blob container if it does not exist.  Set the directoryQuotaInMB attribute value to the same value used for the <LocalStorage> sizeInMB value in step 1.
  3. Add a <LocalResource> element within <DirectoryConfiguration>.  Set the name attribute to the same value used for the name attribute of the <LocalStorage> element in step 1.  Set the relativePath attribute value, “.” means from the root of the local storage.

    Warning:  Not specifying the relativePath attribute or setting the value to an empty string causes WAD to ignore the entire <DirectoryConfiguration> element.

Here is a snippet from within ServiceDefinition.csdef:

<LocalResources>
<LocalStorage name="CustomLogs" cleanOnRoleRecycle="false" sizeInMB="128" />
</LocalResources>

Here is the corresponding snippet from within diagnostics.wadcfg:

<DataSources>
<DirectoryConfiguration container="wad-custom" directoryQuotaInMB="128">
<LocalResource name="CustomLogs" relativePath="." />
</DirectoryConfiguration>
</DataSources>

WAD does not delete or remove custom files, it just persists them to blob storage, so you’ll need to implement a cleanup strategy.  This is actually a good feature considering it may be inappropriate to delete the files.

Regarding setting cleanOnRoleRecycle to false, I suspect there are role recycle scenarios during which WAD will not have had time to transfer the files.  If the files are left on the disk then WAD will transfer them after the recycle.  This is another reason for implementing your own cleanup strategy.

Finally, there is some difference of opinion as to WAD needing an exclusive lock on the files it copies.  I believe some logging code uses file writing techniques which prevent WAD from performing the copy.  Here are some links about the issue:

http://archive.msdn.microsoft.com/azurediag

http://blog.bareweb.eu/2011/01/implementing-azure-diagnostics-with-sdk-v1-3/

http://social.msdn.microsoft.com/Forums/en-US/windowsazure/thread/f4d880c0-3d79-4ddd-af56-2be9bba88a94/

For more posts in my WAD series:

http://blogs.msdn.com/b/davidhardin/archive/tags/wad/

  • What happens when the directoryQuotaInMB is exceeded - roles will be unable to write to that directory?

  • Your best bet is to test what happens but I believe the configured sizeInMB of the LocalStorage element sets a hard limit on how much Azure will allow you to store.  The directoryQuotaInMB along with other WAD quotas are soft limits which WAD can exceed temporarily due to its own processing.

    With the settings I show above you'd only be able to write 128 MB before needing to clean the folder mainly due to the sizeInMB value.

    Take a look at the second half of this blog for additional info:

    blogs.msdn.com/.../configuring-wad-via-the-diagnostics-wadcfg-config-file.aspx

  • The above description works fine for me. - BUT -

    When I change the settings in Visual Studio 2013 with the Windwos Azure Tool V2.2.11003.1601 it changes the line

           <LocalResource name="CustomTrace" relativePath="."/>

    to

           <Absolute path="[CustomTrace]\." expandEnvironment="false" />

    How to reproduce: Go to the properties of the web/worker role. From the "Configuration" tab under "Diagnostics" edit the "Custom plan". Just clicking OK will do.

    When deploying with this change, the custom log files will no longer be transferred to the storage account. I have to manually undo the changes in that line in order to make things work again!

    This issue only occurs when you run it in the Compute Emulator or create a new VM during deploying. Deploying over an already running VM will not take over these changes and you can´t see the effect because the settings are already stored in table storage and the new settings will be ignored.

Page 1 of 1 (3 items)
Leave a Comment
  • Please add 4 and 4 and type the answer here:
  • Post