Here’s the scenario: A tool that’s designed to add log messages from an application to a SharePoint list.  The log messages are themselves “strings”, which when put in a text file may vary anywhere between 1kb to 10mb in size.  At a single time, this logging process can run for anywhere between 20-30 minutes.  And in a day, it runs at least once every hour.  And the SharePoint OM code customer has used (modified to convey the idea):

 

            using (SPSite site = new SPSite("<sharepoint site url>"))

            {

                using (SPWeb web = site.OpenWeb())

                {

                    SPListItemCollection listItems = web.Lists["MyCustomList"].Items;

                    SPListItem listItem = listItems.Add();

                    listItem["Title"] = "title value 5001";

                    listItem["col1"] = "col1 value 5001";

                    listItem["col2"] = "col2 value 5001";

                    listItem.Update();

                    Console.WriteLine("Added item # 5001");

                }

            }

 

So far, it looks good, but how many items are present in their list? ~ 5000.  These were problem statements from a customer of mine.  The above code is deployed as a windows service and lately throws “Out of memory” exceptions.  Worst! the exceptions are unmanaged.  The process just eats up memory continuously until it hits an “Out of memory” and terminates (crashes).

 

The code, when looked at from “outside” does not seem to arise any suspicion.  The SPSite and SPWeb are being properly used and disposed per Best Practices: Using Disposable Windows SharePoint Services Objects article.  But, have a look at:

 

SPListItemCollection listItems = web.Lists["MyCustomList"].Items;

 

Hum, now this call is really suspicious.  web.Lists["MyCustomList"].Items will return the complete list item collection, which in this case is going to be 5000 odd items.  Once, twice – it is going to be okay, but over a period of time, the framework wouldn’t be able to power-lift this heavy a collection.

 

A quick test on my lab environment also proved the same fact.  Collection loaded the first time, but when I hook-up the debugger, it dragged and the item was not added.  Well, if this is the case on a stand-alone test server, I can imagine what impact this could have caused on a production environment of that scale the customer had.

 

It is not a good idea to use the above code snippet to add items to SharePoint lists and libraries.  In this case though, 5000 odd items are clearly on the higher side to what Plan for software boundaries recommends.  That apart, what’s the best way to make this work?  Even if you had 2000 items, will loading that collection into memory be okay?  Absolutely, no!!  If we carefully read through the software boundary recommendations, the list item, number of document library part of it answers specifies on performance from SharePoint UI perspective.  So, from the code part of it, the developer should do it “intelligently”.

 

In this case though, instead of taking the complete collection in memory, we got the collection to pick up a view, load its contents into memory and added the items, which eliminated the “Out of memory” exception.  The code is below (modified to convey the idea):

 

            using (SPSite site = new SPSite("<sharepoint site url>"))

            {

                using (SPWeb web = site.OpenWeb())

                {

                    SPList list = web.Lists["MyCustomList"];

                    SPView view = list.Views["MyCustomView"];

                    SPListItemCollection listItems = list.GetItems(view);

                    SPListItem listItem = listItems.Add();

                    listItem["Title"] = "title value 5003";

                    listItem["col1"] = "col1 value 5003";

                    listItem["col2"] = "col2 value 5003";

                    listItem.Update();

                    Console.WriteLine("Added item # 5003");

                }

            }

 

That did that trick and the statement highlighted above was indeed the issue.  These kind of things could really sneak in (at least I have seen it often) when you write large-scale enterprise applications using SharePoint OM – so watch out!  Documented in SDK: SPListItemCollection Class.