There are few constraints on memory usage in SQLCLR modules. This freedom is one of the benefits of the CLR, however, care should be taken to implement code that integrates well into a scalable database environment. I want to discuss two points in this regard.
Firstly, large allocations should be avoided. The limit to keep in mind is 85,000 bytes. Any single allocation greater than 85,000 bytes will end up on the Large Object Heap. For a closer look at what the LOH is please see the following blog post:
http://blogs.msdn.com/maoni/archive/2006/04/18/large-object-heap.aspx
Essentially, objects on the LOH will not be managed nearly as efficiently as smaller allocations, and because of the fact that the LOH is not compacted, fragmentation can become a real problem. Not only is this a performance concern, but out of memory issues may also become a problem.
There are obvious large allocations to be avoided, but some are a bit more subtle. For example, imagine you need to create a multi-dimensional array. You could do this with a jagged array or a rectangular array. While the rectangular array may have better performance in many situations because the memory is continguous, if it is large enough to be allocated on the LOH there may be scalability problems.
Secondly, stream data when possible. When data is passed from SQL to the CLR by value, the memory allocations are often very large. If data can be chunked and streamed into a smaller buffer, the server will scale much better with load. One important area to consider this is LOB data. For example, if you have a scalar function that takes an nvarchar(max) parameter there are two ways to pass the data. One way is SqlString which copies the entire value in memory. The other way is SqlChars which provides a buffered, streaming interface.
Another common usage of SQLCLR is for XML processing. Many of the System.Xml methods make entire copies of the XML value. One example of this is the XPathNodeIterator class. It can be very handy for extracting xml nodes and values, but if the document is large it will likely not scale well. A more scalable solution would be to use the streaming interface provided by SQL and accessed through SqlXml.CreateReader(). The code to navigate the nodes and extract values will likely be more complex, but if performance is a concern it will be worth it.
There are of course many more ways to misuse memory, and I welcome any other examples or questions.