More often than not we have scenarios where our application would be based on .net, we would have to communicate with a native dll and you do not have the source code for the native dll  . I have described one such scenario here. When working with one of our customers where they were trying to return a string from the native world their application was crashing . I wrote the sample below and some tools aided me during the process and hence I am sharing my experience . Although the dll or the exe’s functionality works splendidly , we end up crashing the application in such issues. It’s mostly because returning a System.String type from a PInvoke function requires great care. The CLR must transfer the memory from the native representation into the managed one .
The problem though comes with what to do with the native memory that was returned from the native function. The CLR assumes that,

a.The native memory needs to be freed .

b.The native memory was allocated with CoTaskMemAlloc

Therefore it will marshal the string and then call CoTaskMemFree on the native memory blob. Unless you actually allocated this memory with CoTaskMemAlloc this will at best cause a crash in the application.

We had the signature of the function in the native dll which is similar to char *TESTIN(char in[]) . Based on my previous experiences I was able to write the signature to import this function in the c# application . How ever a handy tool here would be to try pinvoke interop assistant . You can generate the signature in both vb.net as well as c# .

There are many other cool features which you can explore .


                                     

Getting back to our sample , I wrote the managed part as below ,

   /// Return Type: char*

            ///in: char*

            [DllImport("ConsoleApplication8.dll", EntryPoint = "TESTIN", CallingConvention = CallingConvention.Cdecl,CharSet = CharSet.Ansi)]

        public static extern string TESTIN( string str);   

        private void button1_Click(object sender, EventArgs e)

        {

         string check ;

         check="buff"; 

          .
          .
          .    

          . // :) The useful code               

         textBox1.Text= TESTIN(check);

          .

          . // Some more useful code     
         }

Now let’s hop over to the native world.
Firstly I will have to declare the dll export which would be some thing like 
extern "C" __declspec(dllexport)char *TESTIN(char in[]);

Moving onto the function definition.
If you remember I had mentioned that the clr assumes memory to be allocated via CoTaskMemAlloc . This is what I did

I allocated a buffer using CoTaskMemAlloc and returned the address of that buffer back to the managed world.

                   char *TESTIN(char in[])

                       {
                                  .    

                                  . // Some useful code

                                  .

                         char testinresult[] = "outputanswer";

                         int len=strlen(testinresult);

                         char* buff = (char*) CoTaskMemAlloc(len+2);

                         buff[0]='\0';                                     

                         StringCchCopyA(buff, len+1, testinresult);

                                   .

                                   .
                                   . // Some useful code
                         return buff;

                                 .
                                 .
                         }   

 

Now I pressed f5 and voila ,
                       

In this approach we are not freeing the memory ,CoTaskMemFree to free the buffer . For the return value, it only needs to allocate a new piece of memory by using CoTaskMemAlloc and return it to the caller. After return, the newly allocated memory is now owned by the CLR. The CLR will first create a new managed string from it and then call CoTaskMemFree to free it.  Memory allocation and de-allocation, the biggest problem is what function to use. There are many to choose from: HeapAlloc/HeapFree, malloc/free, new/delete, and so on. However, since the CLR uses CoTaskMemAlloc/CoTaskMemFree in the non-BSTR case and SysStringAlloc/SysStringAllocByteLen/SysStringFree in the BSTR case, you'll have to use those functions. Otherwise, it is likely that you'll get a memory leak or a crash in certain versions of Windows®.

Alternatively for doing the above, one more approach would be returning IntPtr directly  .Then use Marshal.PtrToString* in order to get to a managed String value.You may still need to free the native memory but that will dependent upon the implementation of native function.

One more approach , to do the above if you have control over the native code would be to wrap the string in a structure and then use Marshal.StructureToPtr Method to pass over to the native world and once back in managed world use Marshal.PtrToStructure to get back the structure. The hyperlinks above have a good sample . However if we do not have control over the native code then you will have to pass the string .

 

Below are some very good resources for interoperability .

Managed/Native Code Interoperability

Best Practices For Managed And Native Code Interoperability

Interop (How Do I in Visual C++)

Marshaling between Managed and Unmanaged Code

Calling Native Functions from Managed Code