If you are in a situation where, one process creates a MUTEX with default security descriptor and other process running in the same user’s context is not able to access it.

You might be facing the problem because of newly introduced integrity mechanism in Windows. Integrity level implemented as a SID and is defined in the access token of the user and as a ACE in the SACL of the resource. A token with low integrity level SID cannot access a resource which has high integrity level. Most of time application would not need to define the integrity level. Windows would automatically derive it from the parent process and assign it to the access token. A low integrity level process cannot communicate to the higher integrity level process.

You could refer to all this in detail at –

Designing Applications to Run at a Low Integrity Level

Windows Integrity Mechanism Design

To determine the integrity level of a process you can use “Process Explorer”. The process which is trying to communicate with the server will present Low access token which would have “high” or “medium” integrity level.

Untrusted < low < medium < high < system is the increasing order of integrity level. Untrusted level is far lower than high, therefore the client’s token would not be allowed to communicate with the server and would result into “Access Denied”.

However, a process is running at high integrity level we can certainly create the resource with low integrity level access so that any Access token with low integrity SID can be compared against the DACL of the resource and client gets access to the resource. This is done by adding an ACE in the SACL of the Security Descriptor of the resource. For example,

S:(ML;;NW;;; S-1-16-0) -> Where S-1-16-0 represents untrusted/anonymous integrity level.

S:(ML;;NW;;;LW) -> Where LW represents low integrity level.

Assume you are using following string security descriptor.

"D:(A;;GA;;;WD)(A;;GA;;;AN)"

You would need to change it to following to add a untrusted integrity ACE.

"D:(A;;GA;;;WD)(A;;GA;;;AN)S:(ML;;NW;;;S-1-16-0)

Or

"D:(A;;GA;;;WD)(A;;GA;;;AN)S:(ML;;NW;;;ME) -> for medium integrity

Or

"D:(A;;GA;;;WD)(A;;GA;;;AN)S:(ML;;NW;;;HI) -> for high integrity

Following code converts the SDDL in the actual security descriptor which you can use to create the Semaphore.

   1: wcscat(szStringSecurityDis,L"D:(A;;GA;;;WD)(A;;GA;;;AN)S:(ML;;NW;;;S-1-16-0)");
   2:  
   3: if (!ConvertStringSecurityDescriptorToSecurityDescriptor(szStringSecurityDis, 
   4:  
   5:       SDDL_REVISION_1, 
   6:  
   7:       &psd, 
   8:  
   9:       NULL))
  10:  
  11:       _tprintf(TEXT("ConvertStringSecurityDescriptorToSecurityDescriptor %u\n"), GetLastError());
  12:  

The above discussion holds good for Mutex, Semaphores, Events, NamedPipes and Shared memory. Any of the above securable objects if created with default security descriptor will get integrity level of the process creating it. For example : If you create any of these from a Windows application and try accessing it from an Active X control loaded in Internet Explorer you will get an access denied. Please not that IE and application would run in the same user’s context (for almost every case). Ideally Active X should be able to access the object. IE is designed to run at low integrity level and the interactive windows application usually runs at medium integrity level, therefore Active X loaded by IE will not be able to access the object. You can work it around two ways

1) Run the windows application at low integrity and then create the securable object.

2) Provide a custom SD to the object such that low integrity process can access it. The method is listed above.

The  integrity enhancement has been introduced in Windows Vista. It will not be recognized in Windows XP and therefore you might get an error 1804, "the specified data type is invalid", at runtime. You would need to design a mechanism programmatically so that, you use integrity mechanism only if you are running Vista onwards. For example, see following pseudo code.

IF ( OSVERSION >= Vista )

                SDSTRING = "D:(A;;GA;;;WD)(A;;GA;;;AN)S:(ML;;NW;;;S-1-16-0)"

ELSE

                SDSTRING = "D:(A;;GA;;;WD)(A;;GA;;;AN) "

ENDIF

The above pseudo code of server ensures that clients running on Vista and XP will be able to connect it.

GetVersionEx will help you determining the version of the OS at run time.

-Prateek