+++
Fault Injection is the act of artificially changing the behavior of an existing executable code to simulate various faults. FI is very useful for validation of error handling code paths and for improving code coverage.
There are several types of fault injection. In runtime fault injection, the fault injecting test modifies the execution logic of the application under test (AUT), by injecting faults, triggered by specific runtime conditions. One could for example implement a FI test with the following semantic:
Throw an out-of-memory (OOM) exception, whenever the application calls method CreateWidget of class WidgetManager.
The FI terminology is as follows:
TestApi provides a simple, but powerful runtime fault injection API for injecting faults in managed code. The API was originally designed and implemented by Bill Liu et al from the “Essential Business Server” team, and adapted to TestApi by Sam Terilli from our WPF XAML team. The following content provides a quick introduction to the API.
Following is the code of a trivial AUT that we will use for demonstration purposes:
// // This is a sample application used for demonstration purposes. // using System; class MyApplication { static void Main(string[] args) { int a = 2; int b = 3; for (int i = 0; i < 10; i++) { Console.WriteLine("{0}) {1} + {2} = {3}", i, a, b, Sum(a, b)); } } private static int Sum(int a, int b) { return a + b; } }
The result of running this application is of course:
> MyApplication.exe 0) 2 + 3 = 5 1) 2 + 3 = 5 2) 2 + 3 = 5 3) 2 + 3 = 5 4) 2 + 3 = 5 5) 2 + 3 = 5 6) 2 + 3 = 5 7) 2 + 3 = 5 8) 2 + 3 = 5 9) 2 + 3 = 5
Now, let’s try to inject a fault in the AUT. Let’s assume that we want to modify the return value of Sum. Here’s how we can accomplish that:
// // Simple fault injection test // using System; using System.Diagnostics; using Microsoft.Test.FaultInjection; public class FaultInjectionTest { public static void Main() { // // Set up a fault rule to return –1000 the second time Sum is called. // string method = "MyApplication.Sum(int,int)"; ICondition condition = BuiltInConditions.TriggerOnNthCall(2); IFault fault = BuiltInFaults.ReturnValueFault(-1000); FaultRule rule = new FaultRule(method, condition, fault); // // Establish a session, injecting the faults defined by the fault rule(s) // FaultSession session = new FaultSession(rule); ProcessStartInfo psi = session.GetProcessStartInfo(@".\MyApplication.exe"); // // Launch the target process and observe faults // Process p = Process.Start(psi); p.WaitForExit(); } }
Fairly straightforward. Upon running the test (which will itself spawn the AUT) we observe the following output:
> FaultInjectionTest.exe 0) 2 + 3 = 5 1) 2 + 3 = -1000 2) 2 + 3 = 5 3) 2 + 3 = 5 4) 2 + 3 = 5 5) 2 + 3 = 5 6) 2 + 3 = 5 7) 2 + 3 = 5 8) 2 + 3 = 5 9) 2 + 3 = 5
As intended, we injected a runtime fault in MyApplication.exe, which resulted in Sum returning –1000 the second time it got called.
Under the covers, the managed code Fault Injection API uses the CLR profiling API to modify the prologue of the intercepted method at runtime in order to inject the desired fault. The injected prologue instructions essentially call a method in the library, which then dispatches the call to the specified fault.
Because faults are injected at runtime, the code of the original application is not modified in any way. There is a certain performance degradation, which depends on the number of the injected faults.
The fault injection API provides a variety of built-in conditions and faults (in the BuiltInConditions and BuiltInFaults classes respectively). Users of the API can also create custom conditions and faults (by implementing the ICondition and IFault interfaces respectively). The API also provides a set of classes that expose the ability to fine tune and monitor the injected faults.
In addition, the API provides a facility to set “global faults”, which is useful for server application testing, where one application typically consists of and recycles many different processes.
I have attached the sample above, which should get you up and running with fault injection.
I am trying to use the FaultInjection libraries to simulate data access failures.
Is this proper method signature to get the ExecuteNonQuery() method on DbCommand?
Dim method = "System.Data.Common.DbCommand.ExecuteNonQuery()"
Thanks.
Yep, this should work.
Let me know if you have trouble and we will help.
I had success when using the fault injection libraries on some sample code I had written, but now that I'm trying to use it against our intended target code it doesn't seem to be working.
I want to use the ReturnValueFault on an internal static method which returns an int. The method is within an internal sealed class with the SecuritySafeCritical attribute set.
I'm creating the fault rule like this:
FaultRule rule = new FaultRule("static FullNamespace.ClassName.GetDeviceId(uint, IntPtr, IntPtr, IntPtr)", BuiltInConditions.TriggerOnEveryCall, BuiltInFaults.ReturnValueFault(1000));
I also tried prepending out to IntPtr as these are specified as out parameters in the method signature. I didn't have success either way. Am I doing something wrong here?
My other thought is that this class and method are in a library/dll, and not within the exe code itself which is the process I'm starting a fault injection session for.
Any help is greatly appreciated. So far I've been very happy with the low cost of entry for using this API and the good documentation.
Hi David, can you try to replace "IntPtr" with "System.IntPtr" in the parameters list of the method? the data type of the parameter need to be fully qualified name too.
let me know if it still doesn't work.
I tried System.IntPtr and had no luck here either. Is it an issue if the binary I am trying to inject into is signed?
no, the issue is not related to the sign. i noticed that you mention about 'out parameter'. please make sure that 'out' is also part of the method signature. i.e your method signature should be:
static FullNamespace.ClassName.GetDeviceId(uint, out System.IntPtr,out System.IntPtr,out System.IntPtr)
i wrote a dummy app with the same method signature above, and the fault injection api works for me without problem. if this still doesn't work for you. most likely reason here is that the method signature still not correct somehow, i.e it does not match what clr expecting in the runtime). a few thing you can duble check:
1. make sure the method got invoked.
2. make sure the method signature is correct or no typo (i often see people has typo in the method signature by mistakes).
3. in the %temp% folder, there should have some log, check if there are some error.
I have tried with the out parameter and also it did not work. I coulnd't find any logs in %temp% folder. I realized today that the AUT is a native program that runs the code from the managed assemblies by hosting a CLR runtime. Is this my problem, is there a way to still use TestApi fault injection or will I have to use some native fault injection library instead?
ha. that's the problem. when i saw the IntPtr parameter at the first place, i suspected that your aut might be using navtive code or p/invoke stuff which is out scope of this tool that targets to managed code only. in you case, you have to use native fault injection tool or windows API hooker stuff.
good luck.
Darn, this would have been the perfect solution. Thanks for your help!
I am just getting around to responding to Ivan's response to my original comment.
Ivan, I am unable to get ExecuteNonQuery() to throw an exception for me. Here is all my code from within a (slightly verbose) unit test:
'Dim condition = BuiltInConditions.TriggerOnNthCall(2)
Dim condition = BuiltInConditions.TriggerOnEveryCall
Const ExceptionMessage As String = "This is a fault-injected exception."
Dim fault = BuiltInFaults.ThrowExceptionFault(New DataException(ExceptionMessage))
Dim faultRule = New FaultRule(method, condition, fault)
Dim s = New FaultSession(faultRule)
Dim rp = TestDataFactory.CreateYrtReinsurancePolicy(PolicyNumberFactory.CreateUnique())
Dim isExceptionThrown As Boolean
Try
rp.Save()
Catch ex As DataException
If ex.Message.Equals(ExceptionMessage) Then
isExceptionThrown = True
End If
End Try
Assert.IsTrue(isExceptionThrown)
Note that I gave up on the NthCall and went with EveryCall just to see if I could get it to work.
Any ideas?
if this is your complete version of the code, the fault injection won't work, because i only see you set FaultSession, but not call Launch method. referring to the document/samples, in order to have fault injection works, you have to provide an executable which execute your target API (ExecuteNonQuery, in your case), and you set the fault injection rule and call launch method to launch your executable.
I have a private version to better support unit test, but probably take a while to release publicly. for now, this is the high level work flow (suppose you want to inject fault into function Foo()).
1. write an app to call Foo().
2. set fault injeciton rule and session, as you did above.
4. call Launch method to run your app. and when your app call Foo(), the fault will be injected.
let me know if you have any questions.
The problem with that workflow is that I would need to write a process wrapper for each and every unit test that I write. I think my solution file will get out of control quickly.
Given that, I think that I will wait for your private version.
Thanks for the clarification. I just didn't infer that launching an additional process was necessary.
Hi,
When i try to add the FaultInjectionEngine.dll as a reference to my code in VisualC#, it is giving an error which says, "Please make sure that the file is accessible and that it is a valid assembly or COM object". Please help me on this.
In your VS project, you should add reference to TestApiCore.dll, not faultinjectionengine.dll.
Yah did the same, Still the same error crops in.