Say you have a large flatfile in an application that does messaging only.   You drop a single file and you notice elevated cpu for the receive host for several minutes.    What’s happening in the host is that a single messaging thread is using the ffdasm to disassemble the flat file.   This is a very cpu intensive activity

Now you drop 100 large flatfiles.  Since messaging engine threads per cpu is set to 20, if you have a two processors, then there will be 40 messaging engine threads created and operating on 40 inbound flatfiles.    cpu still goes to 100%, but it takes longer for the first flatfile to be processed because other threads are taking valuable cpu resources.  

In testing on a two processor system I was able to get the best end to end performance by specifying that only one messaging engine thread per cpu be used on the receive host when performance testing with 100 files.  Tuning messaging engine threads per cpu to a lower value results in faster processing of the first file through the system and additionally faster overall processing.

1 messaging engine thread per cpu on the receive host about 4 minutes to process the first file.  1 hour and 30 minutes for all files

10/21/2011  02:46 PM                 set messaging engine threads to 1 for full test.txt

10/21/2011  02:50 PM         2,752,368 {FC0AD608-1F76-43A2-B7D5-44DDB66E66DF}.xml

10/21/2011  04:16 PM         2,752,368 {F7241E83-C7F1-428D-A5DD-2582CF8ABE20}.xml

 

2 messaging engine threads per cpu on the receive host about 4 minutes to process the first file.  1 hour 42 minutes to finish all approx..

10/19/2011  04:29 PM               set messaging engine threads to 2 for full test.txt

10/19/2011  04:33 PM         2,752,368 {C29DB82A-98E8-4B41-B917-093DDC708D21}.xml

10/19/2011  05:02 PM         2,752,368 {E9AFFEC2-B60B-446C-9D63-1BBBB2B2FD12}.xml  33rd file in 34 minutes.  Had to stop test early summary

extrapolated.

 

4 messaging engine threads per cpu on the receive host  10 minutes time to first message and 1 hour and 39 minutes total processing time.

10/19/2011  06:06 PM                 set messaging engine threads to 4 for full test.txt

10/19/2011  06:16 PM         2,752,368 {E96CD77C-8D24-4BB9-A111-871594B06F3F}.xml

10/19/2011  07:45 PM         2,752,368 {2F2EE1F9-C9F0-44A9-B2D1-1A02BC9F7EAD}.xml

 

20 messaging engine threads per cpu on the receive host (the default setting) 18 minutes.  1 hour and 42 minutes for all files to be processed.

10/20/2011  09:35 AM                 set messaging engine threads to 20 for full test.txt

10/20/2011  09:53 AM         2,752,368 {9BA6A42F-A634-4FEF-A29E-F16A774D823F}.xml

10/20/2011  11:17 AM         2,752,368 {262F3035-CACF-423A-BDFA-5EA817D2639B}.xml

 

50 messaging engine threads per cpu on the receive host  23 minutes before first file is processed.  1 hour 54 minutes for all files to be processed.

10/20/2011  02:56 PM                 set messaging engine threads to 50 for full test.txt

10/20/2011  03:19 PM         2,752,368 {40BF75AF-3935-4DC1-99C4-048CD61421D1}.xml

10/20/2011  04:50 PM         2,752,368 {5B72CC56-FB61-4A33-9186-3A3F954AC38A}.xml