Yet again, I've doubled the number of concurrent Subscribers to 1,200 where each Subscriber is equivalant to a Windows Mobile device.  I used 6 servers running 200 Subscribers each to create client load, plus 3 load-balanced IIS servers, and a separate SQL Server Distributor and Publisher.  Each IIS server had its MAX_THREAD_PER_POOL registry setting set to 10 and had to handle 400 concurrent Subscribers.  With 1,200 concurrent Subscribers contending for resources on every tier, the system performed 10,693 syncs per hour, which is half the 22,401 syncs per hour that I saw when running 600 Subscribers back at the Partner Summit.  That being said, the system held steady at the number of rows it could change and replicate per hour:

  • Rows changed:  12,831,600 per hour  |  307,958,400 per day
  • Bytes per row: 116
  • Data replicated:  1.42 GB per hour  |  34 GB per day

The longest and average sync times jumped significantly over the results I got with 600 concurrent Subscribers:

  • Longest sync time: 17 minutes
  • Shortest sync time: .5 seconds
  • Average sync time: 6 minutes, 38 seconds

Just like in Vegas and at the Partner Summit, the IIS and SQL Servers are chilling out throughout this test with the IIS servers and Distributor using more memory:

  • IIS1: CPU: 8%  |  Mem: 229 MB  |  Network Utilization: 1%
  • IIS2: CPU: 14%  |  Mem: 225 MB  |  Network Utilization: 1%
  • IIS3: CPU: 8%  |  Mem: 226 MB  |  Network Utilization: .89%
  • SQL Distributor: CPU: 5%  | Mem: 2.95 GB  |  Network Utilization: .32%
  • SQL Publisher: CPU: 36%  |  Mem: 4.12 GB  |  Network Utilization: 1%

The obvious takeaway from this test is that we can scale to 1,200 concurrent Subscribers but we're no longer sustaining an upward curve in performance.  In fact, the curve has started heading downward.  Does performance always head northward in a linear fashion forever in other systems?  Of course not, so don't be bummed.  I think a system that can change and replicate almost 14 million rows per hour while accomodating 1,200 Subscribers that are making 1,200 row changes per sync is pretty incredible!  Remember, real-world systems don't have each Windows Mobile device make 1,200 rows changes every time they sync.  My portable data center and test harness is designed to dramatically exceed what one would see in the real world!

I might be crazy, but I don't want to give up on pushing the performance of this system higher without a fight!  Therefore, I'm going to run this test again with 1,200 concurrent Subscribers, except next time I'll use 6 IIS servers.  If you remember, I didn't just double the number of Subscribers replicating against SQL Server, I also doubled the number of Subscribers hitting each IIS server.  I went from 200 to 400.  By adding 3 more IIS servers, I'll be backing the number of Subscribers per IIS server down to 200 again.  Don't be fooled by those low CPU numbers on the 3 IIS servers.  The ISAPI DLL gets all it can handle long before you begin to stress the server as a whole.  Also, reading to and writing from all those .IN and .OUT files creates a lot of I/O.  Stay tuned to see what happens...

- Rob