What is the significance of the (int) cast in the computation of dLines?

It isn't. I got the question wrong. The real question should have been "What is the significance of the (int) cast in the computation of g_iWheelCarryover?"

The answer is to ensure that the computation is performed with signed integers throughout. If the cast were missing, then the computation would have been unsigned (since mixing signed and unsigned yields unsigned). dLines is a signed integer, let's say it's -1. Multiply by WHEEL_DELTA yields -120. Now convert it to an unsigned integer and you get a number in excess of four billion. Divided by uScroll (typically 3) yields a number greater than one billion, which is incorrect.

Assuming you don't have a high-resolution wheel mouse, how would you test that your sub-detent mouse wheel handling was working properly?

This is an easy one. Insert temporary code at the top of the OnMouseWheel function that says zDelta /= 4. Ta-da, your mouse now has quadruple its original resolution.