Official blog of the development team of StreamInsight.
Who knows what window output policies in StreamInsight LINQ are? Anyone? Well, you have been using them whenever you have used a window. Output policies specify the timestamps of the result of the aggregation that happens on the window. Even if you didn’t really care about that, you had to specify the output policy parameter for the window. In V1.2 we are introducing
Importantly, there is no breaking change. Whatever you used in 1.1 is still working in 1.2, with the same behavior. For count- and snapshot-windows, the single existing output policy is becoming the default:
SnapshotWindow() is equivalent to SnapshotWindow(SnapshotWindowOutputPolicy.Clip)
CountByStartTimeWindow(num) is equivalent to CountByStartTimeWindow(num, CountWindowOutputPolicy.PointAlignToWindowEnd)
Set-based operations on top of snapshot windows always produce an event with the timestamps of the snapshot window itself, while operations on count windows produce a point event at the end of the window.
The only existing output policy for hopping and tumbling windows in StreamInsight 1.1 was to produce the window size as result event lifetime. Now this turned out to be impractical in many cases, especially with overlapping hopping windows. Have a look at the following diagram:
Just like the windows, also the result events overlap. When using this result in a subsequent operation, like a join with another stream, this easily leads to unexpected results. In many cases, only a single aggregation result at each point in time is needed (instead of four concurrently valid payloads in the diagram above). To this end, we have introduced another output policy for hopping (and tumbling) windows, which produces a point event at the end of the window:
This policy is now also the default for hopping windows:
HoppingWindow(size, hop) is equivalent to HoppingWindow(size, hop, HoppingWindowOutputPolicy.PointAlignToWindowEnd)
TumblingWindow(size) is equivalent to TumblingWindow(size, HoppingWindowOutputPolicy.PointAlignToWindowEnd)
This can now easily be converted into a continuous signal, for instance by using the ToSignal() macro that we discussed earlier. This signal has a payload at each point in time and hence is suitable to be joined with another stream, which is not necessarily synchronized in time. Typical example: join the result of a hopping aggregation with some reference stream:
Dealing with point events instead of overlapping intervals in the context of windows helps avoiding unexpected side-effects and—most of the times—corresponds to the intended semantics.
Regards, The StreamInsight Team