@BugBot review

@amishler - do you have any other suggestions on how we could incrementally compute latency quantiles, so queries don't have to scan the whole table? we can't take on tdigest extensions, and ideally we can implement in sql fully.

@amishler - do you have any other suggestions on how we could incrementally compute latency quantiles, so queries don't have to scan the whole table? we can't take on tdigest extensions, and ideally we can implement in sql fully.

For quantiles you need to maintain the full distribution, so the two options in principle to avoid computing over the whole table are (1) lossy or (2) lossless compression of the latency distribution. Lossy is what this approach does. Lossless would essentially mean maintaining a frequency table with counts per millisecond value - similar to bucketing in that you collapse rows into counts, but you don't bucket over the x-axis. That obviously only saves space if you have a lot of duplicate ms values, and I don't know if it's feasible space-wise for rollups over longer time scales like an hour. You could consider other lossless compression schemes like Huffman encoding but I have no idea if they can be implemented easily in sql.

Basically any lossy approach involves bucketing. The log-scaled buckets seems sensible if the latency distribution is heavily right-skewed (for example approximately log-normal), which seems like is usually the case in practice. You could potentially come up with more principled buckets for specific distributions + specific quantiles of interest, but you'd have to know the distribution in advance. Without that I'd favor this approach.

shifted the bucketing and quantiles logic into rust, now the migration is only for rolling up raw data into minutes/hours tables