Got basic timeseries with linestack working. I've also got some code snippets for interpolating to display heatmap with non-uniformly sampled timeseries data. I should be able to have this fully working with time series very soon :D

Got heatmap to display timeseries working. It should also work with non-uniformly sampled data by interpolating, need to test.

Also need to implementing switching between heatmap and line representations, need to delete the graphic when switching.

So timeseries can be represented with arrays of one of the following shapes (let's ignore x-axis values for now).

If we have:

l: number of timeseries
p: number of datapoints in a timeseries

We can have the following shapes:

p: only y-values
p, 2: yz vals
l, p: l-timeseries with y values
l, p 2: l-timeseries with yz values

Extended to n-dimensional arrays (for example, trajectories projected onto principal components?). If each non-timeseries dim is $d_1, d_2, ... d_n$, then the above becomes:

I don't think we can auto-detect if l is present or not and the user should specify, something like:

multi_timeseries: bool = True

Scatters can be similar to some cases of nd-lines 🤔 , but x values would be directly specified and the current index is parametric (example with time indicating position in a low dim space). This would actually be true for lines as well if representing trajectories.

So for nd-line maybe we have two versions, parametric (y and z are not functions of x, but x, y, z are a function of some other dim) and non-parametric (simple timeseries lines where y and z are functions of x).

I made a more generalist NDPositions which can map data that is:

[s1, s2, ... sn, l, p, 2 | 3]

where:
s1, s2, .... sn are slider dims
l is number of lines or number of scatters, this dimension is optional and the user must specify whether or not it exists
2 | 3 is the last dim, indicating xy or xyz positions.

It can map arrays of these dims to a line, line collection, line stack, scatter, or list of scatters (similar to multi-line).

I think this is a much more elegant way to deal with things, and NDTimeSeries is not necessary. The user can provide a slider mapping (to map from reference units to array index) for the p dimension which is the same as the "x-axis" for time series data!

Example if we have data that is [n_timepoints, 2], and the x-positions here (in the last dim) are in seconds. The NDWidget reference units for the slider can also be in seconds, and we can provide a mapping function that goes from the slider reference units to the n_timepoints index.

I think we can also use this for heatmaps and interpolation. Use the reference units to determine a uniform x-range for the current display window, and we can interpolate using [n_timepoints, 2] data.

EDIT: I think that the NDPositions will also work for PolygonGraphic ! Can think about meshes in general later.

For positions graphics, I should actually do [n_datapoints, n_lines, 2 | 3] so everything before the last 1 or 2 dims is always sliders.

Some more ideas:

Allow any 2-3 dims to be used as the graphic dimensions and specify the slider dims. This would also allow using named dims (such as those used in xarray).
API like:

add_nd_<scatter|lines|heatmap>(
  data=<array>, # array like, defaults to last 2-3 dims are graphical, first (n - 2 | 3) dims are sliders and in order
  x=<int | str | None>  # optionally specify x graphical dim using the integer dim index or dim name
  y = ... # same for other graphical dims
  slider_dims=<None | tuple[int | str]> # specify slider dims, None is auto first (n - 2 | 3), or specify a tuple of slider dims as int or str, examples: (0, 1, 2, 3), ("time", "plane")
  ... # other args
)

We interpret the given order of the slider_dims passed as $$s_1, s_2, ... s_n$$, regardless of their order within the actual array. This will make it clear to users which dims they are syncing when they use the sliders.

EDIT: A limitation of the above is that a user can't collapse multiple "graphic/display dimensions" into "final graphic/display dimensions" if they're hard-coded this like. So something like:

add_nd_<...>(
  data=<array>,
  display_dims=<tuple[int | str]>, # specify display dims in [x, y, z] order, OR display dims that collapse to xyz after the finalizer function
...
)

An example for images would be collapsing [z, m, n] to display a projection over z as slider dims are moved. But we could also have > 3 dims that are used, and then collapsed to 3 or fewer dims for the "final graphic/display dims".

We can use LineCollections to display multiple lines, like behavior tracks of keypoints, with shape [n_keypoints, n_timepoints, 2]. This works well with NDPositions and display windows.

I was thinking of what's the best way to show a scatter for each keypoint, and I think I should make a ScatterCollection that behaves like a LineCollection so the same array with the same shape can be given, the only difference is the graphical representation would be a scatter instead of lines. For typical behavior keypoint viz, the display_window would usually be just 1, but it can be greater than 1 for any viz that needs to show a window of scatter points.

ok I think stuff is working

I think I need to make a PolygonCollection too 🤔 . Would be very similar to the ScatterCollection.

so the basics are all working 🥳

LineStack and heatmap representations are swappable:

Scatter and line collection to show behavior trajectories:

A set of imgui UIs that allow controlling some aspects of the "nd graphics" could be useful, such as:

• graphical representation, dropdown menu to choose scatter, line, heatmap
• display_window
• multi bool

Stuff I should finish before implementing the orchestrator:

• merge auto-replace buffers #974 , so I can implement changing the display_window
• implement logic to allow the display_window to be one of:
  • centered at the index, "center" mode
  • start at the index, "start" mode
  • end at the index, "end" mode
• adding a linear selector, it's "default" positions and its position when the main slider for the n_datapoints dimension changes should correspond whether the display_window is in "center", "start" or "end" mode. Moving the linear selector should probably change the index in all other NDobjects but not its "parent" object. So that the visual representation of the parent object doesn't change, ex: if the linear selector is moved we don't want a linestack to also "move"
• spatial function
• implement mapping from a slider reference index with units (such as time) to array index.
• figure out how to implement stuff like colors, cmaps given that we're essentially performing out of core rendering with these graphics
• auto display_window? Zoom in/out and set graphic data at different subsample levels

Things that make me "uncomfortable" that I need to settle:

dim shapes

Dim shape for nd-positions is [s1, s2, ... sn, n, p, 2 | 3] where:

n: number of lines, scatters, or heatmap rows (optional, can be 1)
p: number of datapoints, often corresponds to a time index.

p would be a slider dim, but there's this n dim that's in between p and all the other slider dims. We could instead use a shape of [s1, ... sn, p, n, 2 | 3] but that feels weird and we'd have to do a transpose to get the array for the graphical representation, i.e. [p, n, 2 | 3] -> [n, p, 2 | 3] is required for the graphical representation.

Do we just document this well, that the n dimension is not a slider dim but p is?

When using nd-positions data in conjunction with nd-image data, we'd have something like this:

# nd-positions
[s1, ... sn, n, p, 2 | 3]

# nd-image
[s1, ... sn, p, r, c, 1 | 3 | 4]

Where r: rows, cols: columns. 1 | 3 | 4 denotes grayscale or RGB(A).

The p dimension in the nd-image array above would correspond to the p in the nd-positions. nd-images don't have any n dimension, it doesn't make sense there.

Working on "implement mapping from a slider reference index with units (such as time) to array index.", which requires proper implementation of slider_dims and n_slider_dims properties. Need to figure out how to properly separate the p n_datapoints dimension from other slider dims, and also apply the window funcs on the p dim. Will do tomorrow.

Window funcs working for p (n_datapoints) dim and all graphical representations

The videos all look really cool!

I like that you can arbitrarily change the window size

I like the switching between LineStack and heatmap too

Some of the names are a bit long/weird; I would maybe consider making the property setters that users would interact with shorter/simpler

Other things that stand out to me:

• Setting the indices with a tuple (change["new"])
• Same thing for a setting the window func (I think in the past we have done a dictionary approach to represent this kind of mapping)

Some thoughts:

Need to have a max limit beyond which window funcs are ignored. Now with OOC we can view timeseries that is tens of GBs+ in size, but we wouldn't want to put the display_window and the entire dataset AND apply a window function using all the data, it would be very slow. At the same time, it doesn't make sense to apply the window_func on a subsample of the full data, because then the window_size is no longer valid. So we should only use the window_func below a certain settable limit for n_datapoints. Maybe by default it can be max_p * some_number.

figure out how to implement stuff like colors, cmaps given that we're essentially performing out of core rendering with these graphics

I think that if a cmap_transform is provided for the "global" dataset, it should look correct even within the display windows.

ok next I'm going to an "auto dynamic display_window" which sets the display_window based on the current x-range. Panning and zooming will set the display_window. This also makes the linear selector management much easier! This will be specifically useful for timeseries data.

I think the way to do this is:

• Animation function to poll current extent, this is is world space. Ignore if unchanged.
• Since we already map ND data from its own "model index" to array index, the world space is actually identical to the ND data's "model index". So we can use the same mapping to set the set the current visible window of graphic data.

Keep manually settable display_window for data that are spatial in xy, such as behavior tracks.

Here's an example of how to the OOC timeseries with some toy data:

n = 100
p = 10_000_000
xs = np.linspace(0, 100_000 * np.pi, p, dtype=np.float32)# + np.random.normal(0, 0.2, size=p)

def xs_mapping(x) -> int:
    """maps world space x position -> array index"""
    return int((x * 100) // np.pi)

xs_mapping(xs[132154])

data = np.zeros((n, p, 2), dtype=np.float32)
for i in range(n):
    data[i] = np.column_stack([xs, np.sin(xs * 5) + 0 + np.random.normal(scale=0.5, size=p)])


ndp = NDPositions(data, graphic=fpl.LineStack, display_window=1000, multi=True)

fig = fpl.Figure()

fig[0, 0].add_graphic(ndp.graphic)

fig.show(maintain_aspect=False)

xrange = (0, 10)

def update_view():
    xmin, ymin, xmax, ymax = fig[0, 0].frame.rect
    yavg = (ymax - ymin) / 2
    
    xrange_world = [fig[0, 0].map_screen_to_world((x, y))[0] for x, y in [(xmin + 1, yavg), (xmax - 1, yavg)]]
    global xrange
    if xrange_world == xrange:
        return
    xrange = xrange_world

    x0, x1 = xs_mapping(xrange[0]), xs_mapping(xrange[1])
    
    disp_window = x1 - x0
    ndp.display_window = disp_window
    ndp.indices = (int(np.mean([x0, x1])),)


fig.add_animations(update_view)

I think we can have a display_window that can be set as "auto-x" which is specific only for timeseries, since we can map the visible x-range in world space directly to array indices. This isn't possible with arbitrary ND-position representations, such as behavior tracks in xy space.

The position of the linear selector and index should be independent of the currently rendered portion of the timeseries data. So we can have an additional property, such as visible_indices, which is separate from indices. The LinearSelector sets the indices. Can think properly about this later.

In the orchestrator we should hide these details from the user, perhaps by separating arbitrary nd positions from timeseries representation which set ``display_window="auto-x"` internally, as well as add the update func as an animation function.

I can make a subclass of ndpositions specific to timeseries data which also allows specifying xs and ys separately. It should also allow specifying arrays of shape [s1,... sn, n, p, 1 | 2] where p is the size of the ys, n is the number of lines to be stacked. The processor will then slice xs and ys (or the array of shape shown above) separately and then column_stack before passing it off to the super class to do the rest (window func etc). Dask seems like it's too slow or uses too much ram for doing this kind of lazy slicing and stacking.

Might be useful to have a non-isolated buffer here so NDPositions can directly use a buffer created by NDPositionsProcessor. NDPositionsProcessor could managed the buffer 🤔 . This would avoid a copy step.

Next stuff to do:

Pull in ND Image branch!

Think about and implement mapping of input data to other graphical buffers, like colors or sizes, in NDProcessor. Maybe return a dict where the keys define the features to set and values define new values for the feature? These could map to colors, markers, sizes, etc.

Similarly, allow using an NDProcessor method to set tooltips. NDProcessor can have an optional method that spits out tooltip info for a displayed data point index.
The current display_window indices in array-index-space are a property of NDProcessor
NDProcessor.tooltip_info() takes in a graphic datapoint index (whatever is given by pick_info["vertex_index"] or pick_info["index"]), and spits out tooltip info. It has access to an NDProcessor.displayed_indices property so that it can get any relevant info based on however the input is formatted.

Idea for dataframes, pass in a dataframe and a list of 2-tuples or 3-tuples indicating columns for (x, y, z) vals. Allow passing the names of other columns that are used to map to other graphic features (colors, sizes, markers, etc.) or tooltips. Kinda like seaborn. This way we don't need a specific NDProcessor for deeplabcut-style results.