Aionoscope synthesizes a signal with known latent factors — a sine of exactly this frequency, or a spike at exactly this time. Then we can ask: does the model build a good representation of those factors from the input signal? This gives you a map of the concepts the model represents internally (or doesn't), and of the peculiarities of your architectural choices, such as input data patching or normalization.
Because we set those factors ourselves, the ground truth isn't inferred from correlations in entangled data — it's the dial we turn. That single fact changes what the readout means: the geometry we recover from a frozen model's activations is a measurement of its representation, not a picture we read into it. The catch is that a synthetic benchmark can just as easily measure itself — a careless signal or a leaky probe tells you about the instrument, not the model — so how it is built matters as much as what it finds.
This post is about the instrument itself. The other posts in the series look at existing time-series foundation models and read the manifolds they build — across layers, across sizes, across training. Here we build the scope: how a signal is generated, how one factor becomes a 1,024-point ruler, the two ways we read the activations back, and the four lenses that turn a manifold into numbers.
What it isA signal generator and a probe
Aionoscope has two halves, and the distinction between them organizes everything that follows.
-
The generator (
aiono). A synthesis engine. A Process samples a latent state — which components are present and their exact parameters — and a View chain renders that state into an observed waveform. The latent state is the answer key. - The probe harness. It feeds generated signals to a frozen foundation model, taps the hidden activations at every layer, and scores them two ways: a linear probe (is the factor decodable?) and a manifold analysis (is the factor's geometry laid out faithfully?).
Nothing about the model is changed — no fine-tuning, no gradient through the encoder. We only read what the frozen network already computes. Click through the pipeline; each stage expands.
The generatorA signal specified to the last decimal
The generator is defined by a fixed grammar. Everything observable is built from a library of 14 components across four families:
-
Periodic —
sine,sawtooth,square(amplitude, frequency, phase, offset, duty cycle). -
Trend —
linear,quadratic,log,sigmoid(slope, curvature, centre, sharpness). -
Event —
spike,level_change,gaussianbump (time, magnitude, width). -
Noise & baseline —
gaussian_noise,uniform_noise,random_walk_noise,constant(scale, offset).
A sample mixes
num_enabled ∈ {1, 2, 3} of them. Every
parameter each component exposes is drawn from a
sampler, and — crucially — the
exact drawn values are written into
meta['process']['samples']. Those numbers
are the labels every probe is trained against and
every manifold is scored against. Build a signal
below: toggle components and resample the draws, and
the recorded answer key updates alongside.
Reproducible by construction. Splits
are materialised from fixed seeds (train seed
0; ten validation streams with disjoint
seeds), so the same command regenerates the same
65,536 signals every time — no dataset to download, no
drift between runs.
The sweepOne factor, a 1,024-point ruler
To study geometry we stop mixing and switch to a
controlled slice. Pick one factor — say a
sine's phase — switch every other
component off (num_enabled = 1), and step
that one parameter across a fine grid of
1,024 values. Push all 1,024 signals
through the model, average the activations at each grid
point into a centroid, and order the centroids
by the value you swept. They trace a path through
representation space. That path is the object
we score.
Below, Toto-2.0-2.5B at layer 4, sweeping phase across a full cycle. Drag the slider or press play: on the left the input sine shifts; on the right a marker moves along the manifold the centroids trace.
Experimental setup (protocol
manifold_v0).
-
Grid. 1,024 points
(
grid_size_1d), one sample each (repeats_per_grid_point = 1); only the target factor varies. -
Factor geometry. Each factor is
swept on the ruler that matches it —
interval(time, amplitude),circlewith a known period (phase), orsigned-log(slope). -
Nuisance. Everything else is
pinned at a canonical, non-degenerate value
(
fixed_factor_policy = canonical_non_degenerate). -
Validation. The same grid shifted
half a step
(
validation_policy = half_grid_offset), train seed0, val seed1— an interleaved hold-out, free of leakage. -
Representation. Activations are
mean-pooled per layer and reduced to
pca_dim = 64before any geometry is computed; every layer is measured. -
Geodesics. A k-nearest-neighbour
graph over
k ∈ {4, 6, 8}, best k chosen per layer.
Two readoutsDecodable is not organized
The standard way to interrogate a frozen representation is a linear probe: train a small linear readout and ask "can I recover the factor from these activations?" We run it as the primary benchmark. But a probe only needs one direction where the factor reads out — so it is silent on a deeper question:
Does the representation arrange examples according to the true shape of the process that generated them?
The manifold analysis answers that. It also surfaces structure the other methods structurally cannot, because they answer different questions:
| Method | The question it answers | Blind spot |
|---|---|---|
| Linear probe | Is factor X linearly decodable? | One direction suffices — silent on curvature, folding, neighbourhood scramble. |
| Attribution / saliency | Which inputs drove this prediction? | Explains an output in input-space, not how a factor is laid out in the representation. |
| SAE features | Which monosemantic features fire? | A curved manifold is diluted across many features — you see the endpoints, not the path. |
| Aionoscope geometry | Did the model build the right shape for a known factor? | Needs a controllable generative factor; v0 reads frozen, mean-pooled features linearly. |
The cleanest demonstration in the suite is a single spike slid across time. Its position is a plain interval from 0 to 1, and a linear probe recovers it well. The raw geometry at the input-facing layer, however, is another matter.
Geometry is also a per-layer reading. With depth, the straight-line score climbs as the network promotes coarse position into a dominant axis — the folding does not disappear, it is outweighed.
Four lensesHow the geometry is scored
"Faithful geometry" isn't one number. The harness scores
each manifold through four independent lenses, and the
artifact even records the best layer under each
(best_isometry_layer,
best_neighborhood_layer,
best_projection_layer,
best_fiber_layer). A representation can
score well on one and poorly on another, which is what
makes them worth separating.
1 · Isometry
Does distance in the representation track distance in factor-space? Compared both as a straight line and as a walk along the manifold; the gap between them is how badly the geometry is folded.
stress_scaled · spearman_latent_vs_linear / _geodesic · geodesic_gain2 · Neighbourhood
Are a point's true neighbours still its neighbours in the representation? Local resolution, independent of the global shape.
knn_recall_at_1 / _3 / _5 · trustworthiness · continuity3 · Projection & dimension
Can a low-dimensional chart hold the factor, and
how many independent axes does the path really
use? Participation ratio
PR = (Σλ)² / Σλ² reads the effective
dimension straight off the covariance spectrum.
4 · Fiber
Over each grid point sits a cloud of repeated samples — the fiber. Does it stay tight on the curve, or smear away from it? For circular factors, does the loop close?
mean / median / max_fiber_ratio · between_to_within_snr · cycle_closure_errorThe lenses can disagree. Sine phase is a clear example: the ring is globally near-perfect at every depth (isometry ≈ 1.0), yet local resolution declines through the middle layers — the model gives up fine neighbour ordering in favour of what the forecasting objective requires.
ReproducibilityWhat you'd need to rerun it
Every reading on the dashboard is one JSON artifact, and it carries its own provenance — the exact grid that was swept, the seeds, the config, and the per-layer scores. Nothing is plotted that isn't stored; this page only reads artifacts. Here is the shape of one (arrays and layers abbreviated):
// results/manifolds/Toto-2.0-2.5B/sine_phase/metrics.json { "schema_version": "manifold_result_v0", "target": { "target_name": "sine_phase", "component": "sine", "parameter": "phase", "geometry": "circle", "period": 6.2831853 }, "config": { "mode": "controlled_factor_slices", "grid_size_1d": 1024, "num_enabled": 1, "pca_dim": 64, "geodesic_neighbors": [4, 6, 8], "validation_policy": "half_grid_offset" }, "train_slice_manifest": { "physical_values": [0.0, 0.0061, …, 6.2832], // the dial we turned "latent_coordinates": [0.0, 0.00098, …, 1.0], // normalised grid "seed": 0, "grid_size": 1024 }, "by_layer": { "4": { "stress_scaled": …, "spearman_latent_vs_geodesic": …, "knn_recall_at_5": …, "projection_r2": …, "circular_order_score": …, "mean_fiber_ratio": … } }, "summary": { "best_isometry_layer": {…}, "best_neighborhood_layer": {…}, "best_projection_layer": {…}, "best_fiber_layer": {…} } }
And the two commands that produce a reading:
# geometry: sweep one factor for one model uv run python scripts/run_manifold_calibration_sequential.py \ --model Toto-2.0-2.5B --target sine_phase # probe benchmark: categorical + dense linear readouts uv run python -m aionoscope_benchmarks.run_model \ --model Toto-2.0-2.5B --num-enabled 1
What this is — and what it isn't.
- Readouts are linear and run on mean-pooled features (v0). Non-linear probes and token-level pooling are future work; pooling can hide structure a probe might still find.
- One dataset seed and a fixed probe schedule. Full seed sweeps multiply runtime by orders of magnitude and aren't run yet.
- Each model is evaluated at its native sequence length — ecological, but lengths differ across models, so absolute layer indices aren't comparable one-to-one.
- Code release. The generator and the benchmark harness aren't public yet — we're preparing them for release and will link them here soon.
Get involved
Point it at your model
If you train or ship a time-series model, the most useful thing you can do is tell us where the geometry should be cleaner than it is — a factor your model ought to nail, a layer that surprises you, a component we don't yet generate. The instrument is only as sharp as the questions put to it.
- Suggest a model or a generative factor to add.
- Tell us which lens matters for your use case.
- Flag a reading that looks wrong — those are the best bug reports.