AlignmentBundle, FlowMap, OMR, 16+ timelines across 3 domains
How to Align Multimodal Data (Beethoven)
Figure 3 acid test for TimeToAlign! — 16+ timelines across all 3 domains (Physical, Logical, Graphical) in 5 TimelineGroups within one AlignmentBundle.
Structure: 1. Part I: Build 3 recording groups (Groups 1-3) — 15 DPTs 2. Part II: Build Score group (Group 4) + align with recordings 3. Part III: Build Emerson group (Group 5) + cross-group coordinate transfer
/home/laser/miniconda3/envs/timetoalign/lib/python3.11/site-packages/partitura/__init__.py:9: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
import pkg_resources
Each EEP recording directory contains 5 modalities (audio, 3 feature types, MoCap) plus .notes files with annotated note events. The function below builds a TimelineGroup from one such directory via the XML manifest.
Structure (per manuscript): - 5 parent physical timelines, each with a SamplesToSeconds c-map - Audio, Tonal, LowLevel, Rhythm parents: 6 children each (mono, binaural, 4 pickups) - MoCap parent: 4 children (one per instrument: vln1, vln2, vla, cello)
def build_recording_group(xml_path, group_id, group_name, dpt_base):"""Build a TimelineGroup from one EEP recording directory via XML manifest. Args: xml_path: Path to the recording's XML manifest file. group_id: ID for the TimelineGroup. group_name: Human-readable name for the group. dpt_base: Starting DPT number (e.g. 1 for dpt1-dpt5). Returns: TimelineGroup with 5 hierarchical DPTs (parent + children). """ rv = RepoVizzLoader.from_file(xml_path) n = dpt_base# 1. Audio (mono as parent, 6 sources as children) audio = rv.create_timeline("mono", uid=f"dpt{n}", name="Audio")for src in AUDIO_SOURCES: audio.add_child(rv.create_timeline(src, uid=src), offset=0)# 2-4. Essentia descriptors (tonal, lowlevel, rhythm) desc_cfgs = [ ("tonal", "ChordsStrength", 1), ("lowlevel", "Dissonance", 2), ("rhythm", "BeatsLoudness", 3), ] descriptors = []for desc_type, desc_name, offset in desc_cfgs: parent = rv.create_timeline(f"{desc_type}.{desc_name}.mono", uid=f"dpt{n + offset}", name=desc_type.title(), )for src in AUDIO_SOURCES: parent.add_child( rv.create_timeline(f"{desc_type}.{desc_name}.{src}", uid=f"{src}_{desc_type}" ), offset=0, ) descriptors.append(parent)# 5. MoCap bb_angle (from the DescriptorGroup section of the XML) mocap = rv.create_timeline( rv.find_descriptor("bb_angle", "vln1"), uid=f"dpt{n +4}", name="MoCap", )for inst in INSTRUMENTS: child = rv.create_timeline( rv.find_descriptor("bb_angle", inst), uid=f"{inst}_mocap", ) mocap.add_child(child, offset=0)# Add notes to pickup childrenfor inst in INSTRUMENTS: notes = rv.store.notes_for_instrument(inst)if notes and (pickup := audio.get_child(f"pickup_{inst}")): pickup.add_events(notes.to_dataframe().to_dict("records"))return TimelineGroup(id=group_id, name=group_name, timelines=[audio, *descriptors, mocap], )
Part I: Three Recording Groups (Groups 1-3)
Each EEP recording = 5 DPTs (audio + 3 feature types + MoCap) at different sampling rates, all sharing the same physical duration. Note events live as a child of the audio DPT.
The ABC score has repeats and volta brackets. The loader’s create_flow_controller() derives the repeat structure from the measure data and computes the default flow (all repeats taken). This is the same flow control machinery used later for CLT2 (the recordings edition) in Part III.
Flow(default): 226 folded → 291 unfolded (×1.29), 11 sections
# MCs Sections Reason
── ─────────── ──────── ──────────────
1 [1, 10) A start
2 [1, 19) A;B repeat → 1
3 [10, 28) B;C repeat → 10
4 [19, 45) C;D;D1 repeat → 19
5 [28, 44) D repeat → 28
6 [45, 85) D2;E skip → 45
7 [78, 94) E;F;F1 repeat → 78
8 [85, 93) F repeat → 85
9 [94, 103) F2;G;G1 skip → 94
10 [95, 102) G repeat → 95
11 [103, 227) G2 skip → 103
Sequence: A A B B C C D D1 D D2 E E F F1 F F2 G G1 G G2
The flow controller and flow will be used in §9.2 to unfold the entire score group at once — not just CLT1, but all timelines.
7. DGT1: OMR Ground Truth
The OMR data contains 3,190 note head bounding boxes across 22 score pages. Each page has 2 systems (except the last which has 1), giving 43 system segments in reading order. Note events use Left (start) and Width (duration) as pixel coordinates. Each system’s onset_beats values provide a c-map from pixels to quarters.
Build the DGT1 bottom-up: system segments → page SegmentLine[DiscreteGraphicalTimeline] → top-level SegmentLine[SegmentLine[DiscreteGraphicalTimeline]]. Events and c-maps must be added before a timeline is locked as a child.
noteheads = pd.DataFrame( {"start": omr_df["Nodes.Node.Left"].astype(int),"end": (omr_df["Nodes.Node.Left"] + omr_df["Nodes.Node.Width"]).astype(int),"onset_beats": omr_df["onset_beats"].astype(float),"pitch": omr_df["pitch"],"staff_id": omr_df["staff_id"].astype(int),"midi_pitch": omr_df["midi_pitch_code"].astype(int),"top": omr_df["Nodes.Node.Top"].astype(int),"page": omr_df["@pageIndex"],"spacing_run_id": omr_df["spacing_run_id"], })dgt1 = SegmentLine( length=0, unit=TimeUnit.pixels, number_type=NumberType.int, segment_type=SegmentLine, inner_segment_type=DiscreteGraphicalTimeline, uid="dgt1",)for page_idx, page_data in noteheads.groupby("page", sort=True):# Systems ordered by vertical position (top first = reading order) sys_top = page_data.groupby("spacing_run_id")["top"].min() sys_order = sys_top.sort_values().index page = SegmentLine( length=0, unit=TimeUnit.pixels, number_type=NumberType.int, segment_type=DiscreteGraphicalTimeline, )for sys_rank, sys_id inenumerate(sys_order): sys_data = page_data[page_data["spacing_run_id"] == sys_id] system = DiscreteGraphicalTimeline( length=IMAGE_WIDTH, uid=f"p{page_idx}_s{sys_rank}", name=f"Page {page_idx +1}, System {sys_rank +1}", ) events = sys_data.drop(columns=["page", "spacing_run_id"]) system.add_events(events.assign(event_type="Notehead").to_dict("records"))# C-map: pixels → quarters (deduplicated for chords at the same x) pairs = ( events[["start", "onset_beats"]] .drop_duplicates("start") .sort_values("start") )iflen(pairs) >=2: system.add_conversion_map( TableMap( x_values=pairs["start"].tolist(), y_values=pairs["onset_beats"].tolist(), source_unit="pixels", target_unit="quarters", uid=f"p{page_idx}_s{sys_rank}_px_to_qb", ) ) page.append_segment(system) dgt1.append_segment(page, name=f"page_{page_idx}")dgt1
The OpenScore edition covers all 4 movements. We use the flow controller to identify section breaks (movement boundaries) and extract the 4th movement as a child timeline.
The loader’s create_flow_controller() derives section boundaries from the score’s flow control markup. Splitting at those coordinates creates one region per movement.
os_flow_controller = os_loader.create_flow_controller()boundaries = os_flow_controller.get_section_boundary_coordinates()os_full.create_regions_from_boundaries( [0, *[float(b) for b in boundaries], float(os_full.length.value)], prefix="movement")openscore = os_full.create_child_from_region("movement_4", uid="openscore")openscore
The playthrough section boundaries (from §6.1) can now be mapped through the score group to DGT1 pixel coordinates. This demonstrates cross-domain coordinate transfer within a TimelineGroup: the InterpolationMap between CLT1 (quarters) and DGT1 (pixels) uses each system’s pixel-to-quarter TableMap as its C-map anchor.
# Build a page-boundary lookup from DGT1's segment structure_page_bounds = []for _seg_id in dgt1.list_segments(): _off = dgt1.get_child_offset(_seg_id) _seg = dgt1.get_child(_seg_id) _page_bounds.append( (float(_off.value), float(_off.value) +float(_seg.length.value)) )_section_rows = []for _sid, _qb in abc_controller.get_atomic_section_coordinates(flow=abc_flow).items(): _ts = score_group.get_timestamp_at(float(_qb), "clt1") _px = _ts.to_dict().get("dgt1") _page =next( (i +1for i, (s, e) inenumerate(_page_bounds) if s <= _px < e),"-", ) _section_rows.append( {"section": _sid, "quarters": float(_qb), "dgt1_pixels": _px, "page": _page} )section_boundary_table = pd.DataFrame(_section_rows).set_index("section")section_boundary_table
quarters
dgt1_pixels
page
section
A
0.0
0
1
B
64.0
7753
2
C
128.0
15506
4
D
192.0
23260
5
D1
253.0
30649
7
D2
317.0
38403
8
E
448.5
54333
11
F
496.5
60148
13
F1
525.0
63601
13
F2
557.0
67477
14
G
561.0
67962
14
G1
589.0
71354
15
G2
621.0
75230
16
Each atomic section’s start coordinate is located precisely on a specific page of the OMR score image. The pixel column gives the linearised x-coordinate across all 22 pages; the page column tells which score image to open.
9.2 Unfolding the Entire Score Group
The score has repeats and volta brackets. Rather than unfolding each timeline individually, TimelineGroup.unfold() does it in one call: the flow controller’s section boundaries are resolved via the group’s interpolation maps, so every timeline — regardless of domain — is sliced and reassembled in playthrough order.
The unfolded CLT1 carries all note events in playthrough order. Extract them for note matching:
clt1_unfolded = score_group_unfolded.get_timeline("clt1")abc_notes_df = clt1_unfolded.get_events( event_type="Note", include_children=False).to_dataframe()# Cast types restored from string (EventData stores extra columns as strings)abc_notes_df["staff"] = pd.to_numeric(abc_notes_df["staff"], errors="coerce").astype("Int64")abc_notes_df["tied"] = pd.to_numeric(abc_notes_df["tied"], errors="coerce")abc_notes_df.loc[abc_notes_df["tied"] ==0, "tied"] = np.nanabc_notes_df["quarterbeats_playthrough"] = abc_notes_df["start"]abc_prepared = prepare_abc_notes_for_matching(abc_notes_df)len(abc_prepared) # note onsets after dropping tied notes
3763
10. Aligning Recordings with the Score via Note Matching
Each EEP recording’s note events (seconds, pitch, staff) are matched against the ABC unfolded score notes (quarterbeats, pitch, staff) prepared in §9.2 using greedy sequential matching. The result: MatchClaim objects that connect recording coordinates to score coordinates. No pre-computed TSV is needed — the unfolded CLT1 carries all the notes.
Match each recording against the score. The source_timeline_id and target_timeline_id are the audio DPT and CLT1 respectively — these appear in the resulting MatchClaim anchors.
We use rv.store.notes to access the EEP notes from the XML manifest’s score section — no direct EepNotesLoader import needed.
The score group unites 3 score representations across 2 domains (Logical + Graphical). Note matching produced MatchClaims connecting each recording group’s audio timeline to CLT1:
Recording
Matched
Unmatched EEP
Unmatched ABC
Normal
3,740
16
23
Mechanical
3,743
13
20
Exaggerated
2,650
4
1,113
Next: Part III adds the Emerson group and demonstrates cross-group coordinate transfer using an AlignmentBundle.
Part III: Emerson Recording + Cascading Alignment (Group 5)
The Emerson group connects a commercial recording to a second score edition via segment-level alignment. Unlike the EEP groups (per-note alignment), the Emerson recording is aligned at the level of 10 structural sections (alpha through kappa), derived from the score’s repeat structure.
The central payoff of this notebook is cascading alignment: by adding the recordings edition’s unfolded score (CLT2) to the same group as CLT1, coordinate transfer chains automatically from the EEP recordings through both score editions to the Emerson recording.
The recordings edition uses the same measure/repeat structure as CLT1 but was encoded independently (ABC v1.0). We load it via Ms3Loader and use its flow controller to compute the traversal map.
ScoreFlowController (226 MCs, 13 atomic sections, 18 flow events)
├─A──┤├─B──┤├─C──┤├─D──┤├─E──┤┌1─E1─┌2─E2─├─F──┤├─G──┤├─H──┤├─I──┤┌1─I1─┌2─I2─
1-9 10-18 19-27 28 29-43 44 45-77 78-84 85-93 94 95-10 102 103-2
║: :║║: :║║: :║ ║: :║ ║: :║║: :║ ║: :║
Flow control:
MC 1: repeat_start (section A)
MC 9: repeat_end → MC 1
MC 10: repeat_start (section B)
MC 18: repeat_end → MC 10
MC 19: repeat_start (section C)
MC 27: repeat_end → MC 19
MC 29: repeat_start (section E)
MC 44: repeat_end → MC 29; volta 1 (section E1)
MC 45: volta 2 (section E2)
MC 78: repeat_start (section F)
MC 84: repeat_end → MC 78
MC 85: repeat_start (section G)
MC 93: repeat_end → MC 85
MC 95: repeat_start (section I)
MC 102: repeat_end → MC 95; volta 1 (section I1)
MC 103: volta 2 (section I2)
Section transitions:
A → [A, B] B → [B, C] C → [C, D] D → [E]
E → [E1, E2] E1 → [E] E2 → [F] F → [F, G]
G → [G, H] H → [I] I → [I1, I2] I1 → [I]
I2 → []
Atomic flow (default):
A → A → B → B → C → C → D → E → E1 → E → E2 → F → F → G → G → H → I → I1 → I → I2
Compute the default flow (all repeats taken) and a single-pass flow (no repeats, last volta only) for comparison:
Flow(single): 226 folded → 224 unfolded (×0.99), 3 sections
# MCs Sections Reason
── ─────────── ──────── ──────────────
1 [1, 44) A;B;C;D;E start
2 [45, 102) E2;F;G;H;I skip → 45
3 [103, 227) I2 skip → 103
Sequence: A B C D E E2 F G H I I2
11.3 Unfolding CLT2
The recordings edition has the same repeat structure as CLT1. We unfold it via the standalone create_unfolded_timeline() function, passing the default flow (all repeats taken). The result is a flat timeline with all sections concatenated in playthrough order — coordinates in quarter-beats, suitable for matching against the Emerson CSV’s unfolded floating-measure boundaries.
The measureMapAudio.csv provides a 10-segment alignment between the unfolded score (floating measures) and the Emerson recording (seconds). Each segment is labelled with a Greek letter (alpha through kappa).
Create DPT16 as a ContinuousPhysicalTimeline in seconds. Unlike the EEP recordings (per-note alignment), the Emerson alignment operates at the level of section boundaries — the coordinates in ema_df will become MatchClaims in §11.5 rather than a C-map.
Each row in the measure-map CSV defines a section boundary: a correspondence between an unfolded floating-measure coordinate on CLT2 and a seconds coordinate on DPT16. We create one MatchClaim per boundary, plus the final end boundary.
These cross-group claims are the key connection between the Emerson recording and the score group. AlignmentAnchor stores unit-bearing Coordinate values, so the units are explicit at the claim boundary even though the source data is numeric.
12.1 The Key Move: Adding CLT2_unfolded to the Unfolded Score Group
The Unfolded Score Group and the Emerson Group are currently independent: neither shares a timeline with the other, and no MatchClaims connect them. The Emerson MatchClaims (§11.5) link CLT2_unfolded to DPT16 — but CLT2_unfolded is not yet in any group that the bundle’s existing WarpMaps can reach.
The insight: CLT1_unfolded and CLT2_unfolded encode the same music from different editions. By adding CLT2_unfolded to the Unfolded Score Group, any coordinate on CLT1_unfolded can be transferred to CLT2_unfolded via within-group interpolation, and from there to DPT16 via the Emerson MatchLine’s WarpMap. The cascading path:
CLT2_unfolded now appears alongside CLT1, DGT1, and OpenScore in the unfolded score group. The group’s interpolation maps link all four timelines pairwise, bridging quarter-beats and floating measures.
12.2 The Emerson Group
The Emerson group contains only DPT16 — the recording timeline. CLT2_unfolded lives in the score group, and the Emerson MatchClaims connect the two groups via cross-group claims.
The bundle collects all 5 groups and connects them via MatchClaims. Within each group, coordinate transfer uses linear interpolation. Between groups, WarpMaps (built from MatchClaims) enable cross-domain transfer.
Before demonstrating bundle-level coordinate transfer, it is instructive to see the intermediate MatchLine and WarpMap that the bundle constructs internally. The MatchLine orders the 11 Emerson anchors by source coordinate; the WarpMap interpolates between them.
The bundle’s get_matchstamp_at() method is the primary interface for cross-domain coordinate transfer. Given a coordinate on any timeline, it returns a MatchStamp with corresponding coordinates on all connected timelines — regardless of domain. With CLT2_unfolded bridging the score group and the Emerson MatchClaims, the bundle now reaches all 5 groups.
14.1 Inspecting CLT1’s Harmony Annotations
Before transferring coordinates, let us see what harmonic events live on CLT1. The annotations child carries all harmony labels from the ABC score:
14.2 USE CASE A — Transfer a Harmony Across All Groups
The V7 at quarterbeat 79 (m. 20) is a dominant seventh — one of the most recognisable sonorities. Where does this moment land across all 5 groups, in every domain? The nested format groups results by TimelineGroup:
MatchLine: dropped 11 stamp(s) that do not contain source timeline 'clt1'
MatchLine: dropped 1430 stamp(s) that do not contain source timeline 'dgt1'
MatchLine: dropped 1430 stamp(s) that do not contain source timeline 'openscore'
MatchLine: dropped 1419 stamp(s) that do not contain source timeline 'clt2_unfolded'
MatchStamp belongs to the same unified stamp family as TimeStamp and GroupTimestamp: get_coordinate() returns a unit-bearing coordinate, while is_interpolated reports whether resolution used interpolation.
Note that the emerson group now appears in the output: the cascading path CLT1 -> CLT2_unfolded -> DPT16 connects the Emerson recording to the rest of the bundle.
The flat format is useful for programmatic access:
14.3 USE CASE B — Reverse Transfer: Emerson to All Groups
The cascading alignment is bidirectional. Starting from a seconds coordinate on DPT16 (the Emerson recording), we can reach every connected timeline — including the three EEP recording groups:
A coordinate at 120 seconds into the Emerson recording is mapped through the WarpMap to CLT2_unfolded, then via interpolation to CLT1, and from there via the per-note WarpMaps to DPT1, DPT6, and DPT11 — all in a single call.
14.4 USE CASE C — Section Boundaries Across All Groups
The score’s repeat structure defines atomic sections (A through M). The flow controller (from §6.1) computes each section’s unfolded quarterbeat start coordinate. With the Emerson group now connected, the boundary table includes DPT16:
Each row gives the exact coordinate of a section boundary in every timeline and domain — including the Emerson recording’s dpt16 column. The sample counts are integers; the seconds and quarterbeats are floats — matching each timeline’s native type.
15. Summary & Key Takeaways
“Any two events in the bundle can be related with each other — regardless of whether they live on the same timeline, in the same group, or even in the same domain — as long as a path of MatchClaims or ConversionMaps connects them.”
The Cascading Alignment Pattern
The central demonstration of this notebook is that a single additional group membership retroactively enriches every timeline already present in the bundle. Adding CLT2_unfolded to the Unfolded Score Group bridges two independent alignment networks:
EEP recordings (per-note MatchClaims) connect DPT1-DPT15 to CLT1
Emerson recording (section-boundary MatchClaims) connects DPT16 to CLT2_unfolded
CLT2_unfolded in the score group bridges the two via within-group interpolation