How to Load Data

Format-agnostic data ingestion with Loaders and EventStores

How to Load Data

This tutorial introduces the Loader pattern and EventStores - the foundation for bringing music data into TimeToAlign!

Learning Objectives: - Use Loaders to ingest music data from various formats - Navigate EventStores and access event data - Understand the harmonized schema that unifies different data sources

Prerequisites: - Basic Python and pandas knowledge - TimeToAlign! installed (pip install timetoalign)

Why Loaders Matter

Music data comes in many formats: MusicXML, MIDI, MEI, Humdrum, proprietary TSV exports, and more. Each format has its own structure, terminology, and quirks.

The problem: Without a unified approach, you’d need format-specific code for every data source, making cross-format analysis difficult and error-prone.

The TimeToAlign! solution: Loaders normalize heterogeneous formats into a consistent EventStore, enabling downstream processing without format-specific code.

MusicXML ─┐
MIDI ─────┼──> Loader ──> EventStore ──> DataFrame
TSV ──────┘

Setup

import pandas as pd

from timetoalign import Ms3Loader
from timetoalign.core.events import EnharmonicPitch
from timetoalign.loader.score.music21 import Music21Loader
from timetoalign.loader.score.partitura import PartituraLoader
from timetoalign.testdata import ensure_data

DATA_DIR = ensure_data("vienna_1x22")

# Our test piece: Chopin Etude Op.10 No.3
CHOPIN_XML = DATA_DIR / "Chopin_op10_no3.musicxml"
CHOPIN_TSV = DATA_DIR / "ms3" / "chopin_op10_no3.notes.tsv"

CHOPIN_XML.name, CHOPIN_TSV.name

/home/laser/miniconda3/envs/timetoalign/lib/python3.11/site-packages/partitura/__init__.py:9: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources

('Chopin_op10_no3.musicxml', 'chopin_op10_no3.notes.tsv')

The Loader Pattern

All TimeToAlign! loaders follow the same three-step pattern:

1. Create a loader instance
2. Load a file using .load(path)
3. Access the store containing EventStores

Let’s see this in action with three different loaders, all loading the same Chopin piece:

# Load from three different sources
tsv_loader = Ms3Loader()
tsv_loader.load(CHOPIN_TSV)

partitura_loader = PartituraLoader()
partitura_loader.load(CHOPIN_XML)

music21_loader = Music21Loader()
music21_loader.load(CHOPIN_XML)

# All produce ScoreStores
{
    "TSV": type(tsv_loader.store).__name__,
    "Partitura": type(partitura_loader.store).__name__,
    "Music21": type(music21_loader.store).__name__,
}

{'TSV': 'ScoreStore', 'Partitura': 'ScoreStore', 'Music21': 'ScoreStore'}

Cross-Loader Validation

One of the key benefits of TimeToAlign! is that different loaders produce comparable output. Let’s verify that all three loaders found the same number of notes:

# Convert to DataFrames
tsv_df = tsv_loader.store.notes.to_dataframe()
partitura_df = partitura_loader.store.notes.to_dataframe()
music21_df = music21_loader.store.notes.to_dataframe()

# Count only Note events (not rests or other event types)
counts = {
    "TSV": len(tsv_df[tsv_df["event_type"] == "Note"]),
    "Partitura": len(partitura_df[partitura_df["event_type"] == "Note"]),
    "Music21": len(music21_df[music21_df["event_type"] == "Note"]),
}

# Validate against gold standard
assert all(c == 498 for c in counts.values()), f"Note count mismatch: {counts}"

pd.Series(counts, name="note_count")

TSV          498
Partitura    498
Music21      498
Name: note_count, dtype: int64

The EventStore

Each ScoreStore contains EventStores - efficient, PyArrow-backed tables that hold musical events.

Key characteristics: - High Performance: Built on Apache Arrow for fast columnar operations - Type Safety: Schema metadata preserves units and types - Pandas Interop: Easy conversion with .to_dataframe()

notes_store = tsv_loader.store.notes

{
    "type": type(notes_store).__name__,
    "n_events": len(notes_store),
    "storage": type(notes_store.table).__name__,
}

{'type': 'NoteEventData', 'n_events': 498, 'storage': 'Table'}

# Examine the schema with metadata
schema_info = []
for field in notes_store.table.schema:
    meta = field.metadata or {}
    meta_str = (
        ", ".join(f"{k.decode()}={v.decode()}" for k, v in meta.items()) if meta else ""
    )
    schema_info.append(
        {"name": field.name, "type": str(field.type)[:30], "metadata": meta_str}
    )

pd.DataFrame(schema_info)

	name	type	metadata
0	id	string
1	name	string
2	temporal_type	string
3	event_type	string
4	start	struct<value: double, numerato	unit=quarters, number_type=fraction
5	end	struct<value: double, numerato	unit=quarters, number_type=fraction
6	duration	struct<value: double, numerato	unit=quarters, number_type=fraction
7	mc	int64	number_type=int64
8	mn	string
9	mc_onset	struct<value: double, numerato	number_type=fraction
10	mn_onset	struct<value: double, numerato	number_type=fraction
11	specific_pitch	struct<step: string, alter: in
12	midi	int64	number_type=int64
13	tpc	int64	number_type=int64, unit=fifths
14	octave	int64	number_type=int64
15	tied	int64
16	gracenote	string
17	chord_id	int64
18	voice	int64	number_type=int64
19	staff	int64	number_type=int64
20	part_id	string

The Harmonized Schema

TimeToAlign! uses a harmonized schema to represent events consistently across formats:

Column	Description
id	Unique identifier for the event
temporal_type	“instant” or “interval”
event_type	Type of event (Note, Rest, etc.)
start, end, duration	Temporal coordinates in quarter notes
mc, mn	Measure count and measure number
specific_pitch	Fully spelled pitch (step + alter + octave) — the default pitch field
midi	The raw source MIDI pitch as an integer (affords an EnharmonicPitch view on request)

# Show selected fields for the first few notes
display_cols = [
    "id",
    "name",
    "temporal_type",
    "event_type",
    "start",
    "duration",
    "mc",
    "mn",
    "octave",
]
tsv_df[display_cols].head(10)

	id	name	temporal_type	event_type	start	duration	mc	mn	octave
0	note:000001	B3	interval	Note	0	1/2	1	1	3
1	note:000002	E2	interval	Note	1/2	1/4	2	2	2
2	note:000003	E2	interval	Note	1/2	1	2	2	2
3	note:000004	G#3	interval	Note	1/2	1/4	2	2	3
4	note:000005	E4	interval	Note	1/2	1/2	2	2	4
5	note:000006	B2	interval	Note	3/4	1/2	2	2	2
6	note:000007	B3	interval	Note	3/4	1/4	2	2	3
7	note:000008	G#3	interval	Note	1	1/4	2	2	3
8	note:000009	D#4	interval	Note	1	1/4	2	2	4
9	note:000010	B2	interval	Note	5/4	1/4	2	2	2

Pitch Information

A spelled score faithfully supports a fully specific pitch, so pitch is represented exactly once: specific_pitch (step + alter + octave) is the single default pitch field, and it preserves the enharmonic spelling (e.g. G♯ vs A♭) that a bare MIDI pitch cannot.

The source MIDI pitch survives as a plain midi integer column. It is redundant with the spelling, so it is not stored as a second pitch field; instead the EventStore affords an EnharmonicPitch view over it on request via get_field(EnharmonicPitch) (or the enharmonic_pitch_field accessor). The spelled and enharmonic views diverge exactly where it matters — an accidental-bearing note.

# The default pitch field: fully spelled SpecificPitch scalars.
specific_pitch = notes_store.specific_pitch_field

# The afforded EnharmonicPitch view over the raw `midi` integer column.
enharmonic_pitch = notes_store.get_field(EnharmonicPitch)

# Index 3 is an accidental-bearing note, where spelling (SP) and the
# MIDI pitch (EP) part ways.
{
    "specific_pitch[3] (default)": repr(specific_pitch[3]),
    "enharmonic_pitch[3] (afforded view)": repr(enharmonic_pitch[3]),
    "raw midi int[3]": tsv_df.iloc[3]["midi"],
}

{'specific_pitch[3] (default)': 'SP(G♯3)',
 'enharmonic_pitch[3] (afforded view)': 'EP(G♯/A♭3)',
 'raw midi int[3]': np.int64(56)}

Duration Analysis

TimeToAlign! stores durations in quarter notes. Let’s analyze the rhythmic content of our piece:

# Duration distribution
tsv_df["duration"].value_counts().sort_index().to_frame("count")

	count
duration
1/4	391
1/2	42
3/4	2
1	54
2	5

# Summary statistics
tsv_df["duration"].describe()

count     494
unique      5
top       1/4
freq      391
Name: duration, dtype: object

Navigating by Measure

The mc (measure count) and mn (measure number) fields allow easy navigation through the score. Note that mn is stored as a string (to support labels like “1a”, “1b”), so we convert to int for proper sorting:

# Notes per measure, sorted numerically
notes_per_measure = tsv_df.groupby("mn").size()

# Convert index to int for proper sorting (works for simple numeric measure numbers)
notes_per_measure.index = notes_per_measure.index.astype(int)
notes_per_measure = notes_per_measure.sort_index()

notes_per_measure.to_frame("notes")

	notes
mn
1	1
2	21
3	24
4	22
5	25
6	25
7	24
8	21
9	22
10	22
11	24
12	22
13	25
14	25
15	28
16	27
17	56
18	18
19	21
20	21
21	17
22	7

# Get all notes in a specific measure
measure_5 = tsv_df[tsv_df["mn"] == "5"]
measure_5[["name", "duration", "voice", "staff"]]

	name	duration	voice	staff
68	E2	1/4	1	2
69	E2	1	3	2
70	B3	1/4	3	1
71	A4	1/4	1	1
72	B2	1/2	1	2
73	E4	1/4	3	1
74	G#4	1/4	1	1
75	G#3	1/4	3	1
76	D#4	1/4	1	1
77	B2	1/4	1	2
78	B3	1/4	3	1
79	E4	1/4	1	1
80	B1	1/4	1	2
81	B1	1	3	2
82	A3	1/4	3	1
83	C#4	1/4	2	1
84	F#4	1	1	1
85	B2	1/2	1	2
86	B3	1/4	3	1
87	D#4	1/4	2	1
88	A3	1/4	3	1
89	C#4	1/4	2	1
90	B2	1/4	1	2
91	B3	1/4	3	1
92	D#4	1/4	2	1

Comparing Loader Outputs

While all loaders produce the same number of notes, there can be subtle differences in how they interpret the score. Let’s compare the first few notes:

# Compare ID schemes across loaders
pd.DataFrame(
    {
        "TSV_id": tsv_df["id"].head(5).values,
        "TSV_name": tsv_df["name"].head(5).values,
        "Partitura_id": partitura_df["id"].head(5).values,
        "Music21_id": music21_df["id"].head(5).values,
    }
)

	TSV_id	TSV_name	Partitura_id	Music21_id
0	note:000001	B3	note:000001	note:000001
1	note:000002	E2	note:000002	note:000002
2	note:000003	E2	note:000003	note:000003
3	note:000004	G#3	note:000004	note:000004
4	note:000005	E4	note:000005	note:000005

Unit Metadata

TimeToAlign! stores unit information in the PyArrow schema metadata. This ensures coordinates are always interpreted correctly:

# Extract unit metadata for temporal fields
temporal_cols = ["start", "end", "duration"]
{
    field.name: field.metadata.get(b"unit", b"(unknown)").decode()
    for field in notes_store.table.schema
    if field.name in temporal_cols and field.metadata
}

{'start': 'quarters', 'end': 'quarters', 'duration': 'quarters'}

Voice and Staff Information

In scores with multiple staves or multiple voices per staff, notes are distributed accordingly:

# Notes by staff and voice
tsv_df.groupby(["staff", "voice"]).size().unstack(fill_value=0)

voice	1	2	3
staff
1	112	50	153
2	142	2	39

Summary

In this tutorial, we learned:

1. The Loader Pattern: Create -> Load -> Access Bundle
2. Three Score Loaders: Ms3Loader, PartituraLoader, Music21Loader
3. EventStore: PyArrow-backed, high-performance event storage
4. Harmonized Schema: Consistent fields across all loaders
5. Cross-Validation: Same piece from different sources yields same note count

Key Takeaway: > Loaders normalize heterogeneous formats into a consistent EventStore, > enabling downstream processing without format-specific code.

Next Steps

• how01_advanced_cmaps.ipynb: Learn how to convert between coordinate systems
• how01_manual_timeline_construction.ipynb: Create Timeline objects from EventStores

---

Exercise: Load Another Score

Task: Load the Beethoven String Quartet from beethoven_op18.mid and analyze its structure.

Hints: 1. Use PartituraLoader for MIDI files 2. Check how many parts are in the score 3. Count notes per part

Solution

# Load the Beethoven quartet
loader = PartituraLoader()
loader.load(DATA_DIR / "beethoven_op18.mid")

df = loader.store.notes.to_dataframe()
{"total_notes": len(df), "notes_per_part": df.groupby("part_id").size().to_dict()}

# Your solution here