Skip to content
MTRL

Author a connector

A connector is the code that turns your file’s bytes into runs and checkpoints. MTRL deliberately ships no instrument parser: every format arrives through a loader you supply, so the core never guesses what your bytes mean. In this guide you will write one for a two-column CSV and see exactly what the core guarantees in return.

This guide assumes: you have finished the tutorial path, so you are comfortable with samples, records and commits.

The two halves of a connector

Section titled “The two halves of a connector”

Every connector does two jobs, and it helps to keep them separate in your mind even when one small class does both.

Parsing turns bytes into structured values. It knows the file format: headers, encodings, delimiters, byte order. It answers “what does row three column two say?”

Mapping turns those values into MTRL records. It knows the science: which column is the axis and which is the signal, what the fields are called, which units they carry, and what metadata belongs on the run. It answers “what does this measurement mean?”

For a one-off file, a single loader class does both jobs, as the example below shows. When a format family grows — one binary container holding several kinds of measurement — the two halves earn their own types: ParseInstrumentFile reads the bytes once, and one or more InstrumentDataMapper implementations map the parsed values to records. The mtrl.connectors.protocols module declares both shapes alongside the loader protocol, and larger connectors (such as vendor instrument suites) build on exactly those types.

The contract

Section titled “The contract”

A loader is any object whose load method takes the captured bytes, the run name, and an optional filename label, and returns one LoadedInstrumentFile:

class Loader:
    def load(self, data: bytes, run_name: str, *, filename: str = "") -> LoadedInstrumentFile:
        ...

The return holds one InstrumentRun (with its checkpoints) and one SourceFile describing the origin: filename, file family, instrument and parser version.

In return, the coordinator owns everything that makes the commit trustworthy. It reads the file once, inside a bounded capture budget. It records the source evidence under the frozen policy: SourcePolicy.RETAIN keeps the bytes with the archive, SourcePolicy.DIGEST_ONLY keeps only the digest, and the first successful call freezes that choice. It commits the loader’s output as one atomic revision, and if the loader raises, the rollback leaves records, evidence and revisions exactly as they were. The loader never reopens the path, and the facade never sniffs the file or chooses a loader from its suffix.

Try it: a CSV connector

Section titled “Try it: a CSV connector”

This standalone script writes a two-column CSV file, reads it with the standard library’s csv module through an explicit loader in the base install, and prints the committed results. It proves the mechanics only; nothing here is a vendor-format claim.

import csv
import tempfile
from pathlib import Path


import numpy as np


from mtrl import (
    InstrumentRun,
    LoadedInstrumentFile,
    MaterialData,
    MetadataField,
    Sample,
    SourceFile,
    Unit,
)
from mtrl.models import CheckpointType




class CsvSpectrumLoader:
    """Read one two-column CSV spectrum from the captured bytes."""


    def load(self, data: bytes, run_name: str, *, filename: str = "") -> LoadedInstrumentFile:
        rows = list(csv.DictReader(data.decode("utf-8").splitlines()))
        content = np.empty(len(rows), dtype=[("wavenumber", "<f8"), ("absorbance", "<f8")])
        content["wavenumber"] = [float(row["wavenumber"]) for row in rows]
        content["absorbance"] = [float(row["absorbance"]) for row in rows]
        checkpoint = MaterialData(
            content=content,
            type=CheckpointType.RAW,
            checkpoint_name=run_name,
            units={
                "wavenumber": Unit.from_raw("cm^-1"),
                "absorbance": Unit.from_raw("a.u."),
            },
        )
        run = InstrumentRun(
            run_name=run_name,
            metadata={"operator": MetadataField(name="operator", value="csv")},
            checkpoints={checkpoint.checkpoint_id: checkpoint},
        )
        source = SourceFile(
            filename=filename,
            file_family="csv-spectrum",
            instrument="generic",
            parser_version="1",
        )
        return LoadedInstrumentFile(run=run, source=source)




with tempfile.TemporaryDirectory() as scratch:
    source_path = Path(scratch) / "spectrum.csv"
    source_path.write_text(
        "wavenumber,absorbance\n2000.0,0.20\n2001.0,0.50\n2002.0,0.30\n",
        encoding="utf-8",
    )


    film = Sample("polymer-film-01")
    ingested = film.ingest(source_path, CsvSpectrumLoader(), name="spectrum")
    print(len(ingested), ingested[0].name, len(film.records))

The script writes its synthetic source into a tempfile.TemporaryDirectory, so it never overwrites an instrument file that already sits in your working directory. The sample keeps the captured bytes, so film and ingested stay usable once that directory is gone.

1 spectrum 1

You should see one handle for one committed checkpoint. One run, one committed checkpoint, and one revision, so the facade returned one handle. A loader that returns several checkpoints returns one handle for each of them, in the run’s own order, and every handle supports transform, select, one, and correct exactly as a recorded result does.

Failures and boundaries

Section titled “Failures and boundaries”

A loader that raises changes nothing: the rollback removes the run, the source evidence and the revision. A missing loader, a blank name, or a source above ArchiveLimits.max_source_bytes fails before any read or commit. A later call that asks for the other source policy fails before it reads the file, because the first successful call already froze the policy.

Two honest limits. The loader protocol is deliberately small, so it says nothing about performance on very large files: the capture budget is the boundary, and a format that needs streaming belongs behind a custom route, not a bigger protocol. And a working connector for one file proves that file’s shape, not the format family behind it — vendor formats need corpus evidence before you claim support.

Where next

Section titled “Where next”