Skip to content
MTRL

Use scientific Python libraries

MTRL stores your data; NumPy, pandas, SciPy, and matplotlib analyse it. This guide shows the four hand-offs: a DataFrame in, DataFrames back out, your own functions as operations, and direct plots of committed data.

sample.transform takes your NumPy or SciPy function and commits its output as one result, while NaN-free extraction for these libraries stays result.to_numpy().

This guide assumes: you have finished the tutorial path, so you are comfortable committing checkpoints and reading lineage.

Bring in a pandas DataFrame

Section titled “Bring in a pandas DataFrame”

A DataFrame is one conversion away from a checkpoint. frame.to_records(index=False) turns the columns into the fields of the structured NumPy array that MTRL stores, and the column names become the field names. Three rules keep the conversion honest:

  • Always pass index=False. The DataFrame index holds row labels, not measurements.
  • Keep the columns numeric. The commit boundary accepts a closed set of NumPy dtypes: booleans, signed and unsigned integers of 8 to 64 bits, floats of 16, 32, and 64 bits, complex values, and structured arrays built from those. A text, object, category, datetime, or timedelta column raises UnsupportedCommittedValueError when you commit the run and stores nothing, so keep labels in run metadata instead. Pandas nullable and extension columns need a conversion first: to_records maps a nullable Int64 or Float64 column to a plain NumPy column, and a missing value becomes NaN, which turns an integer column into a float one. Choose the missing-value policy — fill, drop, or keep NaN — and convert with .astype before you commit.
  • Name the columns yourself. MTRL never guesses which column is the axis and which is the signal; your names become the scientific field names.

Give one unit per column in the same names, and commit as usual. The example also shows the two pandas routes back out: one checkpoint’s to_pandas returns its own table, and the application’s to_pandas stacks every active checkpoint behind identity columns.

Required install: vector-mtrl[tabular]

import numpy as np
import pandas as pd


from mtrl import InstrumentRun, MaterialData, Mtrl, Unit


frame = pd.DataFrame(
    {
        "wavenumber": np.linspace(2000.0, 3000.0, 101),
        "absorbance": np.linspace(0.1, 0.9, 101),
    }
)


application = Mtrl(sample_id="polymer-film-01")
run = InstrumentRun(run_name="ftir-replicate-01")
checkpoint = MaterialData(
    content=frame.to_records(index=False),
    type="raw",
    checkpoint_name="spectrum",
    units=Unit.from_raw_map({"wavenumber": "cm^-1", "absorbance": "a.u."}),
)
run.add_checkpoint(checkpoint)
application.add(run)


committed = application.get_active_record(checkpoint.checkpoint_id)
round_trip = committed.to_pandas()
print(f"checkpoint columns: {round_trip.columns.tolist()}")
print(f"values match: {np.allclose(round_trip['absorbance'], frame['absorbance'])}")


container_frame = application.to_pandas()
print(f"container columns: {container_frame.columns.tolist()}")
print(f"container rows: {len(container_frame)}")
checkpoint columns: ['wavenumber', 'absorbance']
values match: True
container columns: ['run_id', 'checkpoint_id', 'checkpoint_name', 'type', 'wavenumber', 'absorbance']
container rows: 101

The checkpoint table holds your values exactly, and the container frame repeats them behind the identity columns, one row per point: 101 rows.

Apply your own NumPy or SciPy function

Section titled “Apply your own NumPy or SciPy function”

Exploration does not wait for a registry. Pass a named function straight to transform: it receives a detached copy of the committed table and returns the new table. MTRL records the result as a custom operation: the lineage holds the function name and the exact parameters, plus a hash of the function source when MTRL can read it. When it cannot — a function typed at a bare interpreter prompt, for example — it warns with a TransformationWarning and records no hash.

A custom operation is recorded-only. The archive records the step, its parameters, and the result, and it keeps the function name and source hash as declared disclosure. It stores no runnable implementation of your function, so the archive cannot know how that function behaved beyond the recorded output. MTRL therefore cannot re-run the step for you: replay_plan refuses with a ReplayIneligibleError that names the target as recorded-only (custom). This is deliberate. Replay eligibility is a reviewed decision, not an accident of having source available. When a function earns that status, register it as an operation: Control the operation registry shows the route.

SciPy takes the same path. Wrap the SciPy call in a named function — a filter, a peak finder, a baseline fit — and pass it to transform. Install SciPy yourself, or take the vector-mtrl[interp] extra. The result is equally recorded-only until you register it.

import numpy as np


from mtrl import InstrumentRun, MaterialData, Mtrl, ReplayIneligibleError, Unit


axis = np.linspace(2000.0, 3000.0, 101)
content = np.empty(axis.size, dtype=[("wavenumber", "<f8"), ("absorbance", "<f8")])
content["wavenumber"] = axis
content["absorbance"] = np.exp(-((axis - 2500.0) / 35.0) ** 2)


application = Mtrl(sample_id="polymer-film-01")
run = InstrumentRun(run_name="ftir-replicate-01")
checkpoint = MaterialData(
    content=content,
    type="raw",
    checkpoint_name="spectrum",
    units=Unit.from_raw_map({"wavenumber": "cm^-1", "absorbance": "a.u."}),
)
run.add_checkpoint(checkpoint)
application.add(run)




def smooth_absorbance(data, *, width=5):
    """Smooth the absorbance field with a centred moving average."""
    result = data.copy()
    kernel = np.ones(width) / width
    result["absorbance"] = np.convolve(data["absorbance"], kernel, mode="same")
    return result




smoothed = application.transform(
    run.run_id,
    checkpoint.checkpoint_id,
    smooth_absorbance,
    parameters={"width": 7},
)
print(f"operation: {smoothed.lineage.transformation.name}")
print(f"parameters: {dict(smoothed.lineage.transformation.parameters)}")


try:
    application.replay_plan(run.run_id, smoothed.checkpoint_id)
except ReplayIneligibleError as error:
    print(f"replay error: {type(error).__name__}")
    print(f"replay refused as recorded-only: {'recorded-only (custom)' in str(error)}")
operation: smooth_absorbance
parameters: {'width': 7}
replay error: ReplayIneligibleError
replay refused as recorded-only: True

The recorded recipe names your function and the exact parameters. The plan then refuses to re-run the custom step, exactly as promised.

Plot committed data directly

Section titled “Plot committed data directly”

MTRL has no plotting of its own, and that is deliberate: you already have a plotting library. Read the committed arrays and hand them to matplotlib, or to any other library you trust. Install it into the same environment as MTRL — in a notebook, %pip install matplotlib into the active kernel. This snippet continues either example above:

import matplotlib.pyplot as plt


record = application.get_active_record(checkpoint.checkpoint_id)
figure, axes = plt.subplots(figsize=(7, 4))
axes.plot(record.content["wavenumber"], record.content["absorbance"])
axes.set_xlabel("wavenumber / cm^-1")
axes.set_ylabel("absorbance / a.u.")
figure.tight_layout()
figure.savefig("spectrum.png", dpi=150)
plt.show()
print("saved spectrum.png")

The committed content never changes when you read it, so plot freely. Cosmetic choices — colours, offsets that separate curves visually — belong to the chart, not to the record.

Ingest a file with your own loader

Section titled “Ingest a file with your own loader”

Sample.ingest is the facade route to the core instrument loader: pass the file, an object that implements the loader protocol, and the required name. The call delegates to Mtrl.load_instrument_file exactly once, so the core keeps the one-pass bounded capture, the source evidence, the frozen policy, and the single atomic commit; the facade only turns the committed run into owned result handles.

The full walkthrough, including the parser/mapper split and a runnable CSV connector, lives in Author a connector.

What just happened?

Section titled “What just happened?”

Four hand-offs, no new API. to_records(index=False) carried a DataFrame into a checkpoint, and to_pandas carried one checkpoint or the whole container back out. Your own function became a recorded custom operation whose replay limits are stated, not hidden. Matplotlib drew committed data without MTRL taking a plotting dependency. The archive stays the source of truth; every one of these routes is a view or a recorded operation on it.

Section titled “Related guides”