Upgrade from an older SampleKit¶
This version knows its own files, its own configuration and its own names
alone: nothing in it recognises, converts or cleans up what an earlier
version wrote. A tool beside it, samplekit-migrate.pyz, does most of an
upgrade for you, in a copy of your project; what an upgrade asks is also said
here, to do by hand, once. Upgrading from 1.0.0-rc.1 is a matter of a few
renames and a clean-up; upgrading from 0.2 means moving your samples, further
down.
The migration tool¶
Nothing guarantees that the tool carries your data correctly. It cannot fit every use, and it will miss some. It works on a copy, so that nothing is lost: it never writes in the folder it reads. Before anything, keep a backup of your project: three copies, on two kinds of media, one of them elsewhere (the 3-2-1 rule), and, better still, a git repository.
samplekit-migrate.pyz is one file, attached to each
release beside the command
line. It reads a project of 0.1.0, 0.1.1, 0.1.2, 0.2.0 or 1.0.0-rc.1 and
writes a new folder this version reads. It installs nothing: run it with the
Python of an environment where this version's package is installed, the
version samplekit --version says.
$ python -m venv .venv
$ .venv/bin/python -m pip install samplekit==VERSION
$ .venv/bin/python samplekit-migrate.pyz old --to new --model old/specimen.py
Without --write, it reads everything and says what it would write. Here, a
tensile project of 0.2, its model in old/specimen.py:
SampleKit migration tool, for SampleKit 1.0.0-rc.3.
Nothing guarantees that this tool carries your data correctly: it cannot fit
every use, and it will miss some. It works on a copy, so that nothing is lost:
it never writes in the folder it reads.
Before anything, keep a backup of your project: three copies, on two kinds of
media, one of them elsewhere (3-2-1), and, better still, a git repository.
Would write 5 files to new:
.samplekitrc
specimen.py
samples/aluminium-01.md
samples/copper-01.md
samples/steel-01.md
Read as SampleKit 0.x (0.1.0 to 0.2.0).
## Carried
- the text below each frontmatter, as the note, as it is: what 0.x wrote there from template() is a copy of the values then, to remove where it is one
- material: no number, written as an attribute of each sample, and out of the model
- operator: no number, written as an attribute of each sample, and out of the model
- width: readings, given 0.x's statistic, their mean and sample standard deviation
- thickness: readings, given 0.x's statistic, their mean and sample standard deviation
## Not carried: computed again
- area, curve.stress, tensile_strength
## Left behind
- the model's default values, gauge_length = 50.0: 0.x wrote them in every file it saved, and they are carried from there; this version's model declares none
- specimen.py: template(), taken out of the model
## To check by hand
- samples/copper-01.md: thickness, an uncertainty typed over its readings, 0.003: carried beside them, and compute writes the statistic's in its place; to keep it, set it again after computing
- specimen.py: the methods _area, _area_unc, _tensile_strength, _stress, copied as 0.x wrote them, each marked # TODO(migrate). A formula is too varied to be translated: read each against this version's API (a table's columns are read self.curve.values("stress"), a row's cells row.force.value)
## What to do next
1. Make the project's environment, in the new folder: `python -m venv .venv`, then `.venv/bin/python -m pip install samplekit==1.0.0-rc.3`.
2. Read the model and every line marked `# TODO(migrate)`.
3. `samplekit . validate`, then `samplekit . compute --write`, which computes again what was not carried.
4. Compare a few samples with the earlier version's by hand before trusting the rest.
Nothing written: --write writes it.
--write writes the new folder, and in it MIGRATION.md, the same report,
with what this version's validate and status say of the copy. Then do
what it says: make the environment, read every line marked
# TODO(migrate), and compute.
From 0.x,
--modelnames the module of your class: a file of 0.x does not say which of its values a formula computed, and the model does. The model is read, never run. A project of this version has one class: where the module holds several classes of samples,--classnames the one your samples were saved with, and samples of another class are migrated apart. Its__init__is written again in this version's API, a value that is no number (text, a date, a flag) becoming an attribute; every other method is copied as it was and marked, since a formula is too varied to be translated. What a formula computed is not carried, andcompute --writegives it again;--keep-computedcarries it, as values typed over their formulas, whichcompute --forcegives back to them.--verify, with--write, computes the copy and compares each value with the one 0.x wrote.From 1.0.0-rc.1, the sample files are this version's already: they are copied as they are, with the history, and the configuration and the model are renamed as the section below says. Bring every value up to date with rc.1 first, as it says too.
samplekit-migrate.pyz --help lists its options. The sections below say what
the tool does, step by step, for those who would rather do it by hand, or
check what it did.
From 1.0.0-rc.2¶
The sample files, the configuration and the history are the same: nothing in a project changes, and the migration tool is not needed.
Install the command line and the package together, on every computer that shares a project, as below: the package of rc.2 in a project's environment is refused by this version's command line, which names both versions.
The terminal interface has themes: it draws in SampleKit's own where the terminal draws every colour, and the roles your
[colors]names are laid over it. To keep the terminal's sixteen colours, as rc.2 drew, chooseterminalwithT, or writetheme = "terminal"in the project's[colors](Choose the colours).The terminal interface keeps every change waiting until
wwrites it, and a value's window is gone:etypes in a cell,iopens a value's card (The terminal interface).
From 1.0.0-rc.1¶
The sample files are the same, schema 1: nothing in them changes. What
changes is how the command line is written, a few keys of .samplekitrc, a
few names in Python, and where SampleKit keeps which formula computed each
value.
Upgrade every computer that shares a project together, as for 0.2: once one has written with this version, a computer still on rc.1 does not read the project's configuration, nor its history, and what it writes then is kept nowhere.
Before you upgrade: bring every value up to date, with rc.1¶
rc.1 recorded which formula computed each value on the computer, in its state
folder. This version keeps it in each project's history, .samplekit/history,
and does not read what rc.1 recorded. At its first write in a project, it
takes every value the files hold as computed by the formulas as they are then,
and says how many.
A formula changed between your last rc.1 computation and that first write
would go unseen. So, with rc.1 still installed, in each project:
$ samplekit status samples/
$ samplekit compute samples/ --write
until status names no value outdated or formula changed (a value that
waits for an input nobody entered yet may stay). Then change no formula until
this version has written once in the project.
If you upgraded first, or are not sure no formula changed, compute again the values of a formula you changed, or every value, with this version:
$ samplekit samples/ compute -p area --rerun --write
$ samplekit samples/ compute --rerun --write
The second takes as long as computing the project from nothing.
Install the command line and the package together¶
The command line and the Python package in each project's environment are
one version: install both, the command line as Install
SampleKit says, and the package with
.venv/bin/python -m pip install samplekit==VERSION, the version
samplekit --version says. .venv/bin/pip show samplekit then names it. The
model runs in the Python [model] python names, an environment folder or
an interpreter; rc.1 found the project's .venv by itself, and this version
looks for none: add the line.
[model]
path = "model/specimen.py"
class = "Specimen"
python = ".venv"
The command line: paths anywhere¶
Paths and options are read before the command and after it: samplekit samples/ status and samplekit status samples/ read the same, and the help,
this guide and what SampleKit says write the path first. The first word
naming a command is the command: a folder of that name is written status/.
A file SampleKit wrote, copied outside its project, is found by samplekit FILE explain in the projects around it (the one holding it, the current
folder's, those below the current folder, those the TUI opened last):
samplekit FILE explain FOLDER is gone, a folder after explain being its
samples. A preview asks apply these changes? [y/N]: a script answers with
--write. compute --try is gone, refused as any unknown option: compute
asks before it computes, and before it writes.
The configuration¶
A key this version does not have is refused, and the configuration is not read until it is renamed:
rc.1 |
Now |
|---|---|
|
|
|
|
the colour role |
|
|
|
samplekit DIR validate names the first it meets, with its line: run it
again after each rename. A configuration written for a build made before rc.1
may hold other keys those builds read and this version does not:
[format.NAME], a style's quantity, unit or property_exact, a query's
base: each is refused the same way, and the configuration
reference gives the keys there are.
A project reads its subfolders unless it says otherwise: a
.samplekitrcwith norecursivenow reads every folder below its own. A project that kept samples it did not want read in a subfolder writesrecursive = falseunder[collection].
Python scripts¶
A property's formulas are renamed, in this order, so that the second does not catch what the first wrote:
compute=becomescompute_value=, thencompute_quantity=becomescompute=.compute=is now the property's formula, returning a number or(value, uncertainty);compute_value=gives the value alone, beside an uncertainty typed, given by readings or bycompute_uncertainty=. Renamed this way, no value is outdated. A value formula left ascompute=still runs, but the property is then the formula's whole: an uncertainty typed beside it is no longer kept, and acompute=given withuncertainty=is refused.A table's columns are properties:
sk.Column(becomessk.Property(, and the index is named,sk.Table(columns, index="strain"). An entry ofcompute_rowsfilling one column becomes that column'scompute=anddepends_on=,("stress", ["row.force", "area"], f)giving"stress": sk.Property(unit="MPa", compute=f, depends_on=["row.force", "area"]); an entry ofcompute_columnslikewise itscompute_column=. An entry filling several columns becomes ask.ComputedColumns(["low", "high"], compute=f, depends_on=[…])in the table'scomputed=, orcompute_column=ffor one ofcompute_columns. The functions are unchanged, and no value is outdated.Reading computes nothing: where the model declares a statistic of readings,
width.valueisNoneuntilcompute()takes it, as the command line shows it.stale()andis_staleareoutdated()andis_outdated.Property.invalidate()is gone:compute(force=True)gives a value back to its formula.Property.textisstr().
A history kept by rc.1¶
rc.1 named each computer's branch in a project's history after its host; this
version names it after an identifier drawn once, kept in the state folder's
machine file. Nothing is renamed: the first write of this version joins the
branch rc.1 kept, and log shows rc.1's snapshots as SampleKit's, the name
rc.1 signed them with, and that write as you, joins HOST. Nothing is lost.
A history shared through git under rc.1 committed files each computer keeps for itself, and the model's description,
.samplekit/model.json. Take them out of the repository once, and declare the sharing,[history] shared = true, as Share a project says:git rm -r --cached .samplekit/history/HEAD .samplekit/history/config .samplekit/history/description .samplekit/history/hooks .samplekit/model.json, thengit add -A.A value restored from a snapshot rc.1 took, by
restore --ator--ata snapshot from before the upgrade, carries no record of the formula that computed it, and is taken as current: compute it again,compute -p NAME --rerun --write, if its formula changed since.
Clean the state folder¶
The state folder holds what SampleKit keeps on this computer alone:
$XDG_STATE_HOME/samplekit, or ~/.local/state/samplekit, on Linux;
~/Library/Application Support/samplekit on macOS;
%LOCALAPPDATA%\samplekit on Windows; $SAMPLEKIT_STATE_DIR where it is
set. This version keeps there its run logs, logs/compute-PROJECT-….log,
the workbench's state, workbench/, and the computer's machine. What rc.1
and the builds before it left there is read by nothing, and can be deleted:
computed.json, computed, consent, accepted/, and the logs named by
run, logs/compute-YYYY-MM-DDT…Z-PID.log. The migration tool lists them,
samplekit-migrate.pyz --clean-state, and removes them with --write,
nothing else of the folder.
From SampleKit 0.2¶
This recipe moves samples written by SampleKit 0.2 into files this version reads, by hand: what the migration tool does, for those who would rather do it themselves or check what it did. The program itself converts nothing. Either way, the formulas of the model are yours to read again: the tool copies them as they were, marked.
Take your time over the model and the configuration: write them, and try them on copies of a few samples, before the collection's files are moved. Then move the files together, everyone who shares them: SampleKit 0.2 cannot read a moved file, and a colleague still on 0.2 who opens one gets errors, or values that are silently empty.
Find the files to move¶
This version knows its own files alone: it does not recognise a file an older
version wrote, nor say that it is one. A sample of SampleKit 0.2 declares no
schema_version, so it is no sample here, left out as a README is; no
command reads it, and none writes it. samplekit samples/ validate counts
them, this file declares no schema_version, with the first one's name: one
run finds them all.
Move the samples¶
Install both versions, each in its own Python environment, since two versions of one package cannot share one:
python -m venv oldandpython -m venv .venv. SampleKit 0.2 is on the Python Package Index:old/bin/pip install samplekit==0.2.0; this one is.venv/bin/pip install samplekit==VERSION, the versionsamplekit --versionsays. Install the command line too (Install SampleKit): it is what reads, checks and computes the new files. The new environment is.venv/, beside the project, and the project's.samplekitrcnames it,python = ".venv"under[model]:computeruns the model with that Python, and looks for no other.Write the new model: its class, its properties and its formulas, as this version declares them (write a model). Then
samplekit init --model model/specimen.py --writewrites the project's.samplekitrc, naming the model and the.venv/already there.Load the samples with the old version and its old model, and write what they hold to a JSON file, which both versions read.
Read the JSON with the new version, set the values on new samples, and save them, into a new folder, never over the originals.
Carry everything a sample holds: each property's value, uncertainty,
readings and unit, its text values, and its tables. A copy
that keeps only the values gives files in which validate finds nothing
wrong, since every number it kept is right. Compare a few files by hand before
trusting the rest.
# ── in the old environment, with SampleKit 0.2 and the old model ──
import json
from pathlib import Path
import old_model # the module that declares your 0.2 class
carried = []
for path in sorted(Path("samples").glob("*.md")):
sample = old_model.Specimen(path)
properties, attributes, tables = {}, {}, {}
# 0.2 held everything as a Property: a number is a property here, and
# anything else, text or a date, an attribute. A value never given is
# carried as nothing.
for name, held in sample.props.items():
if held.value is None:
continue
if isinstance(held.value, bool) or not isinstance(held.value, (int, float)):
attributes[name] = held.value
else:
# Readings are `data` in 0.2: carried, or the new file holds
# their mean and nothing it was taken from.
readings = getattr(held, "data", None)
properties[name] = {"v": held.value, "u": held.uncertainty,
"unit": held.unit,
"readings": list(readings) if readings else None}
# Each table as its rows, each row its cells by column.
for name, table in sample.tables.items():
tables[name] = table.to_yaml()
carried.append({"file": path.name, "name": sample.name,
"properties": properties, "attributes": attributes,
"tables": tables})
Path("carried.json").write_text(json.dumps(carried, indent=1, default=str))
# ── in the new environment, with the new SampleKit and the new model ──
import datetime
import json
from pathlib import Path
import new_model
import samplekit as sk
def restored(value):
"""A date the JSON holds as its text, a date again: saved as text, a date
is refused, since the file would read it back as a date."""
if isinstance(value, str):
try:
return datetime.date.fromisoformat(value)
except ValueError:
pass
return value
Path("new-samples").mkdir(exist_ok=True)
for held in json.loads(Path("carried.json").read_text()):
target = Path("new-samples") / held["file"]
# Run again after an interruption, it goes on where it stopped.
if target.exists():
continue
fresh = new_model.Specimen.new(target)
fresh.name = held["name"]
for name, numbers in held["properties"].items():
declared = getattr(fresh, name)
# What the new model computes is left to it: copied, each value
# would be a value typed over its formula, which it never replaces.
if declared.is_computed:
continue
if numbers["unit"]:
declared.unit = numbers["unit"]
if numbers["readings"]:
# The readings, and the statistic the model declares takes the
# value from them: a value set over them is typed over its formula.
declared.readings = numbers["readings"]
else:
declared.value = numbers["v"]
declared.uncertainty = numbers["u"]
for name, value in held["attributes"].items():
setattr(fresh, name, restored(value))
# Each table's rows, a cell 0.2 wrote with its uncertainty given with it.
for name, table in held["tables"].items():
rows = [{column: sk.Property(value=cell.get("value"),
uncertainty=cell.get("uncertainty"))
if isinstance(cell, dict) else cell
for column, cell in row.items()}
for row in table["_rows"]]
if rows:
getattr(fresh, name).extend(rows)
fresh.save()
Tables are carried in the JSON as 0.2 wrote them, and their rows added to the
tables the new model declares; a column's precision and symbol, which 0.2
wrote in each file, are declared once, [property."curve.force"].
Readings are carried with their property, and the statistic the new model
declares gives the value from them. What the new model computes is not
carried: compute --write computes it, from the values carried. 0.2 had no tags and no note of your
own: the text below its frontmatter was written by template() from the
values when the file was last saved. The migration tool keeps it as the note,
as it was; this recipe carries nothing of it: copy what you want to keep, and
write the note the new files should have. Then:
$ samplekit new-samples/ validate
$ samplekit new-samples/ compute --write
Scripts written for 0.2¶
sort()sorts the list in place and returnsNone, as a Python list'ssort()does:ordered = specimens.sort("area")leavesorderedatNone. Writeordered = specimens.sorted("area")for a sorted copy.stats()gives a weighted mean where every uncertainty is known and none is 0, each value weighted by 1/u²: a script of 0.2 gets another number. Itsweightedsays which mean it took, andstandard_erroris the weighted mean's, 1/√Σw, beside it.invalidate()is gone: what is outdated is known by itself now, andcompute(force=True)gives a value written over its formula back to it.The names 0.2 had and this version has not are unknown attributes or keywords, answered as any typing mistake, with the nearest name:
0.2
Now
s.get_property("x"),s.get_table("x")s["x"], ors.xs.set_property("x", p),s.set_table("x", t)s["x"] = ps.has_property("x")"x" in ss.remove_property("x"),s.remove_table("x")del s["x"]s.set_value(…),set_uncertainty,set_unit,set_symbols.x.value = …,.uncertainty,.unit,.symbols.property_names(),s.table_names()list(s), ors.keys()Sample.load(path),load_dataSample(path)s.notess.notes.render_view(),render_report()samplekit FILE viewsamples.append(…)and the other changes to a listsamples + other,SampleList([...])samples.names()[s.name for s in samples]samples.query_names()samples.project.queriessamples.sort_multi(…),sort(…, reverse=True)sort(["first", "-second"])samples.to_records(),to_dataframe()samples.to_dict(columns=[...])table.cell(i, c),table.column(c)table.at(i, c),table.values(c)p.datap.readingsp.unit_math,symbol_math,precision_unc[unit]and[property]in.samplekitrcp.is_validp.state != "failed", orp.failure, the error it raised; readingp.valuegives its last value, with a warningProperty(compute_unc=…)compute_uncertainty=sk.load(…, sample_class=Specimen)Specimen(path)Readings give their value once computed: where the model declares a statistic,
width.valueisNoneuntilcompute()takes it, as the command line shows it.
What 0.2 wrote in each file, and this version declares once¶
Digits, symbols and LaTeX spellings, 0.2's
precision,precision_unc,symbol_mathandunit_math, written into every file, are declared once for the project,[property.width] precision = ".4f",symbol_math = …, and[unit."mm²"] math = …, in.samplekitrc(configuration). A file holds none of them.The LaTeX forms need their style:
symbol_mathandmathare what--style mathwrites, and nothing without it. Declare the style once,[style.math], and write a table in it:samplekit samples/ --style math.Numbers shown to a digit apart: at the same precision, a few values and uncertainties are shown differently from 0.2:
1.0401where 0.2 showed1.0400for 1.04005,3.54 ± 0.000 %where it showed no uncertainty of 0,—where it showedN/A. The numbers the files hold are the same: this version rounds a value to the nearest, a half up, and an uncertainty up, never understated, on their shortest decimal writing (Values and uncertainty), where 0.2 used Python'sformat().The uncertainty of readings: 0.2 gave the readings' sample standard deviation. Write a model declares their standard error, √n times smaller; to keep 0.2's numbers, declare
uncertainty=sk.stats.sample_stdev.
Keys renamed since 0.2¶
A 0.2 file spelt a property's numbers value and uncertainty, and its
repeated measurements data. They are v, u and readings now, and the
old spellings are not read. The recipe above needs no change for it:
save() writes the new ones. A file that already says schema_version: 1 and
still holds a long spelling is refused for it, unknown field value: rename
the key.
What needs nothing¶
A file written differently from SampleKit's own form is valid, and is
read. The next command that writes it rewrites it in the fixed form;
validate notes it, and there is nothing to run.