Tasks

The task template, the built files, the config hash, and the benchmarks and their task sets. Templates and cohorts are the input of MedOmni's builder, medomni-v1; the built files and task sets are Core contracts that Bench reads. Tasks explains how they fit together.

Format: task template, src/medomni/studio/schemas/task-template.schema.json, read by medomni-v1. The cohort fields are in Recipes.

Tasks in v0.3.0

One folder per template in studio/datasets/<dataset>/tasks/, with its cohorts.

• 30 templates on credentialed data: 15 on AI-READI, 9 on MIMIC-IV, 6 on BDSP
• 3 templates on VitalDB, on open data
• an n20 cohort for every template, and a ladder cohort for the 5 templates of the modality ladder, whose rungs are in each template's arms/modality-ladder.yaml

Task template

studio/datasets/<dataset>/tasks/<template>/template.yaml

Field	Required	Meaning
template_id	yes	<dataset>_<name>; equals the folder name.
dataset	yes	The dataset the template binds to.
authors	yes	Who wrote the template: name and GitHub handle for each.
targets	yes	One or more. Each has a name, a label from the dataset's library, and its metric: auc, qwk or mae. An optional question replaces the label's.
input	yes	The tables and families the proposal named. A record only: the evidence is the patient's whole record minus the targets' guards.
evidence	no	The evidence variants: [default] (default). v0.3.0 builds only default and refuses <template>:full@<cohort>.
exclude	no	Codes or fields the proposal asked to remove. A record only, like input.
agent	yes	What an agent needs to run the task: the four fields below.
agent.statement	yes	The clinical question alone. Every atom task's instruction opens with it.
agent.instruction	yes	The template's own paragraph on the prediction time. Build adds the rest of the instruction.
agent.environment	yes	CPUs, memory, internet access, agent and verifier timeouts.
agent.submission_path	yes	Where the agent writes its answer.

Check a template, its cohort and the build; the exit code is 1 on any failure:

medomni studio task check <template>@<cohort>

Built files

Build writes each task's content once, then renders it into atom tasks under each arm. The files Bench reads are the built-task contract; record.json is medomni-v1's own.

The content is in atomtasks/_content/<template>@<cohort>/, such as atomtasks/_content/mimic-iv_ckd-staging@n20/:

Path	Holds
content.json	The atom list, the targets and their metrics, each atom's prediction time, the task version, the step counts.
answers.csv	Each patient's answer for every target.
manifest.json	The build manifest: the config hash under each arm and variant, the builder (medomni-v1/<version>), the step counts and the release.
<atom>/record.json	The evidence the instruction shows, as structured data.

An atom task is what an agent runs, rendered under atomtasks/<arm>/<atom>/, or atomtasks/<arm>/<variant>/<atom>/ for an arm with variants, such as atomtasks/modality-ladder/r1/<atom>/. Rendering stages the patient's files from the cache. Its files are real copies, never links:

Path	Holds
task.toml	Task metadata, arm, resources, the <template>@<cohort> and the patient.
instruction.md	The clinical question, the prediction time, the patient's record, and each target with its question.
environment/	The closed-network containers, with the perception service when the arm attaches it.
tests/	expected.json (the target names) and format_check.py, a format check on the answer. It never scores.
data/	The patient's staged files, minus anything that leaks a target.

The answers stay in the content folder; a render never copies them into an atom task. The atom tasks of one <template>@<cohort> are scored together against them.

Config hash

Each <template>@<cohort> under each arm, or arm variant, has a config hash: cfg1: and the SHA-256 over its config files, the builder and two release names. Each file adds its path, its length and its bytes, in path order.

• The files: the ones the builder's config_files names. For medomni-v1, the template, the cohort, the labels and recipes the task uses, and the arm's definition with its variant.
• The builder: its name and version, such as medomni-v1.
• The releases: the dataset release and the MedOmni release.

Build writes it to the build manifest, beside the step counts. Run and score records copy it as config_hash. The prefix names the algorithm, so published config hashes stay checkable after an upgrade. It covers no data.

medomni studio task check --release checks every pinned pair in every task set of the workspace's benchmarks. It fails when a pair's cohort, labels, targets, scoring or dataset release differs from the last release and its N has not risen. It compares config files only. MedOmni runs it before a release; your builds and runs never do.

Result records imported from v0.2 runs are version 0 and carry a v0.2: config hash instead, over the v0.2 template's config files. Only imported records may carry one. They name the sets medomni-bench/v0.2-ladder, v0.2-forced and v0.2-mid, one per v0.2 task tree with runs.

Benchmarks and task sets

A benchmark is one folder, bench/benchmarks/<benchmark>/: its card in benchmark.yaml, and its task sets in sets/<set>.yaml. A task set is named <benchmark>/<set>.

Contract: benchmark card, src/medomni/core/contracts/schemas/benchmark.schema.json.

Field	Required	Meaning
name	yes	The benchmark's name; equals the folder name.
version	yes	The benchmark's own version.
owner	yes	Who maintains it.
license	yes	The license of the benchmark's own files. Each dataset keeps its own license.
accepts	yes	The intelligence kinds it may be run with: agent.

A task set pins each <template>@<cohort> at its version N, and nothing else. Task sets only grow, and an entry never changes: a new N enters only a new task set. The set's content hash, task_set_hash, covers the pairs and their N.

Contract: task set, src/medomni/core/contracts/schemas/task-set.schema.json.

Field	Required	Meaning
name	yes	The task set's name, such as v1; equals the file name.
description	no	One line on what it holds.
builder	yes	The task builder that builds the set, such as medomni-v1.
arms	yes	The arms every pair is built under.
tasks	yes	One or more entries {task: <template>@<cohort>, version: <N>}, with N an integer. Imported v0.2 sets use 0.

medomni-bench, MedOmni's own benchmark: version 1.0, owner MedOmni team, license CC-BY-4.0, accepts agent.

Task set	Data	Arms	Holds
medomni-bench/smoke-dummy	Dummy data	encoders	One pair per template, on generated data.
medomni-bench/vitaldb	VitalDB	encoders, no-perception	The 3 VitalDB templates at n20.
medomni-bench/v1	AI-READI, MIMIC-IV, BDSP	encoders, no-perception	The 30 credentialed templates at n20.
medomni-bench/modality-ladder	AI-READI, MIMIC-IV, BDSP	modality-ladder	The 5 ladder templates at their ladder cohort.

Where next

• Families: the inputs a template can name.
• Recipes: the cohort, label and evidence fields.
• Add a task: write a new template and cohort.
• Datasets: what each dataset needs.