125 lines
4.6 KiB
Markdown
125 lines
4.6 KiB
Markdown
# Evidence pipeline
|
|
|
|
This package is the experiment/evidence layer. The pre-prototype executors in
|
|
`experiments.executors` adapt immutable trial records to H1 retargeting, H2
|
|
synthetic sensitivity, and H3/H4 rigid-body simulation backends.
|
|
|
|
## Minimal workflow
|
|
|
|
From the repository root with `PYTHONPATH=code`:
|
|
|
|
```bash
|
|
python -m experiments.cli plan \
|
|
--spec code/config/experiments/h1_smoke.json \
|
|
--output /tmp/h1-plan.json
|
|
|
|
python -m experiments.cli run \
|
|
--plan /tmp/h1-plan.json \
|
|
--batch-dir output/experiments/h1-smoke \
|
|
--executor experiments.executors:execute_h1_retargeting
|
|
```
|
|
|
|
Available executor/config pairs are:
|
|
|
|
```text
|
|
execute_h1_retargeting h1_smoke.json / h1_calibration.json
|
|
execute_h2_synthetic h2_smoke.json / h2_calibration.json
|
|
execute_bilateral_simulation smoke.json / bilateral_calibration.json
|
|
```
|
|
|
|
Auditable second-stage calibration specifications are:
|
|
|
|
```text
|
|
execute_h1_retargeting h1_calibration_v2.json
|
|
execute_h2_synthetic h2_calibration_v2.json
|
|
execute_bilateral_simulation bilateral_calibration_v2_energy.json
|
|
execute_bilateral_simulation bilateral_calibration_v2_network.json
|
|
```
|
|
|
|
The v3 redesign separates branch-crossing evidence from contact/energy stress:
|
|
|
|
```text
|
|
execute_h1_retargeting h1_calibration_v3.json
|
|
execute_bilateral_simulation bilateral_calibration_v3_stable_contact.json
|
|
execute_bilateral_simulation bilateral_calibration_v3_energy_challenge.json
|
|
```
|
|
|
|
`h1_calibration_v3.json` is a deterministic branch-regression fixture, not
|
|
statistical tail-latency evidence. The bilateral v3 stable grid uses three
|
|
exogenous seed groups shared across haptic gains. The synthetic energy
|
|
challenge shares the corresponding `k=3200 N/m` groups, is explicitly
|
|
ineligible for H3, and must be analyzed only with its H4/challenge metric
|
|
configuration.
|
|
|
|
The bilateral network specification is a gated Stage B template. Its
|
|
`requires_stage_a_selection` flag means the haptic parameters are placeholders;
|
|
do not execute it as a locked study until the energy/gain Stage A acceptance
|
|
gate has passed.
|
|
|
|
An executor callable receives one immutable trial mapping and returns:
|
|
|
|
```python
|
|
TrialPayload(
|
|
samples={"time": time_array, "...": sample_array},
|
|
events=[{"sample_index": 10, "event": "contact"}],
|
|
metadata={"backend": "simulation"},
|
|
)
|
|
```
|
|
|
|
Every sample array must have the same first dimension. Object arrays are
|
|
rejected. Successful trials are committed by one atomic directory rename;
|
|
failures are retained separately and may be retried with `--resume`.
|
|
|
|
Validate a completed batch:
|
|
|
|
```bash
|
|
python -m experiments.cli validate \
|
|
--batch-dir output/experiments/h1-smoke
|
|
```
|
|
|
|
Recompute independent endpoints and paper source-data:
|
|
|
|
```bash
|
|
python -m analysis.make_paper_artifacts \
|
|
--batch-dir output/experiments/h1-smoke \
|
|
--metric-config code/config/experiments/metrics_h1_calibration.json
|
|
```
|
|
|
|
Use the matching independent metric configuration:
|
|
|
|
```text
|
|
h1_calibration.json metrics_h1_calibration.json
|
|
h1_calibration_v2.json metrics_h1_calibration_v2.json
|
|
h2_calibration*.json metrics_h2.json
|
|
bilateral_calibration.json metrics_bilateral.json
|
|
bilateral_calibration_v2_*.json metrics_bilateral_v2.json
|
|
h1_calibration_v3.json metrics_h1_calibration_v3.json
|
|
bilateral_calibration_v3_stable_contact.json
|
|
metrics_bilateral_v3_stable_contact.json
|
|
bilateral_calibration_v3_energy_challenge.json
|
|
metrics_bilateral_v3_energy_challenge.json
|
|
```
|
|
|
|
The bilateral configurations derive H3 for every mapping/supervisor condition,
|
|
but their H4 tables contain only tank-supervised methods; PO/PC and bypass
|
|
conditions cannot be silently mixed into a tank audit.
|
|
|
|
The declared minimal storage contract is JSON for manifests/plans, NPZ for
|
|
numeric sample arrays, JSON Lines for events/trial metrics, and CSV for paper
|
|
source-data. No Parquet dependency is required.
|
|
|
|
## Pairing and random numbers
|
|
|
|
`pair_id` excludes the method and therefore identifies common inputs.
|
|
`trial_id` includes the method. The trajectory, sensor, model, and network
|
|
streams are derived independently with NumPy `SeedSequence`; their serialized
|
|
states are identical across methods in the same pair.
|
|
|
|
Calibration, pilot, and locked studies must use separate specifications. A
|
|
locked plan is immutable: changing a factor, method, trajectory, or seed
|
|
invalidates its hashes.
|
|
|
|
Files named `*_locked_template.json` are deliberately not confirmatory plans.
|
|
Copy and freeze them only after calibration thresholds, safety limits, repeat
|
|
counts, the source commit, and the analysis configuration have been approved.
|