Seal Phase 4AK frozen D18 independent validation
This commit is contained in:
Digital Drosophila Archive
@@ -0,0 +1,81 @@
|
||||
# Phase 4AK — frozen D18 validation, not learning
|
||||
|
||||
Prerequisite: the entire Phase 4AJ fixed-CAL experiment was completed, sealed
|
||||
and published at cb6937666f96b00ae8d435a2cfb2f417ad25c040.
|
||||
Its PASS is not evidence of usable learning. This stage tests the next necessary
|
||||
representation-generalization gate without adjusting the successful branch.
|
||||
|
||||
## No new brain dynamics
|
||||
|
||||
Call the exact SHA-pinned AJ GPU, Brian oracle, comparison and timing code from
|
||||
their original absolute paths. Do not copy or patch their numerical functions,
|
||||
globals, signs, weights, thresholds, APL gain/tau, refractory or delay queue.
|
||||
Keep dt=0.1 ms, anatomical delay=18 steps, and whole generated graded-APL h
|
||||
applied 18 steps later. Preserve positive-zero initialization and pending tail;
|
||||
never advance brain time to drain the tail. Shared APL arithmetic limits the
|
||||
oracle independence claim. The 5845 fast-APL output mask, including 1212
|
||||
unreplaced non-KC edges, is unchanged. No biological measurement claim.
|
||||
|
||||
## Input registration and data boundary
|
||||
|
||||
Read only the four SHA-pinned original static 4L encoder files (panel, vectors,
|
||||
selection, mapping) and pinned encoder configuration/source. Extract only the
|
||||
unchanged pn_drive function AST; do not import or execute unrelated learning
|
||||
experiments. Validate the extraction and request generator against all 60 old
|
||||
AG CAL request/PN-drive arrays before generating any new brain trajectory.
|
||||
Keep the original 12-odor median 2.556107997894287 and original target/RNG column
|
||||
order. Matched energy is expected input, not exactly equal realized events.
|
||||
|
||||
Predeclare both 6-odor x2-mode x5-seed blocks, every sparse request schedule,
|
||||
source hash, gate, and stop rule before any new brain trajectory. Seeds are the
|
||||
fixed first five integers above 2^30, subject to the enumerated static collision
|
||||
audit. They are not chosen using simulation outcomes. Do not claim exhaustive
|
||||
historical uniqueness, statistical independence of RNG streams, or biological
|
||||
replication. Same seed across odors preserves the existing paired comparisons.
|
||||
|
||||
The second six odors are novel to AG/AJ calibration, not never simulated in
|
||||
project history: old 4L traversed the full panel. Never open old heldout dynamic
|
||||
results, mixed old result tables or conditioning/plasticity reports to choose
|
||||
inputs. Full frozen-integrity hashing may read opaque bytes for checksums; that
|
||||
is not permission to parse those outcomes. Static A/B identities are not used
|
||||
to select a favorable subset. No new resource download.
|
||||
|
||||
## Ordered execution
|
||||
|
||||
1. Freshly run the two original AJ D18 fullgraph probe inputs. Verify every
|
||||
scientific output against sealed AJ, allowing only output budget/path
|
||||
differences in spec. Do not fake D0 to use a D0-only helper. Old eight-fixture
|
||||
proofs are source-pinned inherited evidence, not newly rerun fixture claims.
|
||||
2. For original CAL odors on new seeds, fixed first odor/first seed in each mode
|
||||
is a Brian fullgraph probe, with independent LIF/own spikes/FIFO and the
|
||||
existing shared-APL component audit. Any numerical failure stops.
|
||||
3. Complete the 60 seed-validation trajectories plus 12 exact repeats. Reuse
|
||||
the two probe GPU runs rather than duplicate them as independent samples.
|
||||
4. Only if the entire seed block passes numerical, repeat and original
|
||||
representation gates, unlock the second block. Run the first static novel
|
||||
odor/first seed in each mode as two additional Brian probes, then complete
|
||||
that block's 60 trajectories and 12 repeats. All planned conditions remain
|
||||
visible; gated unexecuted cases are NOT PASS and are reported as such.
|
||||
5. Recompute raw evidence, rerun the five old frozen reference trials into new
|
||||
output, verify every frozen artifact, and seal a truthful result.
|
||||
|
||||
Each cohort/mode separately retains every original six-part 4L gate: mean KC
|
||||
fraction in [3%,9%], all runs >0 and <50%, same-odor Jaccard > different-odor,
|
||||
between-odor Euclidean > within-odor, and at least 3 of 15 odor pairs passing
|
||||
their own two-odor within baseline. Every mode has 60 within comparisons and
|
||||
75 between comparisons; each pair has 20 within and 5 between. No pooled
|
||||
12-odor rescue, seed replacement, post-hoc metric/window/gate changes or scans.
|
||||
|
||||
## Outcome scope
|
||||
|
||||
PASS-FROZEN-D18-INDEPENDENT-VALIDATION requires all four cohort/mode gates,
|
||||
complete data, fresh controls, oracle/component checks, exact repeats and old
|
||||
frozen integrity/reference PASS. Scientific failure remains FAIL; incomplete
|
||||
execution is not re-labelled scientific success. Passing does not generate a
|
||||
candidate, establish learning, prove delay biology or repair upstream AL loops.
|
||||
Next required gates remain credit specificity, real KC-to-MBON functional
|
||||
transfer, whole-brain stimulus-specific response, and controlled conditioning.
|
||||
|
||||
All outputs are new under phase4ak-apl-timing-validation. Previous Phase 1–4AJ,
|
||||
base CSR/W_syn/signs, eligibility-v2 and the 14232-edge plastic mask remain
|
||||
read-only. No training, reward interface, game, vision, decoder, RL or Web.
|
||||
@@ -0,0 +1,64 @@
|
||||
# Phase 4AK：仅静态种子审计
|
||||
|
||||
状态：`NO-COLLISION-FOUND-IN-DECLARED-STATIC-SCOPE`。这是输入准备审计，不是科学运行、表征验证或学习 PASS。
|
||||
|
||||
## 固定选择
|
||||
|
||||
使用 `2^30` 后的首五个正整数：
|
||||
|
||||
```text
|
||||
1073741825, 1073741826, 1073741827, 1073741828, 1073741829
|
||||
```
|
||||
|
||||
它们均小于 `2^31`。选择只依据静态声明的历史 namespace 与固定算术规则；没有生成随机输入来挑选、没有检查活动或 separability，也没有读取旧 held-out 动态结果。
|
||||
|
||||
新运行直接把这些整数传给原始 CUDA generator，不增加 odor、mode 或验证块偏移。两验证块、气味及模式继续按已提交计划共享同一组 seeds，以保留配对设计；技术重复重用同一 seed。这不意味着跨气味、模式或验证块的随机流彼此独立，也不是生物重复。运行后不得因结果不利换 seeds。
|
||||
|
||||
## 本次覆盖范围
|
||||
|
||||
先阅读已提交的 `phase4aj-next-validation.md`，随后只审计本地静态源码、配置及 AG/AJ 输入清单。机器记录见同目录 `seed-audit.json`：
|
||||
|
||||
- 217 个静态文件，每个记录 workspace-relative 路径、字节数和 SHA-256。
|
||||
- 1,432 条 seed/RNG 相关源码或配置行保留准确行号；词法证据包括测试与非 RNG 的 `seed_node`，不把每一数字都当成真正 RNG seed。
|
||||
- 检索范围为文件中列出的 12 个 source 根目录内 `.py/.yaml/.yml/.json`，排除名字含 manifest/checksum 的文件；另纳入已提交计划与明确允许的 AG/AJ protocol 静态文件。
|
||||
- 完整文件文本未发现五个新种子的十进制字面值；已检查的配置及派生规则也未发现重用。
|
||||
- AG 和 AJ 的 `protocol/calibration-cases.json` 均为 60 项，seeds 为 `51101–51105`，odor indices 为 `0–5`，两份文件 SHA 完全相同：`d91994c77db82cb40a258c92b222e0d7b28338ffe98764bd600988dae6f72754`。
|
||||
|
||||
没有导入或执行旧源码，没有运行 GPU/脑实验，没有访问网络或远程服务器，没有打开旧 `kc-representation`、`eligible-edges`、`plasticity-magnitude`、`conditioning` 或包含它们结果的综合报告。源码中的旧路径字符串是静态文本，不代表这些路径的输出被读取。
|
||||
|
||||
## 不能漏掉的派生规则
|
||||
|
||||
| 静态来源 | 已声明规则／占用范围 |
|
||||
|---|---|
|
||||
| 4A–4D | 配置中的 `41xxx–44xxx`；4C visual seed 加 `1/2/3/4/10+j`；4D 还内联 `44301–44303`。 |
|
||||
| 4E | `45000 + SHA256(layer)前4字节模1000000`，再加 amplitude/rate/sparsity/window 派生偏移；保守保留完整 hash 输出范围，不读取旧最优条件。 |
|
||||
| 4F/4G | `46101–46103`、`47001–47005`、`47101–47103`、`47201–47205`、`47300`、`47400`、`47901/47902`。4G 的 `stable_seed` 定义被记录，但其源码没有调用，不能宣称已执行。 |
|
||||
| 4H–4K | 训练 seed 加 trial index；4J calibration base 加 odor index、evaluation seed 加 `odor_index*100`；4I 固定随机集合 seed `49301`；若干 DAN helper 接收 `seed+991`，显式 pulse 路径不实际抽样，仍保守记录。 |
|
||||
| 4L–4Q | `51201–51205 + pairing(1..100)`，包括 `ts+pairing` 别名；评估 `51301–51305`；4N random-US 再加 `700000`。`512xx/513xx` 不可当作空白 namespace。 |
|
||||
| 4P/4Q/4R | 旧 odor seed 有 `seed+odor_index*100`；`52101–52105` 和 `52201–52205` 已明确声明。random-US 包括 `930000/950000+actual`；4Q formal 还使用 `940000+train_seed*1000+pairing`。 |
|
||||
| 4S | `54101–54105` 或 `55101–55105 + odor_index*100000`；control 加 `975000`；formal `[61000,62000,63000,64000,65000]+pairing`，评估 `71001–71005`。 |
|
||||
| 4T | `86101–86120` 或 `87101–87120 + odor_index*1000000`；live-US RNG 加 `9700000`。 |
|
||||
| 4U | `50000000 + odor_index*2000000 + seed`，seed 为 `118101–118120` 或 `119101–119120`；control 另加 `29000000`。按已声明最多 12 气味取保守 index `0..11`，最大列出参数为 `101119120`。 |
|
||||
| 4V/W/Y/Z、4AB terminal scope | `80000000 + odor_index*1000000 + seed + block*1000003`；seed 包括 `131101–131103/141101`。1000 ms 以 120 ms 分块，保守含 `block=0..8`；offset 模式也先生成这些块再清零，不能忽略 RNG 使用。 |
|
||||
| 4AE | 数值夹具固定 seed `410301`。 |
|
||||
| AG/AJ、AH/AI | AG/AJ 每 case 直接使用 `51101–51105`，没有旧 4P 的 odor 偏移；AH/AI 使用既有轨迹重建，并非新 RNG 流。AF/motif 的 `seed_node(s)` 是 neuron IDs，不是 RNG seeds。 |
|
||||
|
||||
这些固定配置和已解释公式保守保留在 `[0, 200000000]` 排除区间；这不是说区间内所有整数都实际用过。新种子在该区间之外。完整规则、源码路径和原始行见 JSON。
|
||||
|
||||
## SHA 与远端核验契约
|
||||
|
||||
本地 `phase4l.yaml` 的 SHA 与 AG-r2 已登记 `olfactory-grounding-v1/phase4l.yaml` 相同；本地 `phase4p_revalidate.py` 与已登记 `deterministic-runtime-v2/phase4p_revalidate.py` 相同。
|
||||
|
||||
JSON 的 `inspected_files` 提供 `local_path/sha256/remote_path`。仅对有确切模型相对路径且 SHA 匹配预注册的 AG/AJ sources、两个上述输入 pin，以及明确的 experiment/protocol 副本填写 remote_path，共 43 项。remote_path 均相对于 `/data/digital-drosophila`。其他本地平铺副本一律为 null，不能猜服务器子目录。
|
||||
|
||||
主流程应在注册前对非 null 远端路径再次校验 SHA，并执行自身的旧文件及 manifest-chain 完整性检查；本静态审计不替代服务器完整性验证。
|
||||
|
||||
## 严格限制
|
||||
|
||||
1. 未审计未列出的 server-only 文件、终端历史、CLI 覆写、其他分支/历史版本、外部包或其他用户的运行；因此不能说全历史绝对未用过。
|
||||
2. 配置中允许某 seed 不等于历史实际执行过；反之，任意 `reset(seed)` 或外部 caller 可以传入未在配置里列出的整数。词法检索与命名调用点复核不是完整 Python 数据流证明。
|
||||
3. 没有完整历史执行输入注册表；只审计 AG/AJ 两份明确静态 CAL 清单，没有读取旧动态输出来反推使用情况。
|
||||
4. 新整数不构成 PRNG 流统计独立性的数学证明。后六气味仅称“未用于 AG/AJ 校准”，不是项目历史从未运行过。
|
||||
5. 主代理同时创建的 Phase 4AK 文件不属于历史检索范围，避免候选自身造成字面碰撞；本记录必须在新轨迹前纳入预注册。
|
||||
|
||||
结论仅为：这五个值在所列静态证据及配置派生范围内未发现重用，符合本轮预声明选择规则。未开始验证块、conditioning、参数扫描或候选升级。
|
||||
@@ -0,0 +1,18 @@
|
||||
2d3996933b6e5a31e6a737ade5f3ae6869ffd03e01002c793eccf9b4f71e987d PROTOCOL.md
|
||||
35e8bfe1bff62f3a5d9295c785e1b8cec8a2393eaa67d6c683046e48f30feb56 SEED-AUDIT.md
|
||||
1718d8b01ece24930ff138f25c88f2f0e0f1cf69ad0ba655e5a0de5083e64df2 config.json
|
||||
5e4e500d5ec777bb8ae7ff77ac9a7279bfe995b1a17be2baededdbe065c98b73 context.py
|
||||
0061689abb1832e963718bd836bc8a50009ee38b258143a753b4d59e63414925 encoder_inputs.py
|
||||
7eb638c357cec7f5f85032312c50a616402b5a8a9ab5aa0268c181f6c3ae0ce5 finalize.py
|
||||
d8b8315f8800b3efdc7548620f0fae85da21a12b753b9985dd623bd529e2c50f prepare.py
|
||||
bba6c4a98c6edf1eaba60b8030a9eca646da7dd30b74b2a0c16e374efd022bfa run_case.py
|
||||
3d9890f11f299c07956faa5c0a9ebe2ea28ade58d8acf01a8631c17025073345 run_suite.py
|
||||
8a6629f8cd991cf309b508c670b3009737f9b5f5e1f72560fa5bd4b1ff0bc64d runtime_bridge.py
|
||||
e14c0deb94432787aa1cac7618254aeb3db0bcd5b1d8328e20428496d38b220a seed-audit.json
|
||||
f1b3b8a0ca9bc41152485c05fd5f24fb7bb1f76c6ea111ffb941ae1225d914fe test_encoder_inputs.py
|
||||
a85466891352c15b3103d30e7229734ad6694c8a1347d5e0747da3315223c203 test_final_status.py
|
||||
bc01a5ff38d79a7b66ab2eb9e6731164bb8a1e15b5b5c0c1655161c24e2c37cf test_runtime_bridge.py
|
||||
03f5aacb5a1f9460d10b30744100a43c57e5488d32dada79c888529943c82cf0 test_validation.py
|
||||
3464091a4cc2cba18946cb9c9b3d12a69bd3f79f1001096234b3a8467dc602c5 validation.py
|
||||
a736b774e39ff0ad863e8efc720bd5f9ed7d30261776bb57f0a716a7052dba5a verify_frozen.py
|
||||
3206a8666718b4a99e09b8384c87dd25895b5d7038d23fad7b93e40379183ccb manifest.json
|
||||
@@ -0,0 +1,35 @@
|
||||
{
|
||||
"phase": "4AK",
|
||||
"name": "apl-timing-validation-v1",
|
||||
"human_authorization": "允许独立新模型分支测试动力学假设",
|
||||
"gpu_uuid": "GPU-79f3a85d-adce-7833-7003-6488140ca596",
|
||||
"neurons": 166700,
|
||||
"edges": 25582938,
|
||||
"raw_weight_sum": 124177617,
|
||||
"apl_delay_steps": 18,
|
||||
"dt_ms": 0.1,
|
||||
"apl": {"tau_ms": 20.0, "gain": 0.01, "normalization": 1.0},
|
||||
"validation": {
|
||||
"seeds": [1073741825,1073741826,1073741827,1073741828,1073741829],
|
||||
"modes": ["physiological", "matched_energy"],
|
||||
"duration_ms": 100.0,
|
||||
"odor_count": 6,
|
||||
"repeat_first_seed_each_odor": true,
|
||||
"cohorts": ["seed_validation", "novel_to_aj"],
|
||||
"median_scope": "unchanged original frozen 12-odor panel",
|
||||
"median_mapped_response_sum": 2.556107997894287,
|
||||
"full_oracle_policy": "first static odor, first seed, each mode, for each cohort",
|
||||
"unlock_novel": "Complete seed_validation execution, representation and repeat gates must pass before any novel-to-AJ brain trajectory"
|
||||
},
|
||||
"gates": {"sparse_min_fraction": 0.03, "sparse_max_fraction": 0.09, "max_fraction_strict": 0.5, "min_fraction_strict": 0.0, "minimum_separable_odor_pairs": 3, "same_jaccard_gt_between": true, "between_euclidean_gt_within": true, "both_modes_required": true, "finite_required": true, "repeat_exact_required": true},
|
||||
"control_cases": ["cal-physiological-00-s51101", "cal-matched_energy-00-s51101"],
|
||||
"output_budget_bytes": 25769803776,
|
||||
"training": false,
|
||||
"candidate_generation": false,
|
||||
"old_heldout_dynamic_access": false,
|
||||
"new_validation_dynamic_execution": true,
|
||||
"historical_never_simulated_odor_claim": false,
|
||||
"universal_seed_independence_claim": false,
|
||||
"selection_policy": "Fixed source panel order, both cohorts and input schedules registered before any new trajectory. Stop on numerical failure or seed-block representation failure; retain all results, no replacement seeds/odors, no tuning, no pooled gate rescue.",
|
||||
"hypothesis": "The already frozen AJ D18 operator generalizes to new audited seeds and six odors not used in AG/AJ calibration. No change whatsoever to dynamics, input transform, temporal window, graph, signs, APL mask or metric definitions."
|
||||
}
|
||||
@@ -0,0 +1,120 @@
|
||||
"""A new validation envelope; sealed brain implementations remain read-only."""
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.dont_write_bytecode = True
|
||||
|
||||
ROOT = Path('/data/digital-drosophila')
|
||||
MODEL = Path(__file__).resolve().parent
|
||||
EXP = ROOT / 'experiments/phase4ak-apl-timing-validation'
|
||||
AJ_MODEL = ROOT / 'models/male-cns-lif/apl-timing-lab-v1'
|
||||
AJ = ROOT / 'experiments/phase4aj-apl-timing'
|
||||
AG_MODEL = ROOT / 'models/male-cns-lif/author-apl-lab-v2'
|
||||
AG = ROOT / 'experiments/phase4ag-author-apl-r2'
|
||||
COEFF = ROOT / 'experiments/phase4af-author-fullgraph/oracle/no_input/coefficients.json'
|
||||
STATIC = ROOT / 'experiments/phase4l-olfactory-grounding'
|
||||
CFG = json.loads((MODEL / 'config.json').read_text(encoding='utf-8'))
|
||||
PINS = {
|
||||
'models/male-cns-lif/apl-timing-lab-v1/manifest.json': '52b05dd1ff30b6b51ff960fe95904532f5c147731a83f0b5c31f98ca7e53c216',
|
||||
'experiments/phase4aj-apl-timing/manifest.json': '7f490ada0db30785c61954d8b92147378ddc8bc16b229fe3808bbe501caff44a',
|
||||
'models/male-cns-lif/phase4l.py': 'e97d58d8c1f44920d0e953f8f5831ddc86fa0198784e86460e315de4116ac553',
|
||||
'models/male-cns-lif/olfactory-grounding-v1/phase4l.yaml': '9b8bcb7e031bf9063033a1854a439a7cd0bf6730301c2d2d02ca99a76f098d6e',
|
||||
'models/male-cns-lif/apl-timing-lab-v1/gpu_timing.py': 'af1975ba5e5e700208c0a598d3cbe648a3cf10885c6231239b6e4b28f76d173b',
|
||||
'models/male-cns-lif/apl-timing-lab-v1/oracle_timing.py': '39d8c47c94bc2faddf016864a43e7715907fb987f56f7b27098060a88851f6be',
|
||||
'models/male-cns-lif/apl-timing-lab-v1/compare_timing.py': '9010f070fba9b7c9cf2d86187ee2abc57f3ac606ebc08a42baad4f3922b51734',
|
||||
'models/male-cns-lif/apl-timing-lab-v1/timing_queue.py': '54df6d4818e2a0c26e96fb0aaf5d9b71cd964981706da937562d48af79aca14e',
|
||||
'models/male-cns-lif/apl-timing-lab-v1/calibration.py': 'fb15ddeaa108305c8de697bbae36b23ab0414a7c58db51c66dce9feb4a048a0d',
|
||||
'models/male-cns-lif/author-apl-lab-v2/calibration.py': 'edb9cb2d890471fd07afd88c959ef3540b0cd40af490fac3f26396b5e7f5bf9c',
|
||||
'models/male-cns-lif/author-apl-lab-v2/validate_components.py': '5cfc9a4505da6b8fb4e7d2617cdaedaa8475ed46a11ec9bdf0d4d25b1d22c2d3',
|
||||
'experiments/phase4l-olfactory-grounding/odor-encoder/frozen-odor-panel.parquet': '907f16a56e024a8e2112b5751e53b0e6037b329fa110098af3d31b119c10a1a0',
|
||||
'experiments/phase4l-olfactory-grounding/odor-encoder/mapped-glomerular-vectors.npz': 'f057563152e0fa22060080af13019f319e7f36563a775eff829bbc1929c451b0',
|
||||
'experiments/phase4l-olfactory-grounding/odor-encoder/selection.json': '1b832becf9d08935716fe9ba5e3cd3e619e7d208dfb43c4fa197c969d095f113',
|
||||
'experiments/phase4l-olfactory-grounding/receptor-mapping/olfactory-mapping.parquet': 'a01d27000533e9844458a4c559cac084f1c34ca20a1104e97e6a0f72b0cc93b1'
|
||||
}
|
||||
|
||||
|
||||
def digest(path):
|
||||
result = hashlib.sha256()
|
||||
with Path(path).open('rb') as stream:
|
||||
for block in iter(lambda: stream.read(8 << 20), b''):
|
||||
result.update(block)
|
||||
return result.hexdigest()
|
||||
|
||||
|
||||
def load_module(name, path, expected_hash=None):
|
||||
path = Path(path)
|
||||
data = path.read_bytes()
|
||||
if expected_hash is not None:
|
||||
assert hashlib.sha256(data).hexdigest() == expected_hash, str(path)
|
||||
spec = importlib.util.spec_from_file_location(name, path)
|
||||
assert spec and spec.loader
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
sys.modules[name] = module
|
||||
exec(compile(data, str(path), 'exec'), module.__dict__)
|
||||
return module
|
||||
|
||||
|
||||
def pinned(name, relative):
|
||||
return load_module(name, ROOT / relative, PINS[relative])
|
||||
|
||||
|
||||
def read(path):
|
||||
return json.loads(Path(path).read_text(encoding='utf-8'))
|
||||
|
||||
|
||||
def save(path, value):
|
||||
path = Path(path)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with path.open('x', encoding='utf-8') as stream:
|
||||
json.dump(value, stream, indent=2, ensure_ascii=False, allow_nan=False)
|
||||
stream.write('\n')
|
||||
|
||||
|
||||
def sources():
|
||||
return {p.relative_to(MODEL).as_posix(): digest(p) for p in sorted(MODEL.rglob('*')) if p.is_file()
|
||||
and '__pycache__' not in p.parts and p.name not in ('manifest.json', 'checksums.sha256')}
|
||||
|
||||
|
||||
def integrity():
|
||||
old = load_module('sealed_aj_context_ak', AJ_MODEL / 'context.py')
|
||||
result = old.integrity()
|
||||
sys.path.insert(0, str(ROOT / 'models/male-cns-lif/terminal-scope-diagnostic-v1'))
|
||||
from diagnostic import manifest_check
|
||||
for path in (AJ_MODEL, AJ):
|
||||
result[str(path.relative_to(ROOT))] = manifest_check(path, digest)
|
||||
assert all(row['pass'] for row in result.values())
|
||||
return result
|
||||
|
||||
|
||||
def check():
|
||||
registered = read(EXP / 'protocol/preregistered.json')
|
||||
assert registered['sources'] == sources(), 'Registered validation source changed'
|
||||
assert registered['config'] == CFG
|
||||
for row in registered['inputs']:
|
||||
assert digest(ROOT / row['path']) == row['sha256'], row['path']
|
||||
for row in read(EXP / 'protocol/prepared-hashes.json'):
|
||||
assert digest(EXP / row['path']) == row['sha256'], row['path']
|
||||
return registered
|
||||
|
||||
|
||||
def budget():
|
||||
size = sum(p.stat().st_size for p in EXP.rglob('*') if p.is_file())
|
||||
assert size < CFG['output_budget_bytes'], 'Output budget exceeded; preserve partial evidence'
|
||||
return size
|
||||
|
||||
|
||||
def seal(root):
|
||||
rows = [dict(path=p.relative_to(root).as_posix(), bytes=p.stat().st_size, sha256=digest(p))
|
||||
for p in sorted(root.rglob('*')) if p.is_file() and '__pycache__' not in p.parts]
|
||||
save(root / 'manifest.json', dict(root=str(root), files=rows))
|
||||
with (root / 'checksums.sha256').open('x', encoding='utf-8') as stream:
|
||||
for row in rows:
|
||||
stream.write(row['sha256'] + ' ' + row['path'] + '\n')
|
||||
stream.write(digest(root / 'manifest.json') + ' manifest.json\n')
|
||||
|
||||
|
||||
def bit_equal(a, b):
|
||||
return a.dtype == b.dtype and a.shape == b.shape and a.tobytes() == b.tobytes()
|
||||
@@ -0,0 +1,52 @@
|
||||
"""Static-only frozen encoder extraction and exact request-schedule generation."""
|
||||
import ast
|
||||
import hashlib
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
import numpy as np
|
||||
|
||||
|
||||
def frozen_pn_drive(path, expected_sha):
|
||||
"""Execute the exact one pure function, not unrelated historical experiments."""
|
||||
import pandas as pd
|
||||
source = Path(path).read_bytes()
|
||||
assert hashlib.sha256(source).hexdigest() == expected_sha
|
||||
tree = ast.parse(source, filename=str(path))
|
||||
nodes = [node for node in tree.body if isinstance(node, ast.FunctionDef) and node.name == 'pn_drive']
|
||||
assert len(nodes) == 1 and not nodes[0].decorator_list
|
||||
module = ast.Module(body=nodes, type_ignores=[])
|
||||
namespace = {'np': np, 'pd': pd, 'Any': Any}
|
||||
exec(compile(module, str(path), 'exec'), namespace)
|
||||
return namespace['pn_drive'], hashlib.sha256(ast.dump(nodes[0], include_attributes=False).encode()).hexdigest()
|
||||
|
||||
|
||||
def requests_from_forced(forced, targets):
|
||||
targets = np.asarray(targets)
|
||||
assert targets.dtype == np.int32 and targets.ndim == 1 and len(np.unique(targets)) == len(targets)
|
||||
assert forced.dtype == np.bool_ and forced.ndim == 2 and forced.shape[1] == len(targets)
|
||||
ticks, columns = np.nonzero(forced)
|
||||
requests = np.column_stack([ticks, targets[columns]]).astype(np.int32)
|
||||
if len(requests):
|
||||
requests = requests[np.lexsort((requests[:, 1], requests[:, 0]))]
|
||||
assert np.all(np.diff(requests[:, 0].astype(np.int64) * 166700 + requests[:, 1]) > 0)
|
||||
return requests
|
||||
|
||||
|
||||
def make_requests(targets, rates, seed, steps=1000):
|
||||
import torch
|
||||
assert rates.dtype == np.float32 and rates.shape == targets.shape
|
||||
assert np.isfinite(rates).all() and np.all(rates >= 0) and steps == 1000
|
||||
original = rates * .1 / 1000.0
|
||||
explicit = rates * np.float32(.1) / np.float32(1000.)
|
||||
assert original.dtype == np.float32 and np.array_equal(original.view(np.uint32), explicit.view(np.uint32))
|
||||
assert np.all(original <= 1)
|
||||
generator = torch.Generator(device='cuda')
|
||||
generator.manual_seed(int(seed))
|
||||
probabilities = torch.as_tensor(original, dtype=torch.float32, device='cuda')
|
||||
forced = (torch.rand((steps, len(targets)), generator=generator, device='cuda') < probabilities[None, :]).cpu().numpy()
|
||||
return requests_from_forced(forced, targets)
|
||||
|
||||
|
||||
def write_npz(path, **arrays):
|
||||
with Path(path).open('xb') as stream:
|
||||
np.savez_compressed(stream, **arrays)
|
||||
@@ -0,0 +1,170 @@
|
||||
"""Recompute the new evidence, preserve gated non-execution, then seal truthfully."""
|
||||
from context import ROOT, MODEL, EXP, AJ, AJ_MODEL, AG, CFG, read, save, digest, check, integrity, budget, seal
|
||||
|
||||
|
||||
def optional(path):
|
||||
return read(path) if path.exists() else None
|
||||
|
||||
|
||||
def decide_status(*, numerical, seed, novel, integrity_pass, reference_pass, reference_present,
|
||||
known_numerical_failure=False, unauthorized_novel=False):
|
||||
if not integrity_pass or (reference_present and not reference_pass):
|
||||
return 'FAIL-FROZEN-INTEGRITY-OR-REGRESSION'
|
||||
if known_numerical_failure or unauthorized_novel:
|
||||
return 'FAIL-NUMERICAL-OR-EXECUTION-POLICY'
|
||||
if not numerical or not reference_present or not seed or not seed['execution_gate_pass']:
|
||||
return 'INCOMPLETE-VALIDATION-EXECUTION'
|
||||
if not seed['representation_gate_pass']:
|
||||
return 'FAIL-SEED-VALIDATION-REPRESENTATION'
|
||||
if not novel or not novel['execution_gate_pass']:
|
||||
return 'INCOMPLETE-VALIDATION-EXECUTION'
|
||||
if not novel['representation_gate_pass']:
|
||||
return 'FAIL-NOVEL-ODOR-REPRESENTATION'
|
||||
return 'PASS-FROZEN-D18-INDEPENDENT-VALIDATION'
|
||||
|
||||
|
||||
def component_report(group, name):
|
||||
path = EXP / 'component-audit' / group / (name + '.json')
|
||||
if not path.exists():
|
||||
return None
|
||||
report = read(path)
|
||||
output = EXP / 'gpu' / group / (name + '-r0')
|
||||
assert set(report.get('input_sha256', {})) == {'state.npz', 'summary.json',
|
||||
'apl-drive.float32.npy', 'apl-state.float32.npy', 'apl-inhibition.float32.npy'}
|
||||
for filename, checksum in report['input_sha256'].items():
|
||||
assert digest(output / filename) == checksum
|
||||
assert set(report.get('matrix_checks', {})) == {'input_matrix', 'output_matrix', 'input_raw', 'output_raw',
|
||||
'kc_globals', 'apl_globals', 'pn_globals', 'apl_out_ids', 'input_scale', 'output_scale'}
|
||||
assert set(report.get('ast_replay_bitwise_exact', {})) == {'drive', 'state', 'inhibition'}
|
||||
assert set(report.get('float64_measurement', {})) == {'drive', 'state', 'inhibition'}
|
||||
actual = bool(not report.get('error') and all(report['matrix_checks'].values()) and
|
||||
all(report['ast_replay_bitwise_exact'].values()) and report['theoretical_bound']['applicable'] and
|
||||
all(value['bound_violations'] == 0 for value in report['float64_measurement'].values()))
|
||||
assert report['pass_all'] == actual
|
||||
return actual
|
||||
|
||||
|
||||
def main():
|
||||
from pathlib import Path
|
||||
import runtime_bridge as bridge
|
||||
from validation import summarize
|
||||
check(); budget()
|
||||
controls_inventory, records = bridge.inventories(EXP, CFG)
|
||||
controls, comparisons, components, repeats, summaries = {}, {}, {}, {}, {}
|
||||
for row in controls_inventory:
|
||||
name = row['name']
|
||||
path = EXP / 'audit' / ('control-' + name + '.json')
|
||||
if path.exists():
|
||||
actual = bridge.control_identity(EXP / 'gpu/controls' / (name + '-r0'), Path(row['source_output']),
|
||||
EXP / 'cases/controls' / name, Path(row['source_case']), None)
|
||||
assert actual == read(path), 'Raw control evidence changed'
|
||||
controls[name] = actual['pass_all']
|
||||
component = component_report('controls', name)
|
||||
if component is not None:
|
||||
components['controls/' + name] = component
|
||||
gates = {stage: optional(EXP / 'audit' / (stage + '-gate.json')) for stage in bridge.STAGES}
|
||||
for cohort in CFG['validation']['cohorts']:
|
||||
summary_path = EXP / 'validation' / cohort / 'summary.json'
|
||||
if summary_path.exists():
|
||||
actual = summarize(cohort, out_json=None)
|
||||
assert actual == read(summary_path), 'Raw validation evidence changed: ' + cohort
|
||||
summaries[cohort] = actual
|
||||
for row in bridge.probe_rows(records, CFG, cohort):
|
||||
name = row['name']
|
||||
path = EXP / 'audit' / ('compare-' + cohort + '-' + name + '.json')
|
||||
if path.exists():
|
||||
actual = bridge.compare(EXP / 'cases' / cohort / name, EXP / 'gpu' / cohort / (name + '-r0'),
|
||||
EXP / 'oracle' / cohort / name, None)
|
||||
assert actual == read(path), 'Raw oracle evidence changed'
|
||||
comparisons[cohort + '/' + name] = actual['pass_all']
|
||||
component = component_report(cohort, name)
|
||||
if component is not None:
|
||||
components[cohort + '/' + name] = component
|
||||
for row in bridge.cohort_rows(records, CFG, cohort):
|
||||
if row['seed'] != CFG['validation']['seeds'][0]:
|
||||
continue
|
||||
name = row['name']
|
||||
path = EXP / 'audit' / ('repeat-' + cohort + '-' + name + '.json')
|
||||
if path.exists():
|
||||
actual = bridge.compare_gpu_repeats(EXP / 'gpu' / cohort / (name + '-r0'),
|
||||
EXP / 'gpu' / cohort / (name + '-r1'), None)
|
||||
assert actual == read(path), 'Raw repeat evidence changed'
|
||||
repeats[cohort + '/' + name] = actual['pass_all']
|
||||
seed, novel = summaries.get('seed_validation'), summaries.get('novel_to_aj')
|
||||
seed_pass = bool(seed and seed['execution_gate_pass'] and seed['representation_gate_pass'])
|
||||
novel_paths = [str(path.relative_to(EXP)) for root in ('gpu', 'oracle', 'component-audit')
|
||||
for path in (EXP / root / 'novel_to_aj').glob('*')]
|
||||
unauthorized = bool(novel_paths and not seed_pass)
|
||||
required_cohorts = ['seed_validation', 'novel_to_aj'] if seed_pass else ['seed_validation']
|
||||
expected_controls = {row['name'] for row in controls_inventory}
|
||||
expected_comparisons = {cohort + '/' + row['name'] for cohort in required_cohorts
|
||||
for row in bridge.probe_rows(records, CFG, cohort)}
|
||||
expected_components = expected_comparisons | {'controls/' + name for name in expected_controls}
|
||||
expected_repeats = {cohort + '/' + row['name'] for cohort in required_cohorts
|
||||
for row in bridge.cohort_rows(records, CFG, cohort) if row['seed'] == CFG['validation']['seeds'][0]}
|
||||
required_stages = ['controls', 'seed-probes', 'seed-validation'] + (['novel-probes', 'novel-validation'] if seed_pass else [])
|
||||
numerical = bool(set(controls) == expected_controls and set(comparisons) == expected_comparisons and
|
||||
set(components) == expected_components and set(repeats) == expected_repeats and
|
||||
all(all(values.values()) for values in (controls, comparisons, components, repeats)) and
|
||||
all(gates[stage] and gates[stage]['pass_all'] for stage in required_stages) and
|
||||
all(cohort in summaries and summaries[cohort]['execution_gate_pass'] for cohort in required_cohorts))
|
||||
for cohort, summary in summaries.items():
|
||||
stage = 'seed-validation' if cohort == 'seed_validation' else 'novel-validation'
|
||||
if gates[stage]:
|
||||
assert gates[stage]['execution_gate_pass'] == summary['execution_gate_pass']
|
||||
assert gates[stage]['representation_gate_pass'] == summary['representation_gate_pass']
|
||||
frozen = optional(EXP / 'regression/frozen-five.json')
|
||||
integrity_result = integrity()
|
||||
save(EXP / 'regression/old-integrity-after.json', integrity_result)
|
||||
oldpass = all(row['pass'] for row in integrity_result.values())
|
||||
failures = [read(path) for path in sorted((EXP / 'audit').glob('stopped-*.json'))]
|
||||
known_failure = bool(any(not value for group in (controls, comparisons, components, repeats) for value in group.values()) or
|
||||
any(gate is not None and not gate['pass_all'] for gate in gates.values()) or
|
||||
any(not summary['execution_gate_pass'] for summary in summaries.values()) or
|
||||
any(row.get('stop_kind') == 'execution_or_numerical_failure' for row in failures))
|
||||
status = decide_status(numerical=numerical, seed=seed, novel=novel, integrity_pass=oldpass,
|
||||
reference_present=frozen is not None, reference_pass=bool(frozen and frozen['pass_all']),
|
||||
known_numerical_failure=known_failure, unauthorized_novel=unauthorized)
|
||||
present_runs = []
|
||||
for path in sorted((EXP / 'gpu').glob('*/*/summary.json')):
|
||||
value = read(path)
|
||||
present_runs.append(dict(group=path.parent.parent.name, run=path.parent.name,
|
||||
complete=value.get('complete', False), pass_all=value.get('pass_all', False),
|
||||
**{key: value.get(key) for key in ('case', 'nodes', 'edges', 'total_spikes',
|
||||
'diagnostic_simulation_seconds', 'peak_allocated_vram_bytes')}))
|
||||
completed_primary = [row for row in present_runs if row['group'] in CFG['validation']['cohorts']
|
||||
and row['run'].endswith('-r0') and row['complete']]
|
||||
result = dict(status=status, numerical_gate_pass=numerical, summaries=summaries, gates=gates,
|
||||
controls=controls, comparisons=comparisons, components=components, repeats=repeats, stopped=failures,
|
||||
seed_validation_pass=seed_pass, novel_to_aj_pass=bool(novel and novel['pass_all']),
|
||||
novel_dynamics_executed=bool(novel_paths), novel_unlock_policy_violation=unauthorized,
|
||||
planned_primary_cases=120, actual_primary_cases=len(completed_primary), measurements=present_runs,
|
||||
aggregated_valid_primary_cases=sum(summary['valid_base_trials'] for summary in summaries.values()),
|
||||
actual_checked_repeats=sum(summary['checked_repeats'] for summary in summaries.values()),
|
||||
old_frozen_integrity_pass=oldpass, frozen_five_reference_pass=bool(frozen and frozen['pass_all']),
|
||||
unchanged_sealed_runtime=True, dynamics_changed=False, old_heldout_dynamic_results_parsed=False,
|
||||
novel_scope='Unused for AG/AJ calibration, not historically never simulated',
|
||||
shared_APL_arithmetic_oracle_limitation=True, biological_delay_measured=False,
|
||||
APL_nonKC_fast_edges_unreplaced=1212, learning_achieved=False, candidate_generated=False,
|
||||
full_scientific_goal_achieved=False, payload_bytes_before_manifest=budget(),
|
||||
next_gate='Credit specificity and real KC-to-MBON functional transfer before conditioning' if status.startswith('PASS-')
|
||||
else 'Inspect the first failed gate without tuning this frozen branch or replacing validation inputs')
|
||||
save(EXP / 'audit/final-status.json', result)
|
||||
with (EXP / 'REPORT.md').open('x', encoding='utf-8') as stream:
|
||||
stream.write('\n'.join(['# Phase 4AK — frozen D18 independent validation', '', f'Status: `{status}`.', '',
|
||||
f'Numerical gate: {numerical}. Seed-validation pass: {seed_pass}. Novel-to-AJ pass: {result["novel_to_aj_pass"]}.',
|
||||
f'Novel dynamics executed: {bool(novel_paths)}. Unauthorized unlock: {unauthorized}.',
|
||||
'Both cohorts and all requested inputs were locked before trajectories; no tuning or gate changes.',
|
||||
'Original sealed AJ runtime functions are used unchanged, including 1,212 unreplaced non-KC APL fast edges.',
|
||||
'The second six odors were not used for AG/AJ calibration; old 4L had simulated the full panel.',
|
||||
'Static seed audit is bounded, not proof of universally unused or statistically independent RNG streams.',
|
||||
'Brian LIF/FIFO is independent; APL arithmetic is intentionally shared and separately component-audited.',
|
||||
f'Old frozen integrity: {oldpass}. Five old reference trials: {result["frozen_five_reference_pass"]}.',
|
||||
'No learning, plasticity update, candidate, game, decoder, RL or Web. This is not completion of the learning goal.',
|
||||
'Next gate: ' + result['next_gate'], '']))
|
||||
check(); seal(MODEL); seal(EXP)
|
||||
print(status, flush=True)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,90 @@
|
||||
{
|
||||
"root": "/data/digital-drosophila/models/male-cns-lif/apl-timing-validation-v1",
|
||||
"files": [
|
||||
{
|
||||
"path": "PROTOCOL.md",
|
||||
"bytes": 5002,
|
||||
"sha256": "2d3996933b6e5a31e6a737ade5f3ae6869ffd03e01002c793eccf9b4f71e987d"
|
||||
},
|
||||
{
|
||||
"path": "SEED-AUDIT.md",
|
||||
"bytes": 6540,
|
||||
"sha256": "35e8bfe1bff62f3a5d9295c785e1b8cec8a2393eaa67d6c683046e48f30feb56"
|
||||
},
|
||||
{
|
||||
"path": "config.json",
|
||||
"bytes": 2183,
|
||||
"sha256": "1718d8b01ece24930ff138f25c88f2f0e0f1cf69ad0ba655e5a0de5083e64df2"
|
||||
},
|
||||
{
|
||||
"path": "context.py",
|
||||
"bytes": 5834,
|
||||
"sha256": "5e4e500d5ec777bb8ae7ff77ac9a7279bfe995b1a17be2baededdbe065c98b73"
|
||||
},
|
||||
{
|
||||
"path": "encoder_inputs.py",
|
||||
"bytes": 2431,
|
||||
"sha256": "0061689abb1832e963718bd836bc8a50009ee38b258143a753b4d59e63414925"
|
||||
},
|
||||
{
|
||||
"path": "finalize.py",
|
||||
"bytes": 11542,
|
||||
"sha256": "7eb638c357cec7f5f85032312c50a616402b5a8a9ab5aa0268c181f6c3ae0ce5"
|
||||
},
|
||||
{
|
||||
"path": "prepare.py",
|
||||
"bytes": 12405,
|
||||
"sha256": "d8b8315f8800b3efdc7548620f0fae85da21a12b753b9985dd623bd529e2c50f"
|
||||
},
|
||||
{
|
||||
"path": "run_case.py",
|
||||
"bytes": 3382,
|
||||
"sha256": "bba6c4a98c6edf1eaba60b8030a9eca646da7dd30b74b2a0c16e374efd022bfa"
|
||||
},
|
||||
{
|
||||
"path": "run_suite.py",
|
||||
"bytes": 7914,
|
||||
"sha256": "3d9890f11f299c07956faa5c0a9ebe2ea28ade58d8acf01a8631c17025073345"
|
||||
},
|
||||
{
|
||||
"path": "runtime_bridge.py",
|
||||
"bytes": 22443,
|
||||
"sha256": "8a6629f8cd991cf309b508c670b3009737f9b5f5e1f72560fa5bd4b1ff0bc64d"
|
||||
},
|
||||
{
|
||||
"path": "seed-audit.json",
|
||||
"bytes": 376926,
|
||||
"sha256": "e14c0deb94432787aa1cac7618254aeb3db0bcd5b1d8328e20428496d38b220a"
|
||||
},
|
||||
{
|
||||
"path": "test_encoder_inputs.py",
|
||||
"bytes": 2771,
|
||||
"sha256": "f1b3b8a0ca9bc41152485c05fd5f24fb7bb1f76c6ea111ffb941ae1225d914fe"
|
||||
},
|
||||
{
|
||||
"path": "test_final_status.py",
|
||||
"bytes": 2040,
|
||||
"sha256": "a85466891352c15b3103d30e7229734ad6694c8a1347d5e0747da3315223c203"
|
||||
},
|
||||
{
|
||||
"path": "test_runtime_bridge.py",
|
||||
"bytes": 23574,
|
||||
"sha256": "bc01a5ff38d79a7b66ab2eb9e6731164bb8a1e15b5b5c0c1655161c24e2c37cf"
|
||||
},
|
||||
{
|
||||
"path": "test_validation.py",
|
||||
"bytes": 17829,
|
||||
"sha256": "03f5aacb5a1f9460d10b30744100a43c57e5488d32dada79c888529943c82cf0"
|
||||
},
|
||||
{
|
||||
"path": "validation.py",
|
||||
"bytes": 21631,
|
||||
"sha256": "3464091a4cc2cba18946cb9c9b3d12a69bd3f79f1001096234b3a8467dc602c5"
|
||||
},
|
||||
{
|
||||
"path": "verify_frozen.py",
|
||||
"bytes": 2417,
|
||||
"sha256": "a736b774e39ff0ad863e8efc720bd5f9ed7d30261776bb57f0a716a7052dba5a"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,180 @@
|
||||
"""Lock both validation cohorts before outcomes; read no old novel-odor dynamics."""
|
||||
from datetime import datetime, timezone
|
||||
import ast
|
||||
import json
|
||||
import platform
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import yaml
|
||||
from context import (ROOT, MODEL, EXP, AJ, AJ_MODEL, AG, CFG, PINS, STATIC,
|
||||
read, save, digest, sources, integrity, budget, bit_equal)
|
||||
from encoder_inputs import frozen_pn_drive, make_requests, write_npz
|
||||
|
||||
|
||||
def copy_exact(source, target):
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
with source.open('rb') as source_stream, target.open('xb') as target_stream:
|
||||
shutil.copyfileobj(source_stream, target_stream)
|
||||
assert digest(source) == digest(target)
|
||||
|
||||
|
||||
def main():
|
||||
assert not EXP.exists() or not any(EXP.iterdir()), 'Never reuse a populated experiment'
|
||||
assert shutil.disk_usage(EXP.parent).free > CFG['output_budget_bytes'] + 10 * 1024**3
|
||||
for relative, expected in PINS.items():
|
||||
assert digest(ROOT / relative) == expected, relative
|
||||
previous = read(AJ / 'audit/final-status.json')
|
||||
assert previous['status'] == 'PASS-FIXED-CAL-APL-TIMING-HYPOTHESIS'
|
||||
assert previous['delay18_representation_pass'] and previous['old_frozen_integrity_and_five_reference_pass']
|
||||
assert CFG['apl'] == read(AJ_MODEL / 'config.json')['apl']
|
||||
assert CFG['gates'] == read(AJ_MODEL / 'config.json')['gates']
|
||||
assert CFG['apl_delay_steps'] == 18 and not CFG['training'] and not CFG['candidate_generation']
|
||||
seed_audit = read(MODEL / 'seed-audit.json')
|
||||
assert seed_audit['proposed_seeds'] == CFG['validation']['seeds'] and seed_audit['collision_found'] is False
|
||||
seed_sources = []
|
||||
for row in seed_audit['inspected_files']:
|
||||
if row.get('remote_path'):
|
||||
path = ROOT / row['remote_path']
|
||||
assert path.is_relative_to(ROOT) and '..' not in path.parts
|
||||
assert digest(path) == row['sha256'], row['remote_path']
|
||||
seed_sources.append(dict(path=path.relative_to(ROOT).as_posix(), sha256=row['sha256']))
|
||||
assert seed_sources, 'At least exact-mapped static seed sources must be checked on server'
|
||||
for name in ('protocol', 'cases', 'gpu', 'oracle', 'component-audit', 'audit', 'validation', 'regression', 'logs', 'extension-build'):
|
||||
(EXP / name).mkdir(parents=True, exist_ok=False)
|
||||
command = [sys.executable, '-B', '-m', 'unittest', 'discover', '-v', '-s', str(MODEL), '-p', 'test_*.py']
|
||||
test = subprocess.run(command, cwd=MODEL, text=True, capture_output=True)
|
||||
with (EXP / 'regression/unit-tests.log').open('x', encoding='utf-8') as stream:
|
||||
stream.write(test.stdout + test.stderr)
|
||||
save(EXP / 'regression/unit-tests.json', dict(command=command, exit_code=test.returncode,
|
||||
pass_all=test.returncode == 0, sha256=digest(EXP / 'regression/unit-tests.log')))
|
||||
assert test.returncode == 0
|
||||
inputs = {relative: expected for relative, expected in PINS.items()}
|
||||
for row in seed_sources:
|
||||
inputs[row['path']] = row['sha256']
|
||||
for path in (AJ / 'audit/final-status.json', AJ / 'protocol/preregistered.json',
|
||||
AG / 'protocol/calibration-cases.json', AG / 'protocol/encoder-audit.json'):
|
||||
inputs[path.relative_to(ROOT).as_posix()] = digest(path)
|
||||
save(EXP / 'protocol/preregistered.json', dict(timestamp_utc=datetime.now(timezone.utc).isoformat(),
|
||||
sources=sources(), config=CFG, inputs=[dict(path=path, sha256=sha) for path, sha in sorted(inputs.items())],
|
||||
parent_commit=subprocess.check_output(['git', '-C', str(ROOT), 'rev-parse', 'HEAD'], text=True).strip(),
|
||||
prior_AJ_CAL_outcomes_known=True, before_new_requests_and_brain_trajectories=True,
|
||||
training=False, candidate_generated=False, historical_never_simulated_odor_claim=False,
|
||||
seed_collision_audit_scope='Only enumerated static sources and explicitly evaluated derivations; no universal independence claim',
|
||||
shared_seed_across_conditions=True, new_dynamic_scope='seed_validation then conditionally novel_to_aj',
|
||||
source_engine='Unmodified SHA-pinned AJ backend functions, using original __file__ and globals',
|
||||
oracle_limit='Independent Brian LIF and delay FIFO, shared frozen Torch APL component',
|
||||
python=sys.version, machine=platform.machine(), numpy=np.__version__))
|
||||
save(EXP / 'regression/old-integrity-before.json', integrity())
|
||||
from run_case import preflight
|
||||
preflight()
|
||||
import torch
|
||||
cfg_path = ROOT / 'models/male-cns-lif/olfactory-grounding-v1/phase4l.yaml'
|
||||
oc = yaml.safe_load(cfg_path.read_text())
|
||||
assert oc['odor_encoder']['duration_ms'] == CFG['validation']['duration_ms']
|
||||
assert oc['odor_encoder']['modes'] == CFG['validation']['modes']
|
||||
panel = pd.read_parquet(STATIC / 'odor-encoder/frozen-odor-panel.parquet')
|
||||
assert len(panel) == 12 and panel.odor_id.is_unique
|
||||
assert panel.split.tolist() == ['calibration'] * 6 + ['heldout'] * 6
|
||||
median = float(np.median(panel.mapped_response_sum))
|
||||
assert median == CFG['validation']['median_mapped_response_sum']
|
||||
selection = read(STATIC / 'odor-encoder/selection.json')
|
||||
mapping = pd.read_parquet(STATIC / 'receptor-mapping/olfactory-mapping.parquet')
|
||||
with np.load(STATIC / 'odor-encoder/mapped-glomerular-vectors.npz', allow_pickle=False) as source:
|
||||
assert set(source.files) == set(panel.odor_id) | {'glomeruli'}
|
||||
assert source['glomeruli'].tolist() == selection['glomeruli_order']
|
||||
vectors = {oid: source[oid].copy() for oid in panel.odor_id}
|
||||
encoder_source = ROOT / 'models/male-cns-lif/phase4l.py'
|
||||
drive, ast_sha = frozen_pn_drive(encoder_source, PINS[str(encoder_source.relative_to(ROOT))])
|
||||
cached = {}
|
||||
static_rows = {}
|
||||
for cohort, frame in (('seed_validation', panel.iloc[:6]), ('novel_to_aj', panel.iloc[6:])):
|
||||
static_rows[cohort] = [dict(odor_id=str(odor.odor_id), odor_index=index,
|
||||
source_panel_split=str(odor.split)) for index, (_, odor) in enumerate(frame.iterrows())]
|
||||
for mode in CFG['validation']['modes']:
|
||||
for index, (_, odor) in enumerate(frame.iterrows()):
|
||||
targets, rates, ignored_frame = drive(str(odor.odor_id), vectors[odor.odor_id], mapping,
|
||||
selection['glomeruli_order'], mode, median, oc)
|
||||
assert targets.dtype == np.int32 and rates.dtype == np.float32 and len(targets) > 0
|
||||
assert len(np.unique(targets)) == len(targets) and np.all((targets >= 0) & (targets < CFG['neurons']))
|
||||
assert np.isfinite(rates).all() and np.all(rates > 0)
|
||||
cached[(cohort, mode, index)] = (targets, rates)
|
||||
# Reproduce all 60 known CAL input arrays, without any old heldout trajectory.
|
||||
control_rows = []
|
||||
old_inventory = read(AG / 'protocol/calibration-cases.json')
|
||||
assert len(old_inventory) == 60
|
||||
for row in old_inventory:
|
||||
targets, rates = cached[('seed_validation', row['mode'], row['odor_index'])]
|
||||
old_case = AG / 'cases' / row['name']
|
||||
spec = read(old_case / 'spec.json')
|
||||
assert spec['odor_id'] == static_rows['seed_validation'][row['odor_index']]['odor_id']
|
||||
with np.load(old_case / 'pn-drive.npz', allow_pickle=False) as old:
|
||||
assert bit_equal(targets, old['targets']) and bit_equal(rates, old['rates_hz'])
|
||||
requests = make_requests(targets, rates, row['seed'])
|
||||
with np.load(old_case / 'requests.npz', allow_pickle=False) as old:
|
||||
assert bit_equal(requests[:, 0], old['tick']) and bit_equal(requests[:, 1], old['neuron_idx'])
|
||||
control_rows.append(dict(case=row['name'], pass_all=True, seed=row['seed'],
|
||||
old_requests_sha256=digest(old_case / 'requests.npz'), old_drive_sha256=digest(old_case / 'pn-drive.npz'),
|
||||
target_rate_and_request_arrays_bitwise=True))
|
||||
save(EXP / 'protocol/encoder-control.json', dict(pass_all=True, cases=control_rows,
|
||||
old_heldout_dynamic_access=False, comparison='All original 60 CAL request/drive arrays; no trajectory or RNG-column reorder'))
|
||||
template = read(AJ / 'cases/delay18' / CFG['control_cases'][0] / 'spec.json')
|
||||
assert template['apl_delay_steps'] == 18 and template['steps'] == 1000 and template['record_history'] is False
|
||||
control_cases = []
|
||||
for name in CFG['control_cases']:
|
||||
source = AJ / 'cases/delay18' / name
|
||||
target = EXP / 'cases/controls' / name
|
||||
target.mkdir(parents=True, exist_ok=False)
|
||||
spec = read(source / 'spec.json')
|
||||
spec.update(budget_root=str(EXP), output_budget_bytes=CFG['output_budget_bytes'])
|
||||
save(target / 'spec.json', spec)
|
||||
for filename in ('requests.npz', 'pn-drive.npz'):
|
||||
copy_exact(source / filename, target / filename)
|
||||
control_cases.append(dict(name=name, source_case=str(source), source_output=str(AJ / 'gpu/delay18' / (name + '-r0'))))
|
||||
save(EXP / 'protocol/control-cases.json', control_cases)
|
||||
records = []
|
||||
for cohort, frame in (('seed_validation', panel.iloc[:6]), ('novel_to_aj', panel.iloc[6:])):
|
||||
for mode in CFG['validation']['modes']:
|
||||
for index, (_, odor) in enumerate(frame.iterrows()):
|
||||
targets, rates = cached[(cohort, mode, index)]
|
||||
for seed in CFG['validation']['seeds']:
|
||||
requests = make_requests(targets, rates, seed)
|
||||
name = f'{cohort}-{mode}-{index:02d}-s{seed}'
|
||||
target = EXP / 'cases' / cohort / name
|
||||
target.mkdir(parents=True, exist_ok=False)
|
||||
expected = float(rates.sum()) * 1000 * .1 / 1000.
|
||||
spec = dict(template, name=name, kind='fullgraph', validation_cohort=cohort,
|
||||
source_panel_split=str(odor.split), source_panel_row=int(odor.name),
|
||||
odor_id=str(odor.odor_id), odor_name=str(odor.odor_name), mode=mode, seed=seed,
|
||||
odor_index=index, expected_external_events=expected, target_count=len(targets),
|
||||
rate_sum_hz=float(rates.sum()), budget_root=str(EXP), output_budget_bytes=CFG['output_budget_bytes'])
|
||||
save(target / 'spec.json', spec)
|
||||
write_npz(target / 'requests.npz', tick=requests[:, 0], neuron_idx=requests[:, 1])
|
||||
write_npz(target / 'pn-drive.npz', targets=targets, rates_hz=rates)
|
||||
records.append(dict(name=name, cohort=cohort, source_panel_split=str(odor.split),
|
||||
source_panel_row=int(odor.name), mode=mode, odor_id=str(odor.odor_id), odor_index=index,
|
||||
seed=seed, targets=len(targets), requests=len(requests), expected_external_events=expected,
|
||||
rates_sum=float(rates.sum())))
|
||||
assert len(records) == 120 and len({row['name'] for row in records}) == 120
|
||||
save(EXP / 'protocol/validation-cases.json', records)
|
||||
save(EXP / 'protocol/encoder-audit.json', dict(panel_count=12, cal_count=6, novel_to_aj_count=6,
|
||||
median_mapped_response_sum=median, seeds=CFG['validation']['seeds'],
|
||||
seeds_static_audit_sha256=digest(MODEL / 'seed-audit.json'), cohorts_static_panel_rows=static_rows,
|
||||
requested_before_any_dynamics=True, historical_never_simulated_claim=False,
|
||||
old_heldout_dynamic_access=False, seed_namespace_universally_independent_claim=False,
|
||||
source_function='Unmodified AST node pn_drive from pinned phase4l.py; only np/pd/typing dependencies',
|
||||
source_sha256=digest(encoder_source), function_ast_sha256=ast_sha, torch=torch.__version__, numpy=np.__version__,
|
||||
control_60_original_CAL_arrays_exact=True, target_rng_column_order='Original function output, unchanged',
|
||||
generation='Original float32 rates*.1/1000 GPU Bernoulli; same seed across conditions; sort only resulting sparse requests',
|
||||
expected_not_realized_input_energy_matched=True, no_learning=True))
|
||||
save(EXP / 'protocol/prepared-hashes.json', [dict(path=p.relative_to(EXP).as_posix(), sha256=digest(p), bytes=p.stat().st_size)
|
||||
for directory in ('protocol', 'cases') for p in sorted((EXP / directory).rglob('*'))
|
||||
if p.is_file() and p.name != 'prepared-hashes.json'])
|
||||
budget()
|
||||
print('REGISTERED 2 controls +120 fixed validation cases; 60 original input arrays exact; no brain trajectories', flush=True)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,66 @@
|
||||
"""One registered AK action per fresh process; no changes to frozen runtimes."""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import urllib.request
|
||||
|
||||
from context import ROOT, EXP, CFG, check, budget, read
|
||||
import runtime_bridge as bridge
|
||||
|
||||
|
||||
def preflight():
|
||||
# Establish visibility before the lazy runtime/component Torch import.
|
||||
os.environ['CUDA_VISIBLE_DEVICES'] = CFG['gpu_uuid']
|
||||
os.environ['CUDA_HOME'] = '/usr/local/cuda'
|
||||
os.environ['PATH'] = str(ROOT / 'models/male-cns-lif/.venv/bin') + ':/usr/local/cuda/bin:' + os.environ['PATH']
|
||||
os.environ['MAX_JOBS'] = '2'
|
||||
os.environ['TORCH_CUDA_ARCH_LIST'] = '8.9'
|
||||
for port in (8188, 8189, 8190):
|
||||
with urllib.request.urlopen(f'http://192.168.10.51:{port}/queue', timeout=5) as response:
|
||||
queue = json.load(response)
|
||||
bridge.require(not queue['queue_running'] and not queue['queue_pending'], 'ComfyUI busy: defer, never evict')
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('backend', choices=['gpu', 'oracle', 'component'])
|
||||
parser.add_argument('group', choices=bridge.GROUPS)
|
||||
parser.add_argument('case')
|
||||
parser.add_argument('--repeat', type=int, default=0)
|
||||
args = parser.parse_args()
|
||||
check(); budget()
|
||||
row = bridge.authorize_case(EXP, CFG, args.group, args.case, args.backend, args.repeat)
|
||||
case = EXP / 'cases' / args.group / args.case
|
||||
spec = bridge.case_spec(case)
|
||||
bridge.require(spec['name'] == args.case and spec['budget_root'] == str(EXP) and
|
||||
spec['output_budget_bytes'] == CFG['output_budget_bytes'], 'Case identity/new budget differs')
|
||||
if args.group == 'controls':
|
||||
bridge.control_inputs(case, row['source_case'])
|
||||
else:
|
||||
bridge.require(spec.get('kind') == 'fullgraph' and spec.get('validation_cohort') == args.group and
|
||||
all(spec[key] == row[key] for key in ('mode', 'odor_id', 'odor_index', 'seed')),
|
||||
'New fullgraph case differs from its registered cohort cell')
|
||||
output = EXP / 'gpu' / args.group / f'{args.case}-r{args.repeat}'
|
||||
if args.backend == 'component':
|
||||
summary = read(output / 'summary.json')
|
||||
bridge.require(summary.get('complete') is True and summary.get('pass_all') is True and
|
||||
summary.get('case') == args.case and summary.get('apl_delay_steps') == 18,
|
||||
'Component audit needs a complete matching D18 GPU run')
|
||||
preflight()
|
||||
def execute():
|
||||
if args.backend == 'gpu':
|
||||
result = bridge.run_gpu(case, output, EXP / 'extension-build')
|
||||
elif args.backend == 'oracle':
|
||||
result = bridge.run_oracle(case, EXP / 'oracle' / args.group / args.case)
|
||||
else:
|
||||
result = bridge.component(output, spec['apl_bundle_dir'], EXP / 'component-audit' / args.group / (args.case + '.json'))
|
||||
budget()
|
||||
bridge.require(result.get('pass_all') is True, 'Runtime/component evidence did not pass its own integrity checks')
|
||||
return result
|
||||
result = bridge.guarded_action(EXP, args.backend, args.group, args.case, args.repeat, execute)
|
||||
print(json.dumps(dict(backend=args.backend, group=args.group, case=args.case, repeat=args.repeat,
|
||||
complete=result.get('complete'), pass_all=result.get('pass_all')), allow_nan=False), flush=True)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,151 @@
|
||||
"""Ordered fixed-D18 validation; representation failure never becomes execution failure.
|
||||
|
||||
Only the two predeclared first-odor/first-seed mode probes use Brian per cohort.
|
||||
Their GPU r0 outputs are reused in the full 60-case block. No novel-to-AJ
|
||||
trajectory can start until the complete seed block and its repeats pass.
|
||||
"""
|
||||
import argparse
|
||||
import subprocess
|
||||
import time
|
||||
import traceback
|
||||
|
||||
from context import ROOT, MODEL, EXP, CFG, check, save, read, budget
|
||||
import runtime_bridge as bridge
|
||||
|
||||
|
||||
SCIENCE = ROOT / 'models/male-cns-lif/.venv/bin/python'
|
||||
ORACLE = ROOT / 'experiments/phase4ad-author-semantics/environment/.venv/bin/python'
|
||||
|
||||
|
||||
class RepresentationGateFailed(RuntimeError):
|
||||
"""Completed valid trajectories did not pass the predeclared representation gate."""
|
||||
|
||||
|
||||
def child(backend, group, name, repeat=0):
|
||||
# Do not resolve either interpreter: the venv invocation path is meaningful.
|
||||
interpreter = ORACLE if backend == 'oracle' else SCIENCE
|
||||
bridge.require(interpreter.is_file(), 'Missing fixed runtime interpreter: ' + str(interpreter))
|
||||
bridge.authorize_case(EXP, CFG, group, name, backend, repeat)
|
||||
log = EXP / 'logs' / f'{backend}-{group}-{name}-r{repeat}.log'
|
||||
log.parent.mkdir(parents=True, exist_ok=True)
|
||||
command = [str(interpreter), '-B', str(MODEL / 'run_case.py'), backend, group, name, '--repeat', str(repeat)]
|
||||
print('START', backend, group, name, repeat, flush=True)
|
||||
started = time.perf_counter()
|
||||
with log.open('x', encoding='utf-8') as stream:
|
||||
result = subprocess.run(command, stdout=stream, stderr=subprocess.STDOUT, timeout=1800, check=False)
|
||||
print('FINISH', backend, group, name, repeat, 'exit', result.returncode,
|
||||
'wall', round(time.perf_counter() - started, 2), flush=True)
|
||||
bridge.require(result.returncode == 0, f'{backend} {group}/{name} failed; preserve partial evidence and inspect {log}')
|
||||
|
||||
|
||||
def control(row):
|
||||
name = row['name']
|
||||
child('gpu', 'controls', name)
|
||||
case = EXP / 'cases/controls' / name
|
||||
output = EXP / 'gpu/controls' / (name + '-r0')
|
||||
report = bridge.control_identity(output, row['source_output'], case, row['source_case'],
|
||||
EXP / 'audit' / ('control-' + name + '.json'))
|
||||
bridge.require(report['pass_all'], 'Fresh fixed-D18 control differs from sealed AJ: ' + name)
|
||||
child('component', 'controls', name)
|
||||
return dict(case=name, group='controls', pass_all=True, fresh_gpu_runs=1, new_oracle_runs=0)
|
||||
|
||||
|
||||
def probe(row):
|
||||
name, group = row['name'], row['cohort']
|
||||
child('gpu', group, name)
|
||||
child('oracle', group, name)
|
||||
result = bridge.compare(EXP / 'cases' / group / name, EXP / 'gpu' / group / (name + '-r0'),
|
||||
EXP / 'oracle' / group / name, EXP / 'audit' / ('compare-' + group + '-' + name + '.json'))
|
||||
bridge.require(result['pass_all'], 'Closed-loop fixed-D18 comparison failed: ' + name)
|
||||
child('component', group, name)
|
||||
return dict(case=name, group=group, pass_all=True, fresh_gpu_runs=1, new_oracle_runs=1)
|
||||
|
||||
|
||||
def validation_block(cohort, records):
|
||||
from validation import summarize
|
||||
rows = bridge.cohort_rows(records, CFG, cohort)
|
||||
probes = {row['name'] for row in bridge.probe_rows(records, CFG, cohort)}
|
||||
repeated = []
|
||||
for row in rows:
|
||||
name = row['name']
|
||||
if name not in probes:
|
||||
child('gpu', cohort, name)
|
||||
else:
|
||||
comparison = read(EXP / 'audit' / ('compare-' + cohort + '-' + name + '.json'))
|
||||
bridge.require(comparison.get('pass_all') is True and comparison.get('case') == name,
|
||||
'Missing successful probe evidence for reused r0: ' + name)
|
||||
if row['seed'] == CFG['validation']['seeds'][0]:
|
||||
child('gpu', cohort, name, 1)
|
||||
result = bridge.compare_gpu_repeats(EXP / 'gpu' / cohort / (name + '-r0'),
|
||||
EXP / 'gpu' / cohort / (name + '-r1'), EXP / 'audit' / ('repeat-' + cohort + '-' + name + '.json'))
|
||||
bridge.require(result['pass_all'], 'Fixed-input GPU repeat diverged: ' + name)
|
||||
repeated.append(name)
|
||||
bridge.require(len(rows) == 60 and len(repeated) == 12, 'Incomplete registered base/repeat block')
|
||||
result = summarize(cohort)
|
||||
return result, repeated
|
||||
|
||||
|
||||
def run(stage):
|
||||
bridge.require(stage in bridge.STAGES, 'Unknown stage')
|
||||
check(); budget()
|
||||
bridge.ensure_unstopped(EXP)
|
||||
receipt = EXP / 'audit' / (stage + '-gate.json')
|
||||
bridge.require(not receipt.exists() and not (EXP / 'audit' / ('stopped-' + stage + '.json')).exists(),
|
||||
'Stage already has completion/stop evidence; do not rerun or overwrite')
|
||||
controls, records = bridge.inventories(EXP, CFG)
|
||||
try:
|
||||
if stage == 'controls':
|
||||
rows = [control(row) for row in controls]
|
||||
result = dict(stage=stage, pass_all=True, cases=rows, scope='Same-input fresh D18 GPU identity with sealed AJ; no new model or Brian run')
|
||||
else:
|
||||
bridge.gate_pass(EXP, 'controls')
|
||||
cohort = 'seed_validation' if stage.startswith('seed-') else 'novel_to_aj'
|
||||
if cohort == 'novel_to_aj':
|
||||
bridge.gate_pass(EXP, 'seed-validation', representation=True, cfg=CFG)
|
||||
if stage.endswith('-probes'):
|
||||
rows = [probe(row) for row in bridge.probe_rows(records, CFG, cohort)]
|
||||
result = dict(stage=stage, pass_all=True, cases=rows,
|
||||
scope='Two fixed first-odor first-seed mode probes; independent LIF with shared unchanged Torch APL component')
|
||||
else:
|
||||
bridge.gate_pass(EXP, 'seed-probes' if cohort == 'seed_validation' else 'novel-probes')
|
||||
summary, repeats = validation_block(cohort, records)
|
||||
execution = summary.get('execution_gate_pass') is True
|
||||
representation = summary.get('representation_gate_pass') is True
|
||||
if execution:
|
||||
bridge.validate_summary_coverage(summary, EXP, CFG, cohort)
|
||||
result = dict(stage=stage, cohort=cohort, pass_all=execution,
|
||||
execution_gate_pass=execution, representation_gate_pass=representation,
|
||||
base_count=60, repeat_count=len(repeats), repeat_cases=repeats,
|
||||
summary='validation/' + cohort + '/summary.json',
|
||||
summary_sha256=bridge.digest(EXP / 'validation' / cohort / 'summary.json'),
|
||||
validation_inventory_sha256=bridge.digest(EXP / 'protocol/validation-cases.json'),
|
||||
scope='Complete fixed representation block; no pooled rescue, learning, tuning, or historically-unseen-odor claim')
|
||||
save(receipt, result)
|
||||
if stage.endswith('-validation'):
|
||||
bridge.require(result['execution_gate_pass'], 'Validation execution/numerical/integrity gate failed')
|
||||
if not result['representation_gate_pass']:
|
||||
raise RepresentationGateFailed('Completed fixed validation block failed representation; no later cohort is authorized')
|
||||
budget()
|
||||
return result
|
||||
except Exception as error:
|
||||
save(EXP / 'audit' / ('stopped-' + stage + '.json'), dict(stage=stage,
|
||||
type=type(error).__name__, message=str(error), traceback=traceback.format_exc(),
|
||||
stop_kind='representation_failure' if isinstance(error, RepresentationGateFailed) else 'execution_or_numerical_failure',
|
||||
representation_failure_is_execution_failure=False,
|
||||
later_stages_not_permitted=True, partial_outputs_preserved=True,
|
||||
parameters_changed=False, replacement_seeds_or_odors_selected=False))
|
||||
raise
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('stage', choices=bridge.STAGES)
|
||||
args = parser.parse_args()
|
||||
try:
|
||||
run(args.stage)
|
||||
except RepresentationGateFailed:
|
||||
raise SystemExit(2)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,396 @@
|
||||
"""Phase 4AK orchestration only: call hash-pinned, unchanged AJ functions.
|
||||
|
||||
No LIF, APL, queue, or input-generation equation is implemented here. Fresh
|
||||
processes resolve AJ's one unqualified import to the pinned AJ timing_queue.
|
||||
The D18 control adapter compares evidence; it never changes a case to D0.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
import sys
|
||||
import traceback
|
||||
|
||||
|
||||
AJ_PINS = {
|
||||
'gpu_timing.py': 'af1975ba5e5e700208c0a598d3cbe648a3cf10885c6231239b6e4b28f76d173b',
|
||||
'oracle_timing.py': '39d8c47c94bc2faddf016864a43e7715907fb987f56f7b27098060a88851f6be',
|
||||
'compare_timing.py': '9010f070fba9b7c9cf2d86187ee2abc57f3ac606ebc08a42baad4f3922b51734',
|
||||
'timing_queue.py': '54df6d4818e2a0c26e96fb0aaf5d9b71cd964981706da937562d48af79aca14e',
|
||||
}
|
||||
COMPONENT_SHA256 = '5cfc9a4505da6b8fb4e7d2617cdaedaa8475ed46a11ec9bdf0d4d25b1d22c2d3'
|
||||
COHORTS = ('seed_validation', 'novel_to_aj')
|
||||
GROUPS = ('controls',) + COHORTS
|
||||
STAGES = ('controls', 'seed-probes', 'seed-validation', 'novel-probes', 'novel-validation')
|
||||
_BUDGET_KEYS = frozenset(('budget_root', 'output_budget_bytes'))
|
||||
_LOADED = {}
|
||||
|
||||
|
||||
def require(ok, message):
|
||||
if not ok:
|
||||
raise ValueError(message)
|
||||
|
||||
|
||||
def read(path):
|
||||
return json.loads(Path(path).read_text(encoding='utf-8'))
|
||||
|
||||
|
||||
def digest(path):
|
||||
result = hashlib.sha256()
|
||||
with Path(path).open('rb') as stream:
|
||||
for block in iter(lambda: stream.read(8 << 20), b''):
|
||||
result.update(block)
|
||||
return result.hexdigest()
|
||||
|
||||
|
||||
def save(path, report):
|
||||
if path is None:
|
||||
return
|
||||
path = Path(path)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with path.open('x', encoding='utf-8') as stream:
|
||||
json.dump(report, stream, ensure_ascii=False, indent=2, sort_keys=True, allow_nan=False)
|
||||
stream.write('\n')
|
||||
|
||||
|
||||
def load_checked(name, path, expected_hash):
|
||||
"""Execute the verified bytes with importlib metadata, without old pyc writes."""
|
||||
path = Path(path).absolute()
|
||||
data = path.read_bytes()
|
||||
require(hashlib.sha256(data).hexdigest() == expected_hash, 'Sealed source SHA mismatch: ' + str(path))
|
||||
if name in sys.modules:
|
||||
existing = sys.modules[name]
|
||||
require(Path(getattr(existing, '__file__', '')).absolute() == path and
|
||||
_LOADED.get(name) == (path, expected_hash, existing),
|
||||
'Refusing an existing unverified module name: ' + name)
|
||||
return existing
|
||||
spec = importlib.util.spec_from_file_location(name, path)
|
||||
require(spec is not None and spec.loader is not None, 'Cannot create sealed module specification')
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
sys.modules[name] = module
|
||||
try:
|
||||
exec(compile(data, str(path), 'exec'), module.__dict__)
|
||||
except BaseException:
|
||||
sys.modules.pop(name, None)
|
||||
raise
|
||||
# Keep verification metadata here, not in the sealed module's globals.
|
||||
_LOADED[name] = (path, expected_hash, module)
|
||||
return module
|
||||
|
||||
|
||||
def sealed(filename, aj_model=None):
|
||||
require(filename in AJ_PINS, 'Unregistered AJ module')
|
||||
if aj_model is None:
|
||||
from context import AJ_MODEL
|
||||
aj_model = AJ_MODEL
|
||||
directory = Path(aj_model)
|
||||
if filename == 'gpu_timing.py':
|
||||
load_checked('timing_queue', directory / 'timing_queue.py', AJ_PINS['timing_queue.py'])
|
||||
name = 'phase4ak_sealed_aj_' + Path(filename).stem
|
||||
return load_checked(name, directory / filename, AJ_PINS[filename])
|
||||
|
||||
|
||||
def case_spec(case_dir):
|
||||
spec = read(Path(case_dir) / 'spec.json')
|
||||
require(isinstance(spec.get('name'), str) and re.fullmatch(r'[A-Za-z0-9_-]+', spec['name']), 'Unsafe case name')
|
||||
require(type(spec.get('apl_delay_steps')) is int and spec['apl_delay_steps'] == 18, 'AK permits fixed D18 only')
|
||||
require(spec.get('steps') == 1000 and type(spec['steps']) is int and spec.get('dt_ms') == .1 and
|
||||
spec.get('duration_ms') == 100.0 and spec.get('record_history') is False,
|
||||
'AK requires the unchanged fullgraph 100 ms window')
|
||||
require(spec.get('training') is False and spec.get('poi_used', False) is False and
|
||||
spec.get('precision') == 'float64' and spec.get('apl_precision') == 'float32',
|
||||
'Frozen no-learning / no-poi / precision scope changed')
|
||||
require(spec.get('apl') == {'tau_ms': 20, 'gain': .01, 'normalization': 1}, 'Frozen APL parameters changed')
|
||||
for field in ('graph_dir', 'apl_bundle_dir', 'coefficients_path', 'science_python', 'legacy_apl_worker'):
|
||||
require(field in spec and Path(spec[field]).is_absolute(), 'Missing absolute path: ' + field)
|
||||
require(isinstance(spec.get('model'), dict), 'Missing original author parameter dictionary')
|
||||
return spec
|
||||
|
||||
|
||||
def _new_output(case_dir, out_dir):
|
||||
spec = case_spec(case_dir)
|
||||
output = Path(out_dir).absolute()
|
||||
budget_root = Path(spec['budget_root']).absolute()
|
||||
require(type(spec.get('output_budget_bytes')) is int and spec['output_budget_bytes'] > 0,
|
||||
'Missing positive output budget')
|
||||
require(output.is_relative_to(budget_root) and output != budget_root and not output.exists(),
|
||||
'Output must be a new exclusive child of this case budget root')
|
||||
return spec
|
||||
|
||||
|
||||
def _receipt(output, backend, filename, aj_model):
|
||||
module = sealed(filename, aj_model)
|
||||
require(digest(module.__file__) == AJ_PINS[filename], 'Sealed runtime changed during invocation')
|
||||
receipt = {'backend': backend, 'sealed_runtime': str(module.__file__),
|
||||
'sealed_runtime_sha256': AJ_PINS[filename], 'apl_delay_steps': 18,
|
||||
'old_module_scientific_globals_changed': False, 'dynamics_copied_or_reimplemented': False,
|
||||
'scope': 'Unchanged AJ runtime called with new case/output/build arguments; not a new numerical model'}
|
||||
save(Path(output) / 'runtime-bridge.json', receipt)
|
||||
|
||||
|
||||
def run_gpu(case_dir, out_dir, build_root, *, aj_model=None):
|
||||
spec = _new_output(case_dir, out_dir)
|
||||
build_root = Path(build_root).absolute()
|
||||
require(build_root.is_relative_to(Path(spec['budget_root']).absolute()) and
|
||||
build_root != Path(spec['budget_root']).absolute(), 'Build cache must stay in the new experiment')
|
||||
module = sealed('gpu_timing.py', aj_model)
|
||||
result = module.run_gpu(Path(case_dir), Path(out_dir), build_root)
|
||||
_receipt(out_dir, 'gpu', 'gpu_timing.py', aj_model)
|
||||
return result
|
||||
|
||||
|
||||
def run_oracle(case_dir, out_dir, *, aj_model=None):
|
||||
_new_output(case_dir, out_dir)
|
||||
require('torch' not in sys.modules, 'Pinned Brian process must not import Torch')
|
||||
module = sealed('oracle_timing.py', aj_model)
|
||||
result = module.run_case(Path(case_dir), Path(out_dir))
|
||||
_receipt(out_dir, 'oracle', 'oracle_timing.py', aj_model)
|
||||
return result
|
||||
|
||||
|
||||
def compare(case_dir, gpu_dir, cpu_dir, out_json=None, *, aj_model=None):
|
||||
case_spec(case_dir)
|
||||
return sealed('compare_timing.py', aj_model).compare(case_dir, gpu_dir, cpu_dir, out_json)
|
||||
|
||||
|
||||
def compare_gpu_repeats(first, second, out_json=None, *, aj_model=None):
|
||||
require(all(read(Path(path) / 'summary.json').get('apl_delay_steps') == 18 for path in (first, second)),
|
||||
'Repeat evidence must be fixed D18')
|
||||
return sealed('compare_timing.py', aj_model).compare_gpu_repeats(first, second, out_json)
|
||||
|
||||
|
||||
def component(output, bundle, out_json, *, ag_model=None):
|
||||
if ag_model is None:
|
||||
from context import AG_MODEL
|
||||
ag_model = AG_MODEL
|
||||
module = load_checked('phase4ak_sealed_ag_components', Path(ag_model) / 'validate_components.py', COMPONENT_SHA256)
|
||||
return module.validate(Path(output), Path(bundle), Path(out_json))
|
||||
|
||||
|
||||
def control_inputs(newcase, oldcase):
|
||||
"""Only budget metadata may differ; all saved request/drive bytes are identical."""
|
||||
newcase, oldcase = Path(newcase), Path(oldcase)
|
||||
fresh, old = case_spec(newcase), case_spec(oldcase)
|
||||
require({k: v for k, v in fresh.items() if k not in _BUDGET_KEYS} ==
|
||||
{k: v for k, v in old.items() if k not in _BUDGET_KEYS},
|
||||
'D18 control scientific specification changed beyond the two budget fields')
|
||||
evidence = {str(folder / 'spec.json'): digest(folder / 'spec.json') for folder in (newcase, oldcase)}
|
||||
files = ['requests.npz', 'pn-drive.npz']
|
||||
require((newcase / 'arrays.npz').exists() == (oldcase / 'arrays.npz').exists(), 'Control initial-state presence changed')
|
||||
if (newcase / 'arrays.npz').exists():
|
||||
files.append('arrays.npz')
|
||||
for name in files:
|
||||
first, second = digest(newcase / name), digest(oldcase / name)
|
||||
require(first == second, 'Control input byte mismatch: ' + name)
|
||||
evidence[str(newcase / name)], evidence[str(oldcase / name)] = first, second
|
||||
return fresh, old, evidence
|
||||
|
||||
|
||||
def control_identity(newdir, olddir, newcase, oldcase, out_json=None, *, aj_model=None):
|
||||
"""D18 scientific identity using unchanged AJ comparison primitives.
|
||||
|
||||
Unlike the AJ D0-vs-AG adapter, this checks real D18 applied/pending files.
|
||||
Input SHA fields are checked against each real spec, not compared after
|
||||
deleting metadata. Only build/cache paths and the two budget fields are
|
||||
permitted to differ; all numerical/timing/component evidence stays exact.
|
||||
"""
|
||||
require(out_json is None or not Path(out_json).exists(), 'Refusing existing control report')
|
||||
aj = sealed('compare_timing.py', aj_model)
|
||||
checks, differences, used = {}, {}, set()
|
||||
report = {'kind': 'fixed-D18 same-input sealed-AJ control identity', 'pass_all': False,
|
||||
'new_output': str(newdir), 'old_output': str(olddir), 'new_case': str(newcase), 'old_case': str(oldcase),
|
||||
'apl_delay_steps': 18, 'tolerance': None, 'signed_zero_included': True,
|
||||
'dynamics_copied_or_reimplemented': False, 'old_globals_patched': False,
|
||||
'independent_CPU_all_components_claim': False,
|
||||
'allowed_case_metadata_differences': sorted(_BUDGET_KEYS),
|
||||
'build_and_wall_metadata_compared': False}
|
||||
try:
|
||||
spec, old_spec, inputs = control_inputs(newcase, oldcase)
|
||||
used.update(Path(path) for path in inputs)
|
||||
checks['control_input_spec_and_bytes'] = True
|
||||
steps, n, edges = spec['steps'], 166700, 25582938
|
||||
left = aj._load_run(Path(newdir), 'fresh_gpu', n, edges, steps, False, spec['name'], 18, used, checks)
|
||||
right = aj._load_run(Path(olddir), 'sealed_gpu', n, edges, steps, False, spec['name'], 18, used, checks)
|
||||
aj._pair_outputs(left, right, checks, differences, n, steps, False)
|
||||
for role, run, case, case_input in (('fresh_gpu', left, Path(newcase), spec),
|
||||
('sealed_gpu', right, Path(oldcase), old_spec)):
|
||||
aj._timing_sequence_checks(run['apl']['inhibition'], run['apl']['applied'], run['pending'],
|
||||
18, role, checks, differences)
|
||||
aj._timing_metadata_checks(run, role, 18, steps, checks)
|
||||
expected = {name: digest(case / name) for name in ('spec.json', 'requests.npz')}
|
||||
if (case / 'arrays.npz').exists():
|
||||
expected['arrays.npz'] = digest(case / 'arrays.npz')
|
||||
weight, coeff = Path(case_input['graph_dir']) / 'weights.npy', Path(case_input['coefficients_path'])
|
||||
used.update((weight, coeff))
|
||||
expected.update({'base_weights.npy': digest(weight), 'coefficients.json': digest(coeff)})
|
||||
checks[role + '.actual_input_hashes'] = run['summary'].get('input_sha256') == expected
|
||||
stable = ('case', 'complete', 'pass_all', 'steps', 'completed_ticks', 'nodes', 'edges', 'precision',
|
||||
'apl_precision', 'state_units', 'total_spikes', 'requested', 'accepted_requests', 'rejected_requests',
|
||||
'arriving_edges', 'accepted_arriving_edges', 'counts_match_sparse_events', 'all_finite', 'record_history',
|
||||
'apl_fast_edges_zeroed', 'kc_targets_present', 'effective_weights_raw_sha256', 'apl_schedule',
|
||||
'apl_si_conversion', 'sealed_ae_backend_sha256', 'learning', 'apl_delay_steps', 'apl_timing',
|
||||
'source_provenance', 'pending_generated_count', 'pending_initial_zero_prefix', 'pending_tail_shape',
|
||||
'future_generated_decay_tail_included')
|
||||
for key in stable:
|
||||
checks['summary.' + key] = key in left['summary'] and key in right['summary'] and left['summary'][key] == right['summary'][key]
|
||||
checks['timing_audit.parsed_exact'] = left['timing'] == right['timing']
|
||||
report.update(case=spec['name'], steps=steps, nodes=n, edges=edges,
|
||||
scope='Six state arrays, every tick state/queue/request, generated/applied/pending APL, component and timing metadata')
|
||||
except Exception as error:
|
||||
report['error'] = {'type': type(error).__name__, 'message': str(error)}
|
||||
return aj._finish(report, checks, differences, used, out_json)
|
||||
|
||||
|
||||
def inventories(exp, cfg):
|
||||
"""Validate static cells before choosing any dynamic case; no old dynamic read."""
|
||||
exp = Path(exp)
|
||||
controls = read(exp / 'protocol/control-cases.json')
|
||||
records = read(exp / 'protocol/validation-cases.json')
|
||||
require(isinstance(controls, list) and len(controls) == 2 and
|
||||
[row['name'] for row in controls] == cfg['control_cases'], 'Unexpected control inventory')
|
||||
for row in controls:
|
||||
require(re.fullmatch(r'[A-Za-z0-9_-]+', row['name']) and
|
||||
all(Path(row[key]).is_absolute() for key in ('source_case', 'source_output')), 'Invalid control identity/path')
|
||||
design = cfg['validation']
|
||||
require(design['cohorts'] == list(COHORTS) and len(design['seeds']) == 5 and len(set(design['seeds'])) == 5 and
|
||||
design['modes'] == ['physiological', 'matched_energy'] and design['odor_count'] == 6 and
|
||||
design['repeat_first_seed_each_odor'] is True, 'Fixed validation design changed')
|
||||
require(isinstance(records, list) and len(records) == 120, 'Expected 120 predeclared cases')
|
||||
cells, names = set(), set()
|
||||
for row in records:
|
||||
key = (row['cohort'], row['mode'], row['odor_index'], row['seed'])
|
||||
require(re.fullmatch(r'[A-Za-z0-9_-]+', row['name']) and row['name'] not in names and key not in cells,
|
||||
'Duplicate/unsafe case identity')
|
||||
require(type(row['odor_index']) is int and type(row['seed']) is int, 'Noninteger odor/seed')
|
||||
cells.add(key); names.add(row['name'])
|
||||
expected = {(cohort, mode, oi, seed) for cohort in COHORTS for mode in design['modes']
|
||||
for oi in range(6) for seed in design['seeds']}
|
||||
require(cells == expected, 'Missing or substituted validation cell')
|
||||
return controls, records
|
||||
|
||||
|
||||
def cohort_rows(records, cfg, cohort):
|
||||
require(cohort in COHORTS, 'Unknown validation cohort')
|
||||
design = cfg['validation']
|
||||
return sorted((row for row in records if row['cohort'] == cohort),
|
||||
key=lambda row: (design['modes'].index(row['mode']), row['odor_index'], design['seeds'].index(row['seed'])))
|
||||
|
||||
|
||||
def probe_rows(records, cfg, cohort):
|
||||
rows = [row for row in cohort_rows(records, cfg, cohort)
|
||||
if row['odor_index'] == 0 and row['seed'] == cfg['validation']['seeds'][0]]
|
||||
require(len(rows) == 2 and [row['mode'] for row in rows] == cfg['validation']['modes'], 'Fixed probe selection changed')
|
||||
return rows
|
||||
|
||||
|
||||
def ensure_unstopped(exp):
|
||||
markers = sorted((Path(exp) / 'audit').glob('stopped-*.json'))
|
||||
require(not markers, 'A prior stop marker forbids all new scientific actions: ' + ', '.join(path.name for path in markers))
|
||||
|
||||
|
||||
def guarded_action(exp, backend, group, name, repeat, action):
|
||||
"""Persist failure after invoking an authorized scientific action, even standalone.
|
||||
|
||||
The caller performs input authorization and external idle preflight first.
|
||||
A failure here is an execution/audit failure, not evidence of instability.
|
||||
"""
|
||||
require(backend in ('gpu', 'oracle', 'component') and group in GROUPS and
|
||||
isinstance(name, str) and re.fullmatch(r'[A-Za-z0-9_-]+', name) and
|
||||
type(repeat) is int and repeat in (0, 1), 'Invalid guarded action identity')
|
||||
ensure_unstopped(exp)
|
||||
try:
|
||||
return action()
|
||||
except Exception as error:
|
||||
save(Path(exp) / 'audit' / f'stopped-case-{backend}-{group}-{name}-r{repeat}.json', {
|
||||
'stage': 'case', 'backend': backend, 'group': group, 'case': name, 'repeat': repeat,
|
||||
'type': type(error).__name__, 'message': str(error), 'traceback': traceback.format_exc(),
|
||||
'stop_kind': 'execution_or_numerical_failure', 'scientific_action_invoked': True,
|
||||
'numerical_instability_inferred_from_execution_error': False,
|
||||
'later_stages_not_permitted': True, 'partial_outputs_preserved': True,
|
||||
'parameters_changed': False, 'replacement_seeds_or_odors_selected': False})
|
||||
raise
|
||||
|
||||
|
||||
def validate_summary_coverage(summary, exp, cfg, cohort, *, representation=False):
|
||||
"""Bind the unlock to this cohort's exact 60 base / 12 repeat / 72 timing cells."""
|
||||
_, records = inventories(exp, cfg)
|
||||
rows = cohort_rows(records, cfg, cohort)
|
||||
expected = {row['name'] for row in rows}
|
||||
repeated = {row['name'] for row in rows if row['seed'] == cfg['validation']['seeds'][0]}
|
||||
timing = {(name, 0) for name in expected} | {(name, 1) for name in repeated}
|
||||
require(summary.get('cohort') == cohort and summary.get('phase') == '4AK' and
|
||||
summary.get('apl_delay_steps') == 18 and summary.get('seeds') == cfg['validation']['seeds'] and
|
||||
summary.get('duration_ms') == 100.0, 'Validation summary identity/window differs')
|
||||
require(summary.get('expected_base_trials') == summary.get('valid_base_trials') == 60 and
|
||||
summary.get('expected_repeats') == summary.get('checked_repeats') == 12 and
|
||||
summary.get('checked_timing_trials') == 72, 'Incomplete validation coverage')
|
||||
base_rows, repeat_rows, timing_rows = (summary.get(key, []) for key in ('runs', 'repeats', 'timing_checks'))
|
||||
require(len(base_rows) == 60 and {row['case'] for row in base_rows} == expected and
|
||||
all(row.get('cohort') == cohort and row.get('validation_cohort') == cohort for row in base_rows),
|
||||
'Base-trial identities differ from the fixed cohort inventory')
|
||||
require(len(repeat_rows) == 12 and {row['case'] for row in repeat_rows} == repeated and
|
||||
all(row.get('pass_all') is True for row in repeat_rows), 'Exact-repeat identities/coverage failed')
|
||||
require(len(timing_rows) == 72 and {(row['case'], row['repeat']) for row in timing_rows} == timing and
|
||||
all(type(row['repeat']) is int and row.get('pass_all') is True and row.get('apl_delay_steps') == 18
|
||||
and row.get('steps') == 1000 for row in timing_rows), 'Timing identities/coverage failed')
|
||||
proof = summary.get('input_output_provenance', [])
|
||||
require(len(proof) == 72 and {(row['case'], row['repeat']) for row in proof} == timing,
|
||||
'Per-run evidence provenance does not cover exactly the registered cells')
|
||||
require(summary.get('prerequisite_input_sha256', {}).get('validation_inventory') ==
|
||||
digest(Path(exp) / 'protocol/validation-cases.json'), 'Summary was derived from a different validation inventory')
|
||||
require(summary.get('execution_gate_pass') is True and summary.get('numerical_integrity_pass') is True and
|
||||
summary.get('exact_repeat_pass') is True, 'Validation execution/numerical/repeat gate failed')
|
||||
require(set(summary.get('modes', {})) == set(cfg['validation']['modes']), 'Incomplete mode summaries')
|
||||
if representation:
|
||||
require(summary.get('representation_gate_pass') is True and summary.get('pass_all') is True and
|
||||
all(summary['modes'][mode].get('gate') is True for mode in cfg['validation']['modes']),
|
||||
'Seed representation summary does not authorize novel dynamics')
|
||||
|
||||
|
||||
def gate_pass(exp, stage, *, representation=False, cfg=None):
|
||||
report = read(Path(exp) / 'audit' / (stage + '-gate.json'))
|
||||
require(report.get('stage') == stage and report.get('pass_all') is True, 'Required execution gate failed: ' + stage)
|
||||
if representation:
|
||||
require(stage == 'seed-validation', 'Only the complete seed block unlocks novel trajectories')
|
||||
require(report.get('execution_gate_pass') is True and report.get('representation_gate_pass') is True,
|
||||
'Complete seed representation/execution gate required before novel dynamics')
|
||||
if cfg is None:
|
||||
from context import CFG
|
||||
cfg = CFG
|
||||
path = Path(exp) / 'validation/seed_validation/summary.json'
|
||||
require(report.get('cohort') == 'seed_validation' and
|
||||
report.get('summary') == 'validation/seed_validation/summary.json' and
|
||||
report.get('summary_sha256') == digest(path), 'Seed gate is not bound to this exact summary')
|
||||
require(report.get('validation_inventory_sha256') == digest(Path(exp) / 'protocol/validation-cases.json'),
|
||||
'Seed gate inventory digest differs')
|
||||
validate_summary_coverage(read(path), exp, cfg, 'seed_validation', representation=True)
|
||||
return report
|
||||
|
||||
|
||||
def authorize_case(exp, cfg, group, name, backend, repeat):
|
||||
ensure_unstopped(exp)
|
||||
require(group in GROUPS and backend in ('gpu', 'oracle', 'component') and type(repeat) is int and repeat in (0, 1),
|
||||
'Invalid runtime action')
|
||||
controls, records = inventories(exp, cfg)
|
||||
rows = controls if group == 'controls' else cohort_rows(records, cfg, group)
|
||||
selected = [row for row in rows if row['name'] == name]
|
||||
require(len(selected) == 1, 'Case not in the registered group inventory')
|
||||
row = selected[0]
|
||||
if group == 'controls':
|
||||
require(repeat == 0 and backend in ('gpu', 'component'), 'Controls are single fresh GPU identity runs')
|
||||
else:
|
||||
gate_pass(exp, 'controls')
|
||||
if group == 'novel_to_aj':
|
||||
gate_pass(exp, 'seed-validation', representation=True, cfg=cfg)
|
||||
probes = {item['name'] for item in probe_rows(records, cfg, group)}
|
||||
probe_stage = 'seed-probes' if group == 'seed_validation' else 'novel-probes'
|
||||
if name not in probes or repeat:
|
||||
gate_pass(exp, probe_stage)
|
||||
if backend in ('oracle', 'component'):
|
||||
require(name in probes and repeat == 0, 'Oracle/component scope is the two fixed probes per cohort')
|
||||
if repeat:
|
||||
require(backend == 'gpu' and row['seed'] == cfg['validation']['seeds'][0], 'Only declared first-seed repeats')
|
||||
return row
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,53 @@
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
import numpy as np
|
||||
from encoder_inputs import frozen_pn_drive, requests_from_forced, write_npz
|
||||
|
||||
|
||||
class EncoderInputTests(unittest.TestCase):
|
||||
def test_exact_frozen_encoder_ast_on_synthetic_static_mapping(self):
|
||||
import pandas as pd
|
||||
local = Path(__file__).resolve().parents[1] / 'models/male-cns-lif/phase4l.py'
|
||||
source = local if local.exists() else Path('/data/digital-drosophila/models/male-cns-lif/phase4l.py')
|
||||
pin = 'e97d58d8c1f44920d0e953f8f5831ddc86fa0198784e86460e315de4116ac553'
|
||||
function, ast_sha = frozen_pn_drive(source, pin)
|
||||
self.assertEqual(len(ast_sha), 64)
|
||||
mapping = pd.DataFrame(dict(glomerulus=['G2', 'G2', 'G1', 'G1'],
|
||||
mapping_confidence=['exact', 'alias-normalized', 'exact', 'unknown'], neuron_idx=[3, 1, 7, 999]))
|
||||
cfg = {'odor_encoder': {'max_total_rate_per_glomerulus_hz': 400}}
|
||||
target, rates, _ = function('synthetic', np.array([.5, .25], np.float32), mapping,
|
||||
['G2', 'G1'], 'matched_energy', 1.5, cfg)
|
||||
np.testing.assert_array_equal(target, np.array([3, 1, 7], np.int32))
|
||||
np.testing.assert_array_equal(rates, np.array([200, 200, 200], np.float32))
|
||||
with self.assertRaises(AssertionError):
|
||||
frozen_pn_drive(source, '0' * 64)
|
||||
|
||||
def test_rng_column_order_preserved_until_request_sort(self):
|
||||
targets = np.array([10, 1, 3], np.int32)
|
||||
forced = np.array([[True, True, False], [False, True, True]], np.bool_)
|
||||
actual = requests_from_forced(forced, targets)
|
||||
np.testing.assert_array_equal(actual, np.array([[0, 1], [0, 10], [1, 1], [1, 3]], np.int32))
|
||||
np.testing.assert_array_equal(targets, np.array([10, 1, 3], np.int32))
|
||||
|
||||
def test_duplicate_targets_rejected_not_merged(self):
|
||||
with self.assertRaises(AssertionError):
|
||||
requests_from_forced(np.ones((2, 2), np.bool_), np.array([1, 1], np.int32))
|
||||
|
||||
def test_empty_sparse_schedule_has_exact_shape_and_dtype(self):
|
||||
result = requests_from_forced(np.zeros((3, 2), np.bool_), np.array([2, 1], np.int32))
|
||||
self.assertEqual(result.shape, (0, 2))
|
||||
self.assertEqual(result.dtype, np.int32)
|
||||
|
||||
def test_npz_creation_is_exclusive(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
path = Path(directory) / 'schedule.npz'
|
||||
write_npz(path, tick=np.array([1], np.int32))
|
||||
with np.load(path, allow_pickle=False) as stored:
|
||||
np.testing.assert_array_equal(stored['tick'], np.array([1], np.int32))
|
||||
with self.assertRaises(FileExistsError):
|
||||
write_npz(path, tick=np.array([2], np.int32))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -0,0 +1,39 @@
|
||||
import unittest
|
||||
from finalize import decide_status
|
||||
|
||||
|
||||
class FinalStatusTests(unittest.TestCase):
|
||||
def decide(self, **overrides):
|
||||
args = dict(numerical=True, seed=dict(execution_gate_pass=True, representation_gate_pass=True),
|
||||
novel=dict(execution_gate_pass=True, representation_gate_pass=True), integrity_pass=True,
|
||||
reference_pass=True, reference_present=True)
|
||||
args.update(overrides)
|
||||
return decide_status(**args)
|
||||
|
||||
def test_complete_only_representation_pass_is_not_named_learning(self):
|
||||
self.assertEqual(self.decide(), 'PASS-FROZEN-D18-INDEPENDENT-VALIDATION')
|
||||
|
||||
def test_seed_failure_without_novel_is_completed_scientific_failure(self):
|
||||
self.assertEqual(self.decide(seed=dict(execution_gate_pass=True, representation_gate_pass=False), novel=None),
|
||||
'FAIL-SEED-VALIDATION-REPRESENTATION')
|
||||
|
||||
def test_novel_gate_failure_is_preserved(self):
|
||||
self.assertEqual(self.decide(novel=dict(execution_gate_pass=True, representation_gate_pass=False)),
|
||||
'FAIL-NOVEL-ODOR-REPRESENTATION')
|
||||
|
||||
def test_missing_execution_or_reference_is_not_scientific_pass(self):
|
||||
for changes in (dict(numerical=False), dict(novel=None), dict(reference_present=False), dict(seed=None)):
|
||||
with self.subTest(changes=changes):
|
||||
self.assertEqual(self.decide(**changes), 'INCOMPLETE-VALIDATION-EXECUTION')
|
||||
|
||||
def test_numerical_and_unlock_failure_are_not_representation_results(self):
|
||||
self.assertEqual(self.decide(known_numerical_failure=True), 'FAIL-NUMERICAL-OR-EXECUTION-POLICY')
|
||||
self.assertEqual(self.decide(unauthorized_novel=True), 'FAIL-NUMERICAL-OR-EXECUTION-POLICY')
|
||||
|
||||
def test_old_integrity_or_reference_fail_overrides_new_pass(self):
|
||||
self.assertEqual(self.decide(integrity_pass=False), 'FAIL-FROZEN-INTEGRITY-OR-REGRESSION')
|
||||
self.assertEqual(self.decide(reference_pass=False), 'FAIL-FROZEN-INTEGRITY-OR-REGRESSION')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -0,0 +1,409 @@
|
||||
"""CPU-only synthetic tests: no Torch/Brian import, GPU, remote, or brain run."""
|
||||
import copy
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import tempfile
|
||||
import types
|
||||
import unittest
|
||||
from unittest import mock
|
||||
|
||||
import numpy as np
|
||||
|
||||
import runtime_bridge as bridge
|
||||
|
||||
|
||||
HERE = Path(__file__).absolute().parent
|
||||
AJ = HERE.parent / 'apl-timing-lab-v1'
|
||||
|
||||
|
||||
def write_json(path, data):
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(data), encoding='utf-8')
|
||||
|
||||
|
||||
def spec(root, name='cal-physiological-00-s51101'):
|
||||
return dict(name=name, kind='calibration', steps=1000, dt_ms=.1, duration_ms=100.,
|
||||
model={'fixed': True}, graph_dir=str(root / 'graph'), apl_bundle_dir=str(root / 'bundle'),
|
||||
coefficients_path=str(root / 'coefficients.json'), science_python=str(root / 'venv/python'),
|
||||
legacy_apl_worker=str(root / 'legacy_apl_worker.py'),
|
||||
apl={'tau_ms': 20., 'gain': .01, 'normalization': 1.}, apl_delay_steps=18,
|
||||
record_history=False, training=False, precision='float64', apl_precision='float32',
|
||||
budget_root=str(root / 'old-exp'), output_budget_bytes=1024)
|
||||
|
||||
|
||||
def inventory(root):
|
||||
cfg = dict(control_cases=['control-phys', 'control-matched'], validation={
|
||||
'cohorts': list(bridge.COHORTS), 'seeds': [101, 102, 103, 104, 105],
|
||||
'modes': ['physiological', 'matched_energy'], 'odor_count': 6,
|
||||
'repeat_first_seed_each_odor': True})
|
||||
controls = [dict(name=name, source_case=str(root / 'old-cases' / name),
|
||||
source_output=str(root / 'old-gpu' / name)) for name in cfg['control_cases']]
|
||||
rows = [dict(cohort=cohort, mode=mode, odor_index=oi, seed=seed,
|
||||
name=f'{cohort}-{mode}-{oi:02d}-s{seed}') for cohort in bridge.COHORTS
|
||||
for mode in cfg['validation']['modes'] for oi in range(6) for seed in cfg['validation']['seeds']]
|
||||
write_json(root / 'protocol/control-cases.json', controls)
|
||||
write_json(root / 'protocol/validation-cases.json', rows)
|
||||
return cfg, rows
|
||||
|
||||
|
||||
def gate(root, stage, **values):
|
||||
write_json(root / 'audit' / (stage + '-gate.json'), dict(stage=stage, pass_all=True, **values))
|
||||
|
||||
|
||||
def seed_summary(root, cfg, rows):
|
||||
base = bridge.cohort_rows(rows, cfg, 'seed_validation')
|
||||
repeated = [row for row in base if row['seed'] == cfg['validation']['seeds'][0]]
|
||||
pairs = [(row['name'], 0) for row in base] + [(row['name'], 1) for row in repeated]
|
||||
return dict(phase='4AK', cohort='seed_validation', apl_delay_steps=18, duration_ms=100.,
|
||||
seeds=cfg['validation']['seeds'], pass_all=True, execution_gate_pass=True,
|
||||
representation_gate_pass=True, numerical_integrity_pass=True, exact_repeat_pass=True,
|
||||
expected_base_trials=60, valid_base_trials=60, expected_repeats=12, checked_repeats=12,
|
||||
checked_timing_trials=72, runs=[dict(case=row['name'], cohort='seed_validation',
|
||||
validation_cohort='seed_validation') for row in base],
|
||||
repeats=[dict(case=row['name'], pass_all=True) for row in repeated],
|
||||
timing_checks=[dict(case=name, repeat=repeat, pass_all=True, apl_delay_steps=18, steps=1000)
|
||||
for name, repeat in pairs], input_output_provenance=[dict(case=name, repeat=repeat) for name, repeat in pairs],
|
||||
modes={mode: dict(gate=True) for mode in cfg['validation']['modes']},
|
||||
prerequisite_input_sha256={'validation_inventory': bridge.digest(root / 'protocol/validation-cases.json')})
|
||||
|
||||
|
||||
def bind_seed_gate(root, summary):
|
||||
path = root / 'validation/seed_validation/summary.json'
|
||||
write_json(path, summary)
|
||||
gate(root, 'seed-validation', cohort='seed_validation', execution_gate_pass=True,
|
||||
representation_gate_pass=True, summary='validation/seed_validation/summary.json',
|
||||
summary_sha256=bridge.digest(path), validation_inventory_sha256=bridge.digest(root / 'protocol/validation-cases.json'))
|
||||
|
||||
|
||||
class LoaderTests(unittest.TestCase):
|
||||
def test_verified_import_preserves_file_and_globals_without_pyc(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
path = Path(temporary) / 'sealed.py'
|
||||
data = b'VALUE = 37\n'
|
||||
path.write_bytes(data)
|
||||
name = '_test_phase4ak_sealed'
|
||||
try:
|
||||
module = bridge.load_checked(name, path, hashlib.sha256(data).hexdigest())
|
||||
self.assertEqual(module.VALUE, 37)
|
||||
self.assertEqual(Path(module.__file__), path)
|
||||
self.assertIs(bridge.load_checked(name, path, hashlib.sha256(data).hexdigest()), module)
|
||||
self.assertNotIn('__phase4ak_loaded_sha256__', module.__dict__)
|
||||
self.assertFalse((path.parent / '__pycache__').exists())
|
||||
finally:
|
||||
sys.modules.pop(name, None); bridge._LOADED.pop(name, None)
|
||||
|
||||
def test_sha_checked_before_execution(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
path = Path(temporary) / 'sealed.py'
|
||||
path.write_text('raise RuntimeError("must not execute")', encoding='utf-8')
|
||||
with self.assertRaisesRegex(ValueError, 'SHA mismatch'):
|
||||
bridge.load_checked('_test_bad_pin', path, '0' * 64)
|
||||
self.assertNotIn('_test_bad_pin', sys.modules)
|
||||
|
||||
def test_existing_name_collision_is_not_silently_replaced(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
path = Path(temporary) / 'sealed.py'
|
||||
path.write_text('VALUE = 1', encoding='utf-8')
|
||||
old = types.ModuleType('_test_collision')
|
||||
sys.modules[old.__name__] = old
|
||||
try:
|
||||
with self.assertRaisesRegex(ValueError, 'unverified module'):
|
||||
bridge.load_checked(old.__name__, path, bridge.digest(path))
|
||||
self.assertIs(sys.modules[old.__name__], old)
|
||||
finally:
|
||||
sys.modules.pop(old.__name__, None)
|
||||
|
||||
def test_real_pure_sealed_module_load_does_not_import_torch_or_brian(self):
|
||||
self.assertNotIn('torch', sys.modules)
|
||||
self.assertNotIn('brian2', sys.modules)
|
||||
compare = bridge.sealed('compare_timing.py', AJ)
|
||||
self.assertEqual(bridge.digest(compare.__file__), bridge.AJ_PINS['compare_timing.py'])
|
||||
self.assertTrue(callable(compare.compare))
|
||||
self.assertNotIn('torch', sys.modules)
|
||||
self.assertNotIn('brian2', sys.modules)
|
||||
|
||||
|
||||
class InputTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temporary = tempfile.TemporaryDirectory()
|
||||
self.root = Path(self.temporary.name)
|
||||
self.old, self.new = self.root / 'old', self.root / 'new'
|
||||
self.old.mkdir(); self.new.mkdir()
|
||||
self.old_spec = spec(self.root)
|
||||
self.new_spec = dict(self.old_spec, budget_root=str(self.root / 'new-exp'), output_budget_bytes=2048)
|
||||
write_json(self.old / 'spec.json', self.old_spec)
|
||||
write_json(self.new / 'spec.json', self.new_spec)
|
||||
for name in ('requests.npz', 'pn-drive.npz'):
|
||||
for directory in (self.old, self.new):
|
||||
(directory / name).write_bytes(b'identical synthetic saved input bytes')
|
||||
|
||||
def tearDown(self):
|
||||
self.temporary.cleanup()
|
||||
|
||||
def test_control_allows_only_budget_difference(self):
|
||||
fresh, old, evidence = bridge.control_inputs(self.new, self.old)
|
||||
self.assertEqual(fresh['apl_delay_steps'], old['apl_delay_steps'])
|
||||
self.assertEqual(len(evidence), 6)
|
||||
changed = dict(self.new_spec, mode='changed')
|
||||
write_json(self.new / 'spec.json', changed)
|
||||
with self.assertRaisesRegex(ValueError, 'beyond the two budget fields'):
|
||||
bridge.control_inputs(self.new, self.old)
|
||||
|
||||
def test_drive_and_request_bytes_both_checked(self):
|
||||
for name in ('requests.npz', 'pn-drive.npz'):
|
||||
with self.subTest(name=name):
|
||||
original = (self.new / name).read_bytes()
|
||||
(self.new / name).write_bytes(original + b'changed')
|
||||
with self.assertRaisesRegex(ValueError, name):
|
||||
bridge.control_inputs(self.new, self.old)
|
||||
(self.new / name).write_bytes(original)
|
||||
|
||||
def test_real_d18_required_no_fake_d0(self):
|
||||
for value in (0, True, 18.0):
|
||||
with self.subTest(value=value):
|
||||
write_json(self.new / 'spec.json', dict(self.new_spec, apl_delay_steps=value))
|
||||
with self.assertRaisesRegex(ValueError, 'D18'):
|
||||
bridge.control_inputs(self.new, self.old)
|
||||
|
||||
def test_initial_state_presence_and_bytes_checked(self):
|
||||
(self.new / 'arrays.npz').write_bytes(b'a')
|
||||
with self.assertRaisesRegex(ValueError, 'presence'):
|
||||
bridge.control_inputs(self.new, self.old)
|
||||
(self.old / 'arrays.npz').write_bytes(b'b')
|
||||
with self.assertRaisesRegex(ValueError, 'arrays.npz'):
|
||||
bridge.control_inputs(self.new, self.old)
|
||||
|
||||
def test_exclusive_output_and_new_budget_containment(self):
|
||||
root = Path(self.new_spec['budget_root'])
|
||||
root.mkdir()
|
||||
out = root / 'gpu' / 'run'
|
||||
bridge._new_output(self.new, out)
|
||||
out.mkdir(parents=True)
|
||||
with self.assertRaisesRegex(ValueError, 'exclusive'):
|
||||
bridge._new_output(self.new, out)
|
||||
with self.assertRaisesRegex(ValueError, 'exclusive'):
|
||||
bridge._new_output(self.new, self.root / 'elsewhere')
|
||||
|
||||
def test_control_refuses_overwrite_before_reading_outputs(self):
|
||||
report = self.root / 'report.json'
|
||||
report.write_text('keep me', encoding='utf-8')
|
||||
with self.assertRaisesRegex(ValueError, 'existing control'):
|
||||
bridge.control_identity('missing-new', 'missing-old', self.new, self.old, report)
|
||||
self.assertEqual(report.read_text(encoding='utf-8'), 'keep me')
|
||||
|
||||
|
||||
class GateTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temporary = tempfile.TemporaryDirectory()
|
||||
self.root = Path(self.temporary.name)
|
||||
self.cfg, self.rows = inventory(self.root)
|
||||
|
||||
def tearDown(self):
|
||||
self.temporary.cleanup()
|
||||
|
||||
def test_exact_120_cells_and_fixed_two_probes(self):
|
||||
controls, rows = bridge.inventories(self.root, self.cfg)
|
||||
self.assertEqual(len(controls), 2)
|
||||
for cohort in bridge.COHORTS:
|
||||
probes = bridge.probe_rows(rows, self.cfg, cohort)
|
||||
self.assertEqual(len(probes), 2)
|
||||
self.assertTrue(all(row['odor_index'] == 0 and row['seed'] == 101 for row in probes))
|
||||
|
||||
def test_duplicate_or_replaced_cell_rejected(self):
|
||||
rows = copy.deepcopy(self.rows)
|
||||
rows[-1] = rows[0]
|
||||
write_json(self.root / 'protocol/validation-cases.json', rows)
|
||||
with self.assertRaises(ValueError):
|
||||
bridge.inventories(self.root, self.cfg)
|
||||
|
||||
def test_controls_do_not_run_brian_or_repeats(self):
|
||||
name = self.cfg['control_cases'][0]
|
||||
bridge.authorize_case(self.root, self.cfg, 'controls', name, 'gpu', 0)
|
||||
for backend, repeat in (('oracle', 0), ('gpu', 1)):
|
||||
with self.assertRaises(ValueError):
|
||||
bridge.authorize_case(self.root, self.cfg, 'controls', name, backend, repeat)
|
||||
|
||||
def test_probes_before_other_seed_runs(self):
|
||||
gate(self.root, 'controls')
|
||||
probe = bridge.probe_rows(self.rows, self.cfg, 'seed_validation')[0]
|
||||
bridge.authorize_case(self.root, self.cfg, 'seed_validation', probe['name'], 'oracle', 0)
|
||||
later = next(row for row in self.rows if row['cohort'] == 'seed_validation' and row['seed'] == 102)
|
||||
with self.assertRaises(FileNotFoundError):
|
||||
bridge.authorize_case(self.root, self.cfg, 'seed_validation', later['name'], 'gpu', 0)
|
||||
gate(self.root, 'seed-probes')
|
||||
bridge.authorize_case(self.root, self.cfg, 'seed_validation', later['name'], 'gpu', 0)
|
||||
with self.assertRaises(ValueError):
|
||||
bridge.authorize_case(self.root, self.cfg, 'seed_validation', later['name'], 'oracle', 0)
|
||||
with self.assertRaises(ValueError):
|
||||
bridge.authorize_case(self.root, self.cfg, 'seed_validation', later['name'], 'gpu', 1)
|
||||
|
||||
def test_novel_denied_on_representation_failure_even_execution_passes(self):
|
||||
gate(self.root, 'controls')
|
||||
gate(self.root, 'seed-validation', execution_gate_pass=True, representation_gate_pass=False)
|
||||
probe = bridge.probe_rows(self.rows, self.cfg, 'novel_to_aj')[0]
|
||||
with self.assertRaisesRegex(ValueError, 'representation'):
|
||||
bridge.authorize_case(self.root, self.cfg, 'novel_to_aj', probe['name'], 'gpu', 0)
|
||||
|
||||
def test_novel_requires_both_gate_and_underlying_summary(self):
|
||||
gate(self.root, 'controls')
|
||||
probe = bridge.probe_rows(self.rows, self.cfg, 'novel_to_aj')[0]
|
||||
summary = seed_summary(self.root, self.cfg, self.rows)
|
||||
summary['representation_gate_pass'] = False
|
||||
bind_seed_gate(self.root, summary)
|
||||
with self.assertRaisesRegex(ValueError, 'summary'):
|
||||
bridge.authorize_case(self.root, self.cfg, 'novel_to_aj', probe['name'], 'gpu', 0)
|
||||
summary['representation_gate_pass'] = True
|
||||
bind_seed_gate(self.root, summary)
|
||||
bridge.authorize_case(self.root, self.cfg, 'novel_to_aj', probe['name'], 'gpu', 0)
|
||||
|
||||
def test_novel_rejects_summary_hash_or_cohort_misbinding(self):
|
||||
gate(self.root, 'controls')
|
||||
summary = seed_summary(self.root, self.cfg, self.rows)
|
||||
bind_seed_gate(self.root, summary)
|
||||
probe = bridge.probe_rows(self.rows, self.cfg, 'novel_to_aj')[0]
|
||||
summary['cohort'] = 'novel_to_aj'
|
||||
write_json(self.root / 'validation/seed_validation/summary.json', summary)
|
||||
with self.assertRaisesRegex(ValueError, 'exact summary'):
|
||||
bridge.authorize_case(self.root, self.cfg, 'novel_to_aj', probe['name'], 'gpu', 0)
|
||||
bind_seed_gate(self.root, summary)
|
||||
with self.assertRaisesRegex(ValueError, 'identity'):
|
||||
bridge.authorize_case(self.root, self.cfg, 'novel_to_aj', probe['name'], 'gpu', 0)
|
||||
|
||||
def test_novel_rejects_replaced_case_repeat_or_timing_even_with_correct_counts(self):
|
||||
gate(self.root, 'controls')
|
||||
probe = bridge.probe_rows(self.rows, self.cfg, 'novel_to_aj')[0]
|
||||
for key in ('runs', 'repeats', 'timing_checks', 'input_output_provenance'):
|
||||
with self.subTest(key=key):
|
||||
summary = seed_summary(self.root, self.cfg, self.rows)
|
||||
summary[key][0]['case'] = 'not-a-registered-case'
|
||||
bind_seed_gate(self.root, summary)
|
||||
with self.assertRaises(ValueError):
|
||||
bridge.authorize_case(self.root, self.cfg, 'novel_to_aj', probe['name'], 'gpu', 0)
|
||||
|
||||
def test_any_stop_marker_prevents_all_scientific_actions(self):
|
||||
for kind in ('representation_failure', 'execution_or_numerical_failure'):
|
||||
with self.subTest(kind=kind):
|
||||
write_json(self.root / 'audit/stopped-seed-validation.json', {'stop_kind': kind})
|
||||
for backend in ('gpu', 'oracle', 'component'):
|
||||
with self.assertRaisesRegex(ValueError, 'stop marker'):
|
||||
bridge.authorize_case(self.root, self.cfg, 'controls', self.cfg['control_cases'][0], backend, 0)
|
||||
|
||||
def test_standalone_action_failure_persists_stop_and_never_retries(self):
|
||||
action = mock.Mock(side_effect=RuntimeError('synthetic failure; no science executed'))
|
||||
with self.assertRaisesRegex(RuntimeError, 'synthetic failure'):
|
||||
bridge.guarded_action(self.root, 'gpu', 'controls', 'control-phys', 0, action)
|
||||
marker = self.root / 'audit/stopped-case-gpu-controls-control-phys-r0.json'
|
||||
report = bridge.read(marker)
|
||||
self.assertEqual(report['stop_kind'], 'execution_or_numerical_failure')
|
||||
self.assertFalse(report['numerical_instability_inferred_from_execution_error'])
|
||||
before = marker.read_bytes()
|
||||
with self.assertRaisesRegex(ValueError, 'stop marker'):
|
||||
bridge.guarded_action(self.root, 'gpu', 'controls', 'control-phys', 0, action)
|
||||
self.assertEqual(action.call_count, 1)
|
||||
self.assertEqual(marker.read_bytes(), before)
|
||||
|
||||
def test_successful_guarded_action_adds_no_stop(self):
|
||||
expected = {'pass_all': True}
|
||||
self.assertIs(bridge.guarded_action(self.root, 'gpu', 'controls', 'control-phys', 0, lambda: expected), expected)
|
||||
bridge.ensure_unstopped(self.root)
|
||||
|
||||
|
||||
class InvocationTests(unittest.TestCase):
|
||||
def test_original_gpu_function_receives_new_paths_unchanged(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
root = Path(temporary)
|
||||
case, output, build = root / 'case', root / 'new-exp/gpu/run', root / 'new-exp/build'
|
||||
data = spec(root)
|
||||
data['budget_root'] = str(root / 'new-exp')
|
||||
write_json(case / 'spec.json', data)
|
||||
runtime = types.SimpleNamespace(run_gpu=mock.Mock(return_value={'pass_all': True}))
|
||||
with mock.patch.object(bridge, 'sealed', return_value=runtime), mock.patch.object(bridge, '_receipt'):
|
||||
result = bridge.run_gpu(case, output, build)
|
||||
self.assertTrue(result['pass_all'])
|
||||
runtime.run_gpu.assert_called_once_with(case, output, build)
|
||||
|
||||
def test_original_oracle_function_is_not_given_candidate_trace(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
root = Path(temporary)
|
||||
case, output = root / 'case', root / 'new-exp/oracle/run'
|
||||
data = spec(root); data['budget_root'] = str(root / 'new-exp')
|
||||
write_json(case / 'spec.json', data)
|
||||
runtime = types.SimpleNamespace(run_case=mock.Mock(return_value={'pass_all': True}))
|
||||
with mock.patch.object(bridge, 'sealed', return_value=runtime), mock.patch.object(bridge, '_receipt'):
|
||||
bridge.run_oracle(case, output)
|
||||
runtime.run_case.assert_called_once_with(case, output)
|
||||
|
||||
|
||||
class ControlEvidenceTests(unittest.TestCase):
|
||||
def test_real_sealed_comparator_accepts_complete_synthetic_d18_and_rejects_signed_zero_tail(self):
|
||||
# Artifact construction only: these zero traces were not produced by any model.
|
||||
aj = bridge.sealed('compare_timing.py', AJ)
|
||||
timing = bridge.sealed('timing_queue.py', AJ)
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
root = Path(temporary)
|
||||
cases, outputs = [root / name for name in ('old-case', 'new-case')], [root / name for name in ('old-out', 'new-out')]
|
||||
base = spec(root)
|
||||
(root / 'graph').mkdir()
|
||||
np.save(root / 'graph/weights.npy', np.zeros(1, np.float64), allow_pickle=False)
|
||||
write_json(root / 'coefficients.json', {'synthetic_only': True})
|
||||
states = dict(v=np.zeros(166700, np.float64), g=np.zeros(166700, np.float64),
|
||||
ref=np.zeros(166700, np.int32), counts=np.zeros(166700, np.int32),
|
||||
step=np.empty(0, np.int32), neuron_idx=np.empty(0, np.int32))
|
||||
generated = np.zeros((1000, 4064), np.float32)
|
||||
pending = np.zeros((18, 4064), np.float32)
|
||||
audit = timing.audit_sequence(generated, generated, pending, 18)
|
||||
audit.update(generated_rows=1000, applied_generated_rows=982, pending_shape=[18, 4064],
|
||||
pending_application_tick_start=1000, pending_application_tick_stop_exclusive=1018,
|
||||
pending_generated_tick_start=982, pending_generated_tick_stop_exclusive=1000,
|
||||
pending_raw_sha256=aj.raw_hash(pending), final_apl_state_float32=[0., 0.],
|
||||
final_apl_state_u32_bits=[0, 0], final_apl_state_nonzero=False,
|
||||
finite_generated_sequence_only=True, future_apl_decay_or_feedback_generated=False,
|
||||
closed_loop_branch_integrals_claimed_equal=False, biological_delay_measurement_claimed=False)
|
||||
hashes = [{**{name + '_sha256': aj.raw_hash(states[name]) for name in ('v', 'g', 'ref')}, 'tick': tick}
|
||||
for tick in range(1000)]
|
||||
queues = [aj._queue_digest(tick, np.empty(0, np.int64), np.empty(0, np.uint8)) for tick in range(1000)]
|
||||
for index, (case, output) in enumerate(zip(cases, outputs)):
|
||||
data = dict(base, budget_root=str(root / ('budget-' + str(index))), output_budget_bytes=10000 + index)
|
||||
write_json(case / 'spec.json', data)
|
||||
for name in ('requests.npz', 'pn-drive.npz'):
|
||||
(case / name).write_bytes(b'synthetic-identical-input')
|
||||
output.mkdir()
|
||||
np.savez_compressed(output / 'state.npz', **states)
|
||||
for name, value in (('drive', np.zeros((1000, 2), np.float32)), ('state', np.zeros((1000, 2), np.float32)),
|
||||
('inhibition', generated), ('applied', generated), ('pending-tail', pending)):
|
||||
np.save(output / ('apl-' + name + '.float32.npy'), value, allow_pickle=False)
|
||||
for name, rows in (('tick-state-hashes.jsonl', hashes), ('queue.jsonl', queues)):
|
||||
(output / name).write_text(''.join(json.dumps(row) + '\n' for row in rows), encoding='utf-8')
|
||||
(output / 'requests.csv').write_text('tick,neuron_idx,attempted,accepted,rejected\n', encoding='utf-8')
|
||||
component = {'synthetic_fixture': True}
|
||||
write_json(output / 'apl-component.json', component)
|
||||
write_json(output / 'apl-timing-audit.json', audit)
|
||||
inputs = {name: bridge.digest(case / name) for name in ('spec.json', 'requests.npz')}
|
||||
inputs.update({'base_weights.npy': bridge.digest(root / 'graph/weights.npy'),
|
||||
'coefficients.json': bridge.digest(root / 'coefficients.json')})
|
||||
summary = dict(case=base['name'], complete=True, pass_all=True, steps=1000, completed_ticks=1000,
|
||||
nodes=166700, edges=25582938, precision='float64', apl_precision='float32', state_units='SI volts',
|
||||
total_spikes=0, requested=0, accepted_requests=0, rejected_requests=0, arriving_edges=0,
|
||||
accepted_arriving_edges=0, counts_match_sparse_events=True, all_finite=True, record_history=False,
|
||||
apl_fast_edges_zeroed=5845, kc_targets_present=4064, effective_weights_raw_sha256='0' * 64,
|
||||
apl_schedule='same synthetic schedule', apl_si_conversion='same synthetic conversion',
|
||||
sealed_ae_backend_sha256='1' * 64, learning=False, apl_delay_steps=18, apl_timing=audit,
|
||||
source_provenance={'same_sealed_source': True}, pending_generated_count=18,
|
||||
pending_initial_zero_prefix=0, pending_tail_shape=[18, 4064], future_generated_decay_tail_included=False,
|
||||
apl_component=component, input_sha256=inputs, build={'path': str(root / ('build-' + str(index)))})
|
||||
write_json(output / 'summary.json', summary)
|
||||
result = bridge.control_identity(outputs[1], outputs[0], cases[1], cases[0], aj_model=AJ)
|
||||
self.assertTrue(result['pass_all'], result.get('error', result['failed_checks']))
|
||||
pending[0, 0] = np.float32(-0.)
|
||||
np.save(outputs[1] / 'apl-pending-tail.float32.npy', pending, allow_pickle=False)
|
||||
failed = bridge.control_identity(outputs[1], outputs[0], cases[1], cases[0], aj_model=AJ)
|
||||
self.assertFalse(failed['pass_all'])
|
||||
self.assertIn('apl.pending.bitwise_exact', failed['failed_checks'])
|
||||
self.assertIn('fresh_gpu.fixed_delay_pending.bitwise_exact', failed['failed_checks'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -0,0 +1,298 @@
|
||||
"""Synthetic-only validation tests: no historical outputs or brain execution."""
|
||||
import copy
|
||||
import json
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import types
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
import validation as val
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
LEGACY = val.load_frozen(HERE.parent / "author-apl-lab-v2/calibration.py", val.AG_SHA256, "ak_test_ag")
|
||||
TIMING = val.load_frozen(HERE.parent / "apl-timing-lab-v1/calibration.py", val.AJ_SHA256, "ak_test_aj")
|
||||
|
||||
|
||||
def config():
|
||||
cfg = val.read(HERE / "config.json")
|
||||
cfg["validation"]["seeds"] = [1001, 1002, 1003, 1004, 1005] # Synthetic, not proposed scientific seeds.
|
||||
return cfg
|
||||
|
||||
|
||||
def panel():
|
||||
return [dict(odor_id=oid, odor_name=oid, split="calibration" if i < 6 else "heldout",
|
||||
mapped_response_sum=val.MEDIAN)
|
||||
for i, oid in enumerate(sorted(LEGACY.ORIGINAL_CAL_IDS) + ["synthetic-new-" + str(i) for i in range(6)])]
|
||||
|
||||
|
||||
def records(cfg):
|
||||
rows = []
|
||||
pp = panel()
|
||||
for ci, cohort in enumerate(val.COHORTS):
|
||||
for mode in val.MODES:
|
||||
for oi in range(6):
|
||||
for seed in cfg["validation"]["seeds"]:
|
||||
rows.append(dict(name=f"test-{mode}-{oi}-{seed}", cohort=cohort,
|
||||
source_panel_split=val.SPLITS[cohort], mode=mode, odor_id=pp[6 * ci + oi]["odor_id"],
|
||||
odor_index=oi, seed=seed, targets=1, requests=0))
|
||||
return rows
|
||||
|
||||
|
||||
def encoder(cfg, inventory):
|
||||
return dict(panel_count=12, cal_count=6, novel_to_aj_count=6, median_mapped_response_sum=val.MEDIAN,
|
||||
seeds=cfg["validation"]["seeds"], seeds_static_audit_sha256="a" * 64,
|
||||
cohorts_static_panel_rows={cohort: [dict(odor_id=oid, odor_index=i, source_panel_split=val.SPLITS[cohort])
|
||||
for i, oid in enumerate(inventory[cohort]["odor_ids"])] for cohort in val.COHORTS},
|
||||
requested_before_any_dynamics=True, historical_never_simulated_claim=False)
|
||||
|
||||
|
||||
def save(path, value):
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with path.open("x", encoding="utf-8") as stream:
|
||||
json.dump(value, stream, allow_nan=False)
|
||||
|
||||
|
||||
def make_raw_trial(root):
|
||||
"""Tiny neuron state with the real full-width APL trace schema."""
|
||||
output, case, bundle = root / "out", root / "case", root / "bundle"
|
||||
output.mkdir(); case.mkdir(); bundle.mkdir()
|
||||
steps, neurons = 20, 2
|
||||
cfg = dict(neurons=neurons, edges=0)
|
||||
for filename in ("input_matrix.npy", "output_matrix.npy", "kc_globals.npy", "apl_globals.npy"):
|
||||
np.save(bundle / filename, np.zeros(1, np.int32))
|
||||
spec = dict(name="synthetic", steps=steps, validation_cohort="novel_to_aj", source_panel_split="heldout",
|
||||
apl_delay_steps=18, mode="physiological", odor_id="synthetic", odor_name="synthetic", seed=1001,
|
||||
expected_external_events=1., rate_sum_hz=50., apl_bundle_dir=str(bundle))
|
||||
save(case / "spec.json", spec)
|
||||
np.savez(case / "requests.npz", tick=np.empty(0, np.int32), neuron_idx=np.empty(0, np.int32))
|
||||
state = {"step": np.empty(0, np.int32), "neuron_idx": np.empty(0, np.int32),
|
||||
"counts": np.zeros(neurons, np.int32), "v": np.zeros(neurons, np.float64),
|
||||
"g": np.zeros(neurons, np.float64), "ref": np.zeros(neurons, np.int32)}
|
||||
np.savez(output / "state.npz", **state)
|
||||
hashes = [dict(tick=t, **{key + "_sha256": LEGACY.raw_hash(state[key]) for key in ("v", "g", "ref")}) for t in range(steps)]
|
||||
queues = [dict(tick=t, count=0, accepted_count=0, ordered_edge_sha256="0" * 64,
|
||||
accepted_mask_sha256="0" * 64, joint_sha256="0" * 64) for t in range(steps)]
|
||||
for filename, rows in (("tick-state-hashes.jsonl", hashes), ("queue.jsonl", queues)):
|
||||
(output / filename).write_text("".join(json.dumps(row) + "\n" for row in rows), encoding="utf-8")
|
||||
(output / "requests.csv").write_text("tick,neuron_idx,attempted,accepted,rejected\n", encoding="utf-8")
|
||||
for filename, width in LEGACY.APL_FILES.items():
|
||||
np.save(output / filename, np.zeros((steps, width), np.float32))
|
||||
np.save(output / "apl-applied.float32.npy", np.zeros((steps, 4064), np.float32))
|
||||
pending = np.zeros((18, 4064), np.float32)
|
||||
np.save(output / "apl-pending-tail.float32.npy", pending)
|
||||
audit = dict(pass_all=True, apl_delay_steps=18, steps=steps, pending_generated_count=18,
|
||||
pending_initial_zero_prefix=0, pending_application_tick_start=steps,
|
||||
pending_application_tick_stop_exclusive=steps + 18, pending_generated_tick_start=steps - 18,
|
||||
pending_generated_tick_stop_exclusive=steps, initial_padding_positive_zero=True,
|
||||
sample_bits_preserved=True, brain_tail_advanced=False, generated_trace="apl-inhibition.float32.npy",
|
||||
applied_trace="apl-applied.float32.npy", pending_trace="apl-pending-tail.float32.npy",
|
||||
pending_raw_sha256=LEGACY.raw_hash(pending), pending_queue_drained=False,
|
||||
finite_generated_sequence_only=True, future_apl_decay_or_feedback_generated=False,
|
||||
closed_loop_branch_integrals_claimed_equal=False, biological_delay_measurement_claimed=False,
|
||||
final_apl_state_float32=[0., 0.], final_apl_state_u32_bits=[0, 0], final_apl_state_nonzero=False)
|
||||
save(output / "apl-timing-audit.json", audit)
|
||||
component = dict(ok=True, n_neurons=neurons, n_kc=2, n_apl=2, dt_ms=.1,
|
||||
files_sha256={name: val.digest(bundle / name) for name in
|
||||
("input_matrix.npy", "output_matrix.npy", "kc_globals.npy", "apl_globals.npy")})
|
||||
save(output / "apl-component.json", component)
|
||||
save(output / "summary.json", dict(complete=True, pass_all=True, all_finite=True, steps=steps,
|
||||
completed_ticks=steps, nodes=neurons, edges=0, precision="float64", apl_precision="float32",
|
||||
case=spec["name"], record_history=False, learning=False, requested=0, accepted_requests=0,
|
||||
rejected_requests=0, total_spikes=0, apl_component=component, apl_delay_steps=18, apl_timing=audit,
|
||||
input_sha256={name: val.digest(case / name) for name in ("spec.json", "requests.npz")}))
|
||||
return output, case, spec, cfg
|
||||
|
||||
|
||||
def fake_trial(row, cohort, silent=False):
|
||||
counts = np.zeros(1000, np.int32)
|
||||
if not silent:
|
||||
counts[row["odor_index"] * 50:row["odor_index"] * 50 + 50] = 1
|
||||
return dict(kc_counts=counts, expected_external_events=1., external_events=1,
|
||||
record=dict(case=row["name"], mode=row["mode"], seed=row["seed"], odor_id=row["odor_id"],
|
||||
requested=row["requests"], split=cohort, cohort=cohort, source_panel_split=val.SPLITS[cohort]),
|
||||
timing=dict(report=dict(pass_all=True)), apl={})
|
||||
|
||||
|
||||
class ValidationTests(unittest.TestCase):
|
||||
def test_source_pins_and_globals_are_unchanged(self):
|
||||
self.assertEqual(LEGACY.ORIGINAL_SEEDS, (51101, 51102, 51103, 51104, 51105))
|
||||
self.assertFalse(hasattr(LEGACY, "EXP"))
|
||||
self.assertFalse(hasattr(TIMING, "EXP"))
|
||||
with self.assertRaisesRegex(ValueError, "SHA"):
|
||||
val.load_frozen(HERE.parent / "author-apl-lab-v2/calibration.py", "0" * 64, "bad_pin")
|
||||
|
||||
def test_inventory_accepts_configured_seeds_and_keeps_static_halves(self):
|
||||
cfg = config(); inv = val.validate_inventory(records(cfg), cfg, panel(), LEGACY)
|
||||
self.assertEqual([len(inv[c]["records"]) for c in val.COHORTS], [60, 60])
|
||||
self.assertEqual(inv["novel_to_aj"]["odor_ids"], ["synthetic-new-" + str(i) for i in range(6)])
|
||||
val.validate_encoder(encoder(cfg, inv), inv, cfg)
|
||||
self.assertEqual(LEGACY.ORIGINAL_SEEDS[0], 51101)
|
||||
|
||||
def test_inventory_rejects_missing_duplicate_mislabelled_and_unaudited_cells(self):
|
||||
cfg = config()
|
||||
variants = []
|
||||
rr = records(cfg); variants.append(rr[:-1])
|
||||
rr = records(cfg); rr[-1] = copy.deepcopy(rr[-2]); variants.append(rr)
|
||||
for field, value in (("source_panel_split", "heldout"), ("odor_id", "wrong"),
|
||||
("seed", 51101), ("name", "../outside"), ("odor_index", True), ("targets", 0)):
|
||||
rr = records(cfg); rr[0][field] = value; variants.append(rr)
|
||||
for rr in variants:
|
||||
with self.subTest(first=rr[0]), self.assertRaises(ValueError):
|
||||
val.validate_inventory(rr, cfg, panel(), LEGACY)
|
||||
|
||||
def test_no_gate_relaxation_or_subset_median(self):
|
||||
for key, value in (("minimum_separable_odor_pairs", 1), ("sparse_min_fraction", .01),
|
||||
("repeat_exact_required", False)):
|
||||
cfg = config(); cfg["gates"][key] = value
|
||||
with self.subTest(key=key), self.assertRaises(ValueError):
|
||||
val.validate_inventory(records(cfg), cfg, panel(), LEGACY)
|
||||
pp = panel()
|
||||
for row in pp:
|
||||
row["mapped_response_sum"] = 1.
|
||||
with self.assertRaisesRegex(ValueError, "median"):
|
||||
val.validate_inventory(records(config()), config(), pp, LEGACY)
|
||||
|
||||
def test_encoder_rejects_history_claim_or_changed_panel_rows(self):
|
||||
cfg = config(); inv = val.validate_inventory(records(cfg), cfg, panel(), LEGACY)
|
||||
for key, value in (("historical_never_simulated_claim", True), ("requested_before_any_dynamics", False),
|
||||
("seeds_static_audit_sha256", ""), ("cohorts_static_panel_rows", {})):
|
||||
ee = encoder(cfg, inv); ee[key] = value
|
||||
with self.subTest(key=key), self.assertRaises(ValueError):
|
||||
val.validate_encoder(ee, inv, cfg)
|
||||
|
||||
def test_encoder_actual_prepare_panel_row_extension(self):
|
||||
cfg = config(); inv = val.validate_inventory(records(cfg), cfg, panel(), LEGACY)
|
||||
ee = encoder(cfg, inv)
|
||||
for ci, cohort in enumerate(val.COHORTS):
|
||||
for oi, row in enumerate(ee["cohorts_static_panel_rows"][cohort]):
|
||||
row["source_panel_row"] = ci * 6 + oi
|
||||
val.validate_encoder(ee, inv, cfg)
|
||||
ee["cohorts_static_panel_rows"]["novel_to_aj"][0]["source_panel_row"] = 0
|
||||
with self.assertRaisesRegex(ValueError, "panel row"):
|
||||
val.validate_encoder(ee, inv, cfg)
|
||||
|
||||
def test_fullgraph_case_matches_static_physics_and_exact_input_expectation(self):
|
||||
cfg = config(); row = records(cfg)[60]
|
||||
row.update(source_panel_row=6, expected_external_events=10., rates_sum=100.)
|
||||
spec = dict(name=row["name"], kind="fullgraph", validation_cohort=row["cohort"],
|
||||
source_panel_split=row["source_panel_split"], source_panel_row=6,
|
||||
**{key: row[key] for key in ("mode", "odor_id", "odor_index", "seed")},
|
||||
apl_delay_steps=18, steps=1000, dt_ms=.1, duration_ms=100., training=False, record_history=False,
|
||||
model={"resting_potential_mv": -52.}, graph_dir="unchanged-graph", apl_bundle_dir="unchanged-bundle",
|
||||
apl={"gain": .01}, coefficients_path="unchanged-coefficients", science_python="unchanged-entry",
|
||||
legacy_apl_worker="unchanged-worker", precision="float64", apl_precision="float32",
|
||||
target_count=1, rate_sum_hz=100., expected_external_events=10.)
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
case = Path(tmp)
|
||||
save(case / "spec.json", spec)
|
||||
np.savez(case / "pn-drive.npz", targets=np.array([10], np.int32), rates_hz=np.array([100.], np.float32))
|
||||
np.savez(case / "requests.npz", tick=np.empty(0, np.int32), neuron_idx=np.empty(0, np.int32))
|
||||
self.assertEqual(val.validate_case(case, row, cfg, spec), spec)
|
||||
wrong = copy.deepcopy(spec); wrong["model"]["resting_potential_mv"] = -51.
|
||||
with self.assertRaisesRegex(ValueError, "physical field"):
|
||||
val.validate_case(case, row, cfg, wrong)
|
||||
wrong_row = dict(row, targets=2)
|
||||
with self.assertRaisesRegex(ValueError, "PN drive"):
|
||||
val.validate_case(case, wrong_row, cfg, spec)
|
||||
|
||||
def test_real_legacy_reader_record_is_relabelled_without_globals_or_numerical_changes(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
output, case, spec, cfg = make_raw_trial(Path(tmp))
|
||||
trial = val.read_validation_trial(LEGACY, TIMING, output, case, spec, np.arange(2, dtype=np.int32), cfg)
|
||||
try:
|
||||
self.assertEqual(trial["record"]["split"], "novel_to_aj")
|
||||
self.assertEqual(trial["record"]["source_panel_split"], "heldout")
|
||||
self.assertFalse(trial["record"]["historical_never_simulated_claim"])
|
||||
self.assertTrue(trial["timing"]["report"]["pass_all"])
|
||||
self.assertEqual(LEGACY.ORIGINAL_SEEDS[0], 51101)
|
||||
finally:
|
||||
val.close_trial(trial)
|
||||
|
||||
def test_real_reader_still_rejects_missing_state_and_delay_zero(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
output, case, spec, cfg = make_raw_trial(Path(tmp))
|
||||
bad = dict(spec, apl_delay_steps=0)
|
||||
with self.assertRaisesRegex(ValueError, "delay"):
|
||||
val.read_validation_trial(LEGACY, TIMING, output, case, bad, np.arange(2, dtype=np.int32), cfg)
|
||||
(output / "state.npz").unlink()
|
||||
with self.assertRaises(FileNotFoundError):
|
||||
val.read_validation_trial(LEGACY, TIMING, output, case, spec, np.arange(2, dtype=np.int32), cfg)
|
||||
|
||||
def collect(self, silent=False, broken_repeat=False, missing=False):
|
||||
cfg = config(); inv = val.validate_inventory(records(cfg), cfg, panel(), LEGACY)
|
||||
cohort = "seed_validation"; rows = inv[cohort]["records"]
|
||||
byname = {row["name"]: row for row in rows}
|
||||
accessed = []
|
||||
def load(legacy, timing, output, case, spec, kc, config_arg):
|
||||
accessed.append(str(output))
|
||||
if missing and spec["name"] == rows[0]["name"]:
|
||||
raise FileNotFoundError("synthetic missing evidence")
|
||||
return fake_trial(byname[spec["name"]], cohort, silent)
|
||||
timing = types.SimpleNamespace(timing_repeat=lambda *args: dict(pass_all=not broken_repeat,
|
||||
checks=dict(apl_applied=not broken_repeat)))
|
||||
with patch.object(val, "validate_case", side_effect=lambda case, row, *args: row), \
|
||||
patch.object(val, "read_validation_trial", side_effect=load), \
|
||||
patch.object(val, "evidence_hashes", return_value={"synthetic": True}):
|
||||
result = val.collect_cohort(cohort, rows, inv[cohort]["odor_ids"], cfg, Path("synthetic"),
|
||||
np.arange(2, dtype=np.int32), {}, LEGACY, timing)
|
||||
return result, accessed
|
||||
|
||||
def test_full_60_plus_12_and_pair_specific_gate_without_other_cohort_read(self):
|
||||
result, accessed = self.collect()
|
||||
self.assertTrue(result["pass_all"])
|
||||
self.assertEqual((result["valid_base_trials"], result["checked_repeats"], result["checked_timing_trials"]), (60, 12, 72))
|
||||
self.assertEqual(len(accessed), 72)
|
||||
self.assertFalse(any("novel_to_aj" in path for path in accessed))
|
||||
for report in result["modes"].values():
|
||||
self.assertEqual((report["pair_count"], report["within_distance_count"], report["between_distance_count"]), (15, 60, 75))
|
||||
json.dumps(result, allow_nan=False)
|
||||
|
||||
def test_representation_failure_keeps_all_runs_and_is_not_execution_failure(self):
|
||||
result, accessed = self.collect(silent=True)
|
||||
self.assertTrue(result["execution_gate_pass"])
|
||||
self.assertFalse(result["representation_gate_pass"])
|
||||
self.assertFalse(result["pass_all"])
|
||||
self.assertEqual(len(accessed), 72)
|
||||
self.assertTrue(all(row["stage"] == "representation_gate" for row in result["failures"]))
|
||||
|
||||
def test_missing_evidence_and_repeat_failure_never_pass_execution(self):
|
||||
for options in (dict(missing=True), dict(broken_repeat=True)):
|
||||
with self.subTest(options=options):
|
||||
result, _ = self.collect(**options)
|
||||
self.assertFalse(result["execution_gate_pass"])
|
||||
self.assertFalse(result["pass_all"])
|
||||
self.assertTrue(result["failures"])
|
||||
|
||||
def test_none_recompute_is_deterministic_and_writes_nothing(self):
|
||||
cfg = config(); rr = records(cfg); inv = val.validate_inventory(rr, cfg, panel(), LEGACY)
|
||||
ee = encoder(cfg, inv)
|
||||
ctx = types.SimpleNamespace(ROOT=Path("synthetic-root"), MODEL=Path("synthetic-model"), EXP=Path("synthetic-exp"),
|
||||
AG_MODEL=Path("ag-model"), AJ_MODEL=Path("aj-model"), AG=Path("ag"), AJ=Path("aj"), CFG=cfg,
|
||||
check=lambda: {}, save=lambda *args: self.fail("Read-only summarize wrote output"))
|
||||
collected, _ = self.collect()
|
||||
def read_fixture(path):
|
||||
if path.name == "validation-cases.json": return rr
|
||||
if path.name == "encoder-audit.json": return ee
|
||||
if path.name == "spec.json": return dict(apl_delay_steps=18, training=False)
|
||||
raise AssertionError("Unexpected input read: " + str(path))
|
||||
def hash_fixture(path):
|
||||
return val.PANEL_SHA256 if path.name == "frozen-odor-panel.parquet" else "a" * 64
|
||||
with patch.dict("sys.modules", context=ctx), patch.object(val, "read", side_effect=read_fixture), \
|
||||
patch.object(val, "digest", side_effect=hash_fixture), \
|
||||
patch.object(val, "load_frozen", side_effect=[LEGACY, TIMING, LEGACY, TIMING]), \
|
||||
patch.object(pd, "read_parquet", return_value=pd.DataFrame(panel())), \
|
||||
patch.object(np, "load", return_value=np.arange(4064, dtype=np.int32)), \
|
||||
patch.object(val, "collect_cohort", side_effect=lambda *args: copy.deepcopy(collected)):
|
||||
first = val.summarize("seed_validation", None)
|
||||
second = val.summarize("seed_validation", None)
|
||||
self.assertEqual(first, second)
|
||||
self.assertTrue(first["pass_all"])
|
||||
json.dumps(first, allow_nan=False)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,344 @@
|
||||
"""Frozen-D18 validation cohorts; no simulation, fitting, or old heldout read.
|
||||
|
||||
Only the requested cohort's new trajectories are opened. The suite owns the
|
||||
seed-validation -> novel-to-AJ unlock. Legacy modules are source-hash pinned,
|
||||
loaded without writing bytecode, and never have their globals rebound.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
import sys
|
||||
import types
|
||||
|
||||
import numpy as np
|
||||
|
||||
AG_SHA256 = "edb9cb2d890471fd07afd88c959ef3540b0cd40af490fac3f26396b5e7f5bf9c"
|
||||
AJ_SHA256 = "fb15ddeaa108305c8de697bbae36b23ab0414a7c58db51c66dce9feb4a048a0d"
|
||||
PANEL_SHA256 = "907f16a56e024a8e2112b5751e53b0e6037b329fa110098af3d31b119c10a1a0"
|
||||
COHORTS = ("seed_validation", "novel_to_aj")
|
||||
SPLITS = dict(zip(COHORTS, ("calibration", "heldout")))
|
||||
MODES = ("physiological", "matched_energy")
|
||||
MEDIAN = 2.556107997894287
|
||||
_DEFAULT_OUTPUT = object()
|
||||
|
||||
|
||||
def require(ok, message):
|
||||
if not ok:
|
||||
raise ValueError(message)
|
||||
|
||||
|
||||
def read(path):
|
||||
return json.loads(Path(path).read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def digest(path):
|
||||
h = hashlib.sha256()
|
||||
with Path(path).open("rb") as stream:
|
||||
for block in iter(lambda: stream.read(8 << 20), b""):
|
||||
h.update(block)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def load_frozen(path, expected, name):
|
||||
path = Path(path)
|
||||
source = path.read_bytes()
|
||||
require(hashlib.sha256(source).hexdigest() == expected, "Frozen source SHA mismatch: " + str(path))
|
||||
module = types.ModuleType(name)
|
||||
module.__file__ = str(path)
|
||||
sys.modules[name] = module
|
||||
exec(compile(source, str(path), "exec"), module.__dict__)
|
||||
return module
|
||||
|
||||
|
||||
def validate_design(cfg):
|
||||
design = cfg["validation"]
|
||||
seeds = design["seeds"]
|
||||
require(isinstance(seeds, list) and len(seeds) == 5 and len(set(seeds)) == 5 and
|
||||
all(type(seed) is int and 0 <= seed < 2**63 for seed in seeds), "Need five explicitly audited seed integers")
|
||||
require(design["cohorts"] == list(COHORTS) and design["modes"] == list(MODES) and
|
||||
design["odor_count"] == 6 and design["duration_ms"] == 100.0 and
|
||||
design["repeat_first_seed_each_odor"] is True, "Validation design changed")
|
||||
require(type(cfg["apl_delay_steps"]) is int and cfg["apl_delay_steps"] == 18 and
|
||||
cfg["neurons"] == 166700 and cfg["edges"] == 25582938 and cfg["training"] is False,
|
||||
"Frozen D18/fullgraph/nonlearning scope changed")
|
||||
gates = cfg["gates"]
|
||||
require(gates["sparse_min_fraction"] == .03 and gates["sparse_max_fraction"] == .09 and
|
||||
gates["max_fraction_strict"] == .5 and gates["min_fraction_strict"] == 0 and
|
||||
gates["minimum_separable_odor_pairs"] == 3 and all(gates[key] is True for key in
|
||||
("same_jaccard_gt_between", "between_euclidean_gt_within", "both_modes_required",
|
||||
"finite_required", "repeat_exact_required")), "Original complete representation/integrity gates changed")
|
||||
return seeds
|
||||
|
||||
|
||||
def validate_inventory(records, cfg, panel_rows, legacy):
|
||||
"""Static 120-row inventory, preserving original panel row order in each half."""
|
||||
seeds = validate_design(cfg)
|
||||
require(isinstance(records, list) and len(records) == 120, "Inventory must contain exactly 120 base cases")
|
||||
require(len(panel_rows) == 12 and [row["split"] for row in panel_rows] == ["calibration"] * 6 + ["heldout"] * 6,
|
||||
"Frozen static panel order/splits changed")
|
||||
all_ids = [row["odor_id"] for row in panel_rows]
|
||||
require(all(isinstance(oid, str) and oid for oid in all_ids) and len(set(all_ids)) == 12 and
|
||||
set(all_ids[:6]) == legacy.ORIGINAL_CAL_IDS, "Frozen panel identities invalid")
|
||||
require(float(np.median([row["mapped_response_sum"] for row in panel_rows])) == MEDIAN,
|
||||
"Matched-energy reference must remain the original full-12 median")
|
||||
result = {}
|
||||
names, keys = set(), set()
|
||||
for row in records:
|
||||
require(row["cohort"] in COHORTS and row["mode"] in MODES, "Unknown cohort/mode")
|
||||
require(type(row["odor_index"]) is int and 0 <= row["odor_index"] < 6 and
|
||||
type(row["seed"]) is int and row["seed"] in seeds, "Invalid odor index/seed")
|
||||
require(isinstance(row["name"], str) and re.fullmatch(r"[A-Za-z0-9_-]+", row["name"]), "Unsafe case name")
|
||||
namekey = (row["cohort"], row["name"])
|
||||
key = (row["cohort"], row["mode"], row["odor_index"], row["seed"])
|
||||
require(namekey not in names and key not in keys, "Duplicate case name or experimental cell")
|
||||
names.add(namekey); keys.add(key)
|
||||
offset = 0 if row["cohort"] == "seed_validation" else 6
|
||||
require(row["source_panel_split"] == SPLITS[row["cohort"]] and
|
||||
row["odor_id"] == all_ids[offset + row["odor_index"]], "Cohort identity/order/source split mismatch")
|
||||
if "source_panel_row" in row:
|
||||
require(type(row["source_panel_row"]) is int and row["source_panel_row"] == offset + row["odor_index"],
|
||||
"Inventory original panel row mismatch")
|
||||
require(type(row["targets"]) is int and 0 < row["targets"] <= cfg["neurons"] and
|
||||
type(row["requests"]) is int and row["requests"] >= 0, "Invalid inventory target/request counts")
|
||||
expected = {(cohort, mode, oi, seed) for cohort in COHORTS for mode in MODES for oi in range(6) for seed in seeds}
|
||||
require(keys == expected, "Inventory missing or replacing a predeclared cell")
|
||||
for index, cohort in enumerate(COHORTS):
|
||||
rows = [row for row in records if row["cohort"] == cohort]
|
||||
rows.sort(key=lambda row: (MODES.index(row["mode"]), row["odor_index"], seeds.index(row["seed"])))
|
||||
result[cohort] = {"records": rows, "odor_ids": all_ids[index * 6:(index + 1) * 6]}
|
||||
return result
|
||||
|
||||
|
||||
def validate_encoder(encoder, inventory, cfg):
|
||||
require(encoder["panel_count"] == 12 and encoder["cal_count"] == 6 and encoder["novel_to_aj_count"] == 6 and
|
||||
encoder["median_mapped_response_sum"] == MEDIAN and encoder["seeds"] == cfg["validation"]["seeds"] and
|
||||
encoder["requested_before_any_dynamics"] is True and encoder["historical_never_simulated_claim"] is False,
|
||||
"Frozen encoder/reference or validation scope changed")
|
||||
require(isinstance(encoder["seeds_static_audit_sha256"], str) and
|
||||
re.fullmatch(r"[0-9a-f]{64}", encoder["seeds_static_audit_sha256"]), "Missing seed audit SHA")
|
||||
expected = {cohort: [dict(odor_id=oid, odor_index=index, source_panel_split=SPLITS[cohort])
|
||||
for index, oid in enumerate(inventory[cohort]["odor_ids"])] for cohort in COHORTS}
|
||||
actual = encoder["cohorts_static_panel_rows"]
|
||||
require(set(actual) == set(COHORTS), "Encoder cohort set changed")
|
||||
for ci, cohort in enumerate(COHORTS):
|
||||
require(len(actual[cohort]) == 6, "Encoder cohort panel count changed")
|
||||
for index, (row, wanted) in enumerate(zip(actual[cohort], expected[cohort])):
|
||||
require(all(row.get(key) == value for key, value in wanted.items()),
|
||||
"Encoder cohort panel rows differ from frozen identities")
|
||||
if "source_panel_row" in row:
|
||||
require(type(row["source_panel_row"]) is int and row["source_panel_row"] == ci * 6 + index,
|
||||
"Encoder original panel row mismatch")
|
||||
|
||||
|
||||
def validate_case(case, row, cfg, frozen_spec):
|
||||
spec = read(case / "spec.json")
|
||||
require(spec["name"] == row["name"] and spec["kind"] == "fullgraph" and
|
||||
spec["validation_cohort"] == row["cohort"] and spec["source_panel_split"] == row["source_panel_split"] and
|
||||
all(spec[key] == row[key] for key in ("mode", "odor_id", "odor_index", "seed")), "Spec differs from validation inventory")
|
||||
require(type(spec["apl_delay_steps"]) is int and spec["apl_delay_steps"] == 18 and spec["steps"] == 1000 and
|
||||
spec["dt_ms"] == .1 and spec["duration_ms"] == 100.0 and spec["training"] is False and
|
||||
spec["record_history"] is False, "Fixed D18/window/nonlearning schema changed")
|
||||
if "source_panel_row" in row or "source_panel_row" in spec:
|
||||
expected_row = (0 if row["cohort"] == "seed_validation" else 6) + row["odor_index"]
|
||||
require(type(spec.get("source_panel_row")) is int and spec["source_panel_row"] == expected_row and
|
||||
row.get("source_panel_row", expected_row) == expected_row, "Spec original panel row mismatch")
|
||||
for key in ("model", "graph_dir", "apl_bundle_dir", "apl", "coefficients_path", "science_python",
|
||||
"legacy_apl_worker", "precision", "apl_precision", "dt_ms", "duration_ms", "steps"):
|
||||
require(spec[key] == frozen_spec[key], "Frozen physical field changed: " + key)
|
||||
with np.load(case / "pn-drive.npz", allow_pickle=False) as source:
|
||||
require(set(source.files) == {"targets", "rates_hz"}, "PN drive schema changed")
|
||||
targets, rates = source["targets"], source["rates_hz"]
|
||||
require(targets.dtype == np.int32 and targets.ndim == 1 and len(targets) == row["targets"] and
|
||||
len(np.unique(targets)) == len(targets) and np.all((targets >= 0) & (targets < cfg["neurons"])) and
|
||||
rates.dtype == np.float32 and rates.shape == targets.shape and np.isfinite(rates).all() and np.all(rates > 0),
|
||||
"Invalid PN drive identities/rates; no sorting or zero-channel insertion is allowed")
|
||||
expected_events = float(rates.sum()) * spec["steps"] * .1 / 1000.
|
||||
require(spec["target_count"] == row["targets"] and spec["rate_sum_hz"] == float(rates.sum()) and
|
||||
spec["expected_external_events"] == expected_events and np.isfinite(expected_events) and expected_events > 0,
|
||||
"PN rates/spec event expectation disagree")
|
||||
if "expected_external_events" in row:
|
||||
require(row["expected_external_events"] == expected_events, "Inventory expected events disagree")
|
||||
if "rates_sum" in row:
|
||||
require(row["rates_sum"] == float(rates.sum()), "Inventory rate sum disagrees")
|
||||
with np.load(case / "requests.npz", allow_pickle=False) as source:
|
||||
require(set(source.files) == {"tick", "neuron_idx"} and len(source["tick"]) == row["requests"] and
|
||||
np.isin(source["neuron_idx"], targets).all(), "Request count/targets differ from prepared input")
|
||||
return spec
|
||||
|
||||
|
||||
def read_validation_trial(legacy, timing, output, case, spec, kc_globals, cfg):
|
||||
"""Reuse all sealed checks; replace only the returned descriptive record."""
|
||||
cohort = spec["validation_cohort"]
|
||||
require(cohort in COHORTS and spec["source_panel_split"] == SPLITS[cohort] and
|
||||
type(spec["apl_delay_steps"]) is int and spec["apl_delay_steps"] == 18, "Wrong validation cohort or delay")
|
||||
trial = legacy.read_trial(output, case, spec, kc_globals, cfg["neurons"], cfg["edges"])
|
||||
try:
|
||||
trial["timing"] = timing.validate_timing(output, spec, trial)
|
||||
except Exception:
|
||||
close_trial(trial)
|
||||
raise
|
||||
trial["record"] = dict(trial["record"], split=cohort, cohort=cohort, validation_cohort=cohort,
|
||||
source_panel_split=spec["source_panel_split"], historical_never_simulated_claim=False,
|
||||
used_for_parameter_selection=False)
|
||||
if "source_panel_row" in spec:
|
||||
trial["record"]["source_panel_row"] = spec["source_panel_row"]
|
||||
return trial
|
||||
|
||||
|
||||
def close_trial(trial):
|
||||
if trial is None:
|
||||
return
|
||||
arrays = list(trial.get("apl", {}).values())
|
||||
arrays += [trial.get("timing", {}).get(key) for key in ("applied", "pending")]
|
||||
for value in arrays:
|
||||
mmap = getattr(value, "_mmap", None)
|
||||
if mmap is not None and not mmap.closed:
|
||||
mmap.close()
|
||||
|
||||
|
||||
def evidence_hashes(output, case, name, repeat):
|
||||
files = ("state.npz", "summary.json", "tick-state-hashes.jsonl", "queue.jsonl", "requests.csv", "apl-component.json",
|
||||
"apl-drive.float32.npy", "apl-state.float32.npy", "apl-inhibition.float32.npy",
|
||||
"apl-applied.float32.npy", "apl-pending-tail.float32.npy", "apl-timing-audit.json")
|
||||
return {"case": name, "repeat": repeat,
|
||||
"input_sha256": {key: digest(case / key) for key in ("spec.json", "requests.npz", "pn-drive.npz")},
|
||||
"output_sha256": {key: digest(output / key) for key in files}}
|
||||
|
||||
|
||||
def collect_cohort(cohort, records, odors, cfg, exp, kc_globals, frozen_spec, legacy, timing):
|
||||
"""Aggregate one block even when its representation gate fails."""
|
||||
seeds = cfg["validation"]["seeds"]
|
||||
failures, runs, rows, repeats, timings, proof, modes = [], {}, [], [], [], [], {}
|
||||
for row in records:
|
||||
name, first, spec = row["name"], None, None
|
||||
case = exp / "cases" / cohort / name
|
||||
try:
|
||||
spec = validate_case(case, row, cfg, frozen_spec)
|
||||
except Exception as error:
|
||||
failures.append(dict(stage="case_input", case=name, type=type(error).__name__, message=str(error)))
|
||||
for repeat_index in ([0, 1] if row["seed"] == seeds[0] else [0]):
|
||||
trial = None
|
||||
try:
|
||||
require(spec is not None, "Case input validation unavailable")
|
||||
output = exp / "gpu" / cohort / (name + "-r" + str(repeat_index))
|
||||
trial = read_validation_trial(legacy, timing, output, case, spec, kc_globals, cfg)
|
||||
require(trial["record"]["requested"] == row["requests"], "Inventory/actual request count differs")
|
||||
entry = evidence_hashes(output, case, name, repeat_index)
|
||||
proof.append(entry)
|
||||
timings.append(dict(case=name, repeat=repeat_index, **trial["timing"]["report"]))
|
||||
if repeat_index == 0:
|
||||
first = trial
|
||||
runs[(row["mode"], row["odor_id"], row["seed"])] = {key: trial[key] for key in
|
||||
("kc_counts", "expected_external_events", "external_events")}
|
||||
rows.append(trial["record"])
|
||||
else:
|
||||
require(first is not None, "Baseline unavailable for exact repeat")
|
||||
report = dict(case=name, **timing.timing_repeat(first, trial, legacy))
|
||||
repeats.append(report)
|
||||
require(report["pass_all"] is True, "Exact repeat differs: " + ", ".join(
|
||||
key for key, value in report["checks"].items() if not value))
|
||||
except Exception as error:
|
||||
failures.append(dict(stage="raw_trial" if repeat_index == 0 else "repeat_trial", case=name,
|
||||
repeat=repeat_index, type=type(error).__name__, message=str(error)))
|
||||
finally:
|
||||
if trial is not first:
|
||||
close_trial(trial)
|
||||
close_trial(first)
|
||||
for mode in MODES:
|
||||
expected = {(mode, odor, seed) for odor in odors for seed in seeds}
|
||||
if expected.issubset(runs):
|
||||
try:
|
||||
report = legacy.summarize_mode(mode, odors, seeds, runs, cfg["gates"])
|
||||
require(report["conditions"] == legacy.evaluate_gates(report, cfg["gates"]), "Original gate conjunction mismatch")
|
||||
require(report["within_distance_count"] == 60 and report["between_distance_count"] == 75 and
|
||||
report["pair_count"] == 15 and all(pair["within_distance_count"] == 20 and
|
||||
pair["between_distance_count"] == 5 for pair in report["pairs"]), "Pair-specific denominator changed")
|
||||
modes[mode] = report
|
||||
if not report["gate"]:
|
||||
failures.append(dict(stage="representation_gate", mode=mode,
|
||||
failed_conditions=[key for key, value in report["conditions"].items() if not value]))
|
||||
except Exception as error:
|
||||
failures.append(dict(stage="representation_calculation", mode=mode, type=type(error).__name__, message=str(error)))
|
||||
numerical = bool(len(runs) == 60 and len(timings) == 72 and not any(row["stage"] in
|
||||
("case_input", "raw_trial", "repeat_trial") for row in failures))
|
||||
repeated = bool(len(repeats) == 12 and all(row["pass_all"] is True for row in repeats))
|
||||
execution = bool(numerical and repeated and not any(row["stage"] == "representation_calculation" for row in failures))
|
||||
represented = bool(len(modes) == 2 and all(row["gate"] is True for row in modes.values()))
|
||||
return dict(cohort=cohort, source_panel_split=SPLITS[cohort], pass_all=bool(execution and represented),
|
||||
execution_gate_pass=execution, numerical_integrity_pass=numerical, exact_repeat_pass=repeated,
|
||||
representation_gate_pass=represented, original_representation_gate_pass=represented,
|
||||
expected_base_trials=60, valid_base_trials=len(runs), expected_repeats=12, checked_repeats=len(repeats),
|
||||
checked_timing_trials=len(timings), modes=modes, runs=rows, repeats=repeats, timing_checks=timings,
|
||||
failures=failures, input_output_provenance=proof)
|
||||
|
||||
|
||||
def summarize(cohort, out_json=_DEFAULT_OUTPUT):
|
||||
"""Default exclusive write; None recomputes deterministically without writes."""
|
||||
require(cohort in COHORTS, "Unknown validation cohort")
|
||||
import context as ctx
|
||||
target = ctx.EXP / "validation" / cohort / "summary.json" if out_json is _DEFAULT_OUTPUT else (
|
||||
None if out_json is None else Path(out_json))
|
||||
require(target is None or not target.exists(), "Refusing existing cohort summary")
|
||||
result = dict(cohort=cohort, source_panel_split=SPLITS[cohort], pass_all=False, execution_gate_pass=False,
|
||||
numerical_integrity_pass=False, exact_repeat_pass=False, representation_gate_pass=False,
|
||||
original_representation_gate_pass=False, expected_base_trials=60, valid_base_trials=0,
|
||||
expected_repeats=12, checked_repeats=0, checked_timing_trials=0,
|
||||
modes={}, runs=[], repeats=[], timing_checks=[], failures=[], input_output_provenance=[])
|
||||
proof, odors, encoder = {}, [], None
|
||||
try:
|
||||
ctx.check()
|
||||
legacy = load_frozen(ctx.AG_MODEL / "calibration.py", AG_SHA256, "phase4ak_frozen_ag_calibration")
|
||||
timing = load_frozen(ctx.AJ_MODEL / "calibration.py", AJ_SHA256, "phase4ak_frozen_aj_calibration")
|
||||
inventory_path = ctx.EXP / "protocol/validation-cases.json"
|
||||
encoder_path = ctx.EXP / "protocol/encoder-audit.json"
|
||||
panel_path = ctx.ROOT / "experiments/phase4l-olfactory-grounding/odor-encoder/frozen-odor-panel.parquet"
|
||||
require(digest(panel_path) == PANEL_SHA256, "Frozen static panel SHA changed")
|
||||
import pandas as pd
|
||||
panel = pd.read_parquet(panel_path).to_dict(orient="records")
|
||||
inventory = validate_inventory(read(inventory_path), ctx.CFG, panel, legacy)
|
||||
encoder = read(encoder_path)
|
||||
validate_encoder(encoder, inventory, ctx.CFG)
|
||||
seed_audit = ctx.MODEL / "seed-audit.json"
|
||||
require(digest(seed_audit) == encoder["seeds_static_audit_sha256"], "Seed-audit receipt/encoder mismatch")
|
||||
kc_path = ctx.AG / "apl-bundle/kc_globals.npy"
|
||||
kc = np.load(kc_path, allow_pickle=False)
|
||||
require(kc.dtype == np.int32 and kc.shape == (4064,) and np.all(np.diff(kc) > 0) and
|
||||
np.all((kc >= 0) & (kc < ctx.CFG["neurons"])), "Frozen KC identity invalid")
|
||||
base_path = ctx.AJ / "cases/delay18/cal-physiological-00-s51101/spec.json"
|
||||
base = read(base_path)
|
||||
require(base["apl_delay_steps"] == 18 and base["training"] is False, "Wrong frozen physical reference")
|
||||
proof = {"validation_inventory": digest(inventory_path), "encoder_audit": digest(encoder_path),
|
||||
"static_panel": digest(panel_path), "seed_audit": digest(seed_audit),
|
||||
"KC_identity": digest(kc_path), "frozen_D18_spec": digest(base_path),
|
||||
"AG_calibration_source": AG_SHA256, "AJ_calibration_source": AJ_SHA256}
|
||||
odors = inventory[cohort]["odor_ids"]
|
||||
result = collect_cohort(cohort, inventory[cohort]["records"], odors, ctx.CFG, ctx.EXP, kc, base, legacy, timing)
|
||||
except Exception as error:
|
||||
result["failures"].append(dict(stage="prerequisites", type=type(error).__name__, message=str(error)))
|
||||
result.update(phase="4AK", apl_delay_steps=18, seeds=ctx.CFG["validation"]["seeds"], duration_ms=100.0,
|
||||
odor_ids=odors, encoder_audit=encoder, prerequisite_input_sha256=proof,
|
||||
representation_failure_is_not_numerical_failure=True, old_module_globals_modified=False,
|
||||
old_heldout_dynamic_access=False, historical_never_simulated_claim=False,
|
||||
requested_cohort_only_dynamic_read=True, suite_owns_sequential_unlock=True,
|
||||
shared_apl_component_oracle_limit=True, independent_oracle_equivalence_claim=False,
|
||||
seed_namespace_claim="Declared static-audit scope only; not an unqualified all-history independence claim",
|
||||
biological_delay_measurement_claimed=False, candidate_generated=False, training=False,
|
||||
metric_source="SHA-pinned AG complete phase4l gate; SHA-pinned AJ timing/repeat checks")
|
||||
# Serializability/finiteness must be established even for read-only finalization.
|
||||
json.dumps(result, allow_nan=False, sort_keys=True)
|
||||
if target is not None:
|
||||
outputs = {target: result}
|
||||
outputs.update({target.parent / (mode + ".json"): report for mode, report in result["modes"].items()})
|
||||
require(all(not path.exists() for path in outputs), "Refusing existing cohort/mode output")
|
||||
for path, value in outputs.items():
|
||||
ctx.save(path, value)
|
||||
return result
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
require(len(sys.argv) == 2, "Usage: validation.py seed_validation|novel_to_aj")
|
||||
answer = summarize(sys.argv[1])
|
||||
print(json.dumps({key: answer[key] for key in ("cohort", "pass_all", "execution_gate_pass", "representation_gate_pass")}))
|
||||
@@ -0,0 +1,49 @@
|
||||
"""Unchanged deterministic-v2 reference rerun, exclusively into this new stage."""
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import yaml
|
||||
from context import ROOT, EXP, CFG, check, save, bit_equal
|
||||
|
||||
|
||||
def main():
|
||||
check()
|
||||
assert not (EXP / 'regression/frozen-five.json').exists()
|
||||
from run_case import preflight
|
||||
preflight()
|
||||
runtime_path = ROOT / 'models/male-cns-lif/deterministic-runtime-v2'
|
||||
pc = yaml.safe_load((runtime_path / 'phase4p.yaml').read_text())
|
||||
sys.path.insert(0, str(ROOT / 'models/male-cns-lif'))
|
||||
sys.path.insert(0, str(runtime_path))
|
||||
import torch
|
||||
import phase4p
|
||||
assert torch.cuda.mem_get_info()[0] > 2 * 1024**3
|
||||
torch.cuda.reset_peak_memory_stats()
|
||||
started = time.perf_counter()
|
||||
runtime = phase4p.CanonicalRuntime(pc, runtime_path, EXP)
|
||||
rows = []
|
||||
for spec in pc['reference_trials']:
|
||||
preflight()
|
||||
reference_dir = ROOT / 'experiments/phase4p-deterministic-runtime/phase4c-v2/reference' / spec['name']
|
||||
events = pd.read_parquet(reference_dir / 'spike-events.parquet')
|
||||
reference = dict(step=events.step.to_numpy(), neuron_idx=events.neuron_idx.to_numpy(),
|
||||
counts=np.load(reference_dir / 'per-neuron-spike-counts.int32.npy'),
|
||||
v=np.load(reference_dir / 'final-voltage.float32.npy'), g=np.load(reference_dir / 'final-g.float32.npy'))
|
||||
result = runtime.run(np.flatnonzero(runtime.pops[spec['population']]), spec['rate_hz'], spec['seed'], spec['duration_ms'])
|
||||
row = dict(trial=spec['name'], exact={key: bool(bit_equal(value, result[key])) for key, value in reference.items()},
|
||||
dtype_shape={key: dict(reference_dtype=str(value.dtype), actual_dtype=str(result[key].dtype),
|
||||
reference_shape=list(value.shape), actual_shape=list(result[key].shape)) for key, value in reference.items()},
|
||||
wall_seconds=result['metrics']['wall_seconds'])
|
||||
rows.append(row)
|
||||
print(row, flush=True)
|
||||
passed = len(rows) == 5 and all(all(row['exact'].values()) for row in rows)
|
||||
save(EXP / 'regression/frozen-five.json', dict(pass_all=passed, trials=rows,
|
||||
wall_seconds=time.perf_counter() - started, peak_allocated_vram=int(torch.cuda.max_memory_allocated()),
|
||||
scope='Old frozen deterministic-v2 five references only; not the new D18 hypothesis'))
|
||||
assert passed
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
Reference in New Issue
Block a user