Seal AO interface failure and verified isolated postprocessing diagnosis

This commit is contained in:
Digital Drosophila Archive
2026-09-30 17:57:02 +08:00
parent 4ff241bdc9
commit 8e9f8f4d56
126 changed files with 92124 additions and 3 deletions
@@ -0,0 +1,70 @@
# AO — bounded two-epoch conditioning-interface validation
This independent branch does not replace the frozen reference, runtime or any
Phase 1–AN result. It tests the installation interface for the existing frozen
eligibility-v2 rule. It is not a new dynamics hypothesis or a learned model.
## Fixed, pre-outcome selection
- Master seed: `20260930`; no seed search or outcome-based retries.
- First canonical block: physiological / PAM15 / reinforce_A / rep01.
- First normal paired epoch requesting a US, and its immediate successor.
- Both selected epochs are odor A (4-ethylphenol), not an A/B discrimination
test. Exact recorded L glomerular/PN mapping and PN column order are retained.
- Each epoch is 120 ms: 100 ms CS plus 20 ms tail, US at tick 900.
- D1, D0 and frozen arms receive identical saved CS and DAN requests. The
scheduled artificial D scalar is not measured dopamine concentration and is
not multiplied by actual DAN request acceptance. Delivery is reported apart.
- Two executors (reference and light), two fresh epochs per arm: 12 calls,
14,400 timesteps. This does not authorize the full conditioning inventory.
## Immutable neural semantics and authority
Each epoch resets V, g, refractory counters, APL state and both delay queues;
the zero-initialized eligibility trace is computed from actual sealed spikes.
Only the float32-mV plastic delta carries between epochs. No update occurs
inside an epoch. Original graph, base weights, transmitter signs, numerical
precision, APL parameters, frozen v2 rule and complete 14,232-edge mask remain
unchanged. Full graph is 166,700 neurons and 25,582,938 edges.
Registration pins source snapshots, actual graph/APL/coefficient files, frozen
L static resources, H mappings, the complete generated semantic plan and the
12 selected call descriptors before CUDA sampling. Sampling receipts bind the
two generated request arrays before any brain call. Every input file is shared
byte-for-byte between arms and executors. The full 63-bit input seed is separate
from the inherited conservative 31-bit neural reset-control value.
A fixed first-seed CUDA check executes only the four original L sampling AST
statements, compares sparse requests, and compares full63-bit versus low32-bit
uniform matrices. This is a sampler test, not a neural trial or seed search.
Two production and three diagnostic matrices are drawn; each is 1200×PN count,
at most 2,000,000 uniform values. No continuous-time Poisson claim is made.
Each neural output is immediately sealed. Outside that seal, the coordinator
binds original inputs, checkpoint events, actual v2 recomputation and the new
delta snapshot. `parent_snapshot_sha256` means the previous **overlay NPZ file**,
not its manifest or the neural checkpoint. Epoch 2 must install that exact
snapshot; D0/frozen deltas must stay positive-zero bitwise.
## Finite gates and reporting limits
Required: all 12 processes complete; six reference/light pairs have identical
sparse events, counts, final V/g/ref and terminal queues/APL; deltas are exact;
D0/frozen identity holds; epoch 2 actually installs epoch 1 delta. D1 must have
nonzero actual update coverage or the result is `INCOMPLETE`, never rescued by
another input. Nonzero delta is interface coverage, not evidence of memory.
The light executor does not record intermediate floating states; no all-tick
bitwise claim is inferred. This validation has only one training replicate, two
A epochs, no independent behavioral evaluation, and no acquisition/reversal/
retention success claim. The post-update retention boundary is still unsupported.
## Resource and failure policy
Use only RTX4090 UUID GPU-79f3a85d-adce-7833-7003-6488140ca596; admit a child only
when all three ComfyUI queues are empty. Never stop or evict services. Cap new
output at 4 GiB plus 10 GiB free reserve and each child at 1800 seconds. Preserve
partial output and stop on any failure; no automatic brain retry. End by
rechecking AN and AM seals and the inherited 28-root integrity inventory. Five
old frozen reference results are retained from sealed AM evidence, not claimed
to have been rerun by this small interface gate.
@@ -0,0 +1,116 @@
# Phase 4AN GPU 工程验证门草案
状态:**未注册 DRAFT;不是执行授权,尚未运行本门。** 日期:2026-09-30。
本门只验证新的轻记录执行器和连续状态接口是否保存冻结 D18 动力学。它不是 conditioning、retention 学习结论,也不产生 learned candidate。AM 是否已正式完成、其最终状态和 seal SHA,必须在本门预注册时从真实证据核验,本文不代填成功状态或哈希。
## 1. 固定范围与最小库存
只使用一个 AM 已准备的 100 ms 全脑输入,分别安装其原有 `zero` 与 `static25` 固定 overlay。选择规则先于本门任何新 GPU 输出:从 AM `protocol/preregistered.json` 的原 `planned` 顺序取第一条 `repeat == 0` 的记录,保存其完整行、序号、case 路径及 SHA;不得按活动强弱、旧 static25 效果或运行快慢挑 case。其原始 case 实际位于 AK `cases/<cohort>/<name>`;AM `run_suite.py` 没有另造一套 case。若预注册者希望改变此规则,应先修订草案并锁定,不得在看新输出后换输入。
每个 overlay 固定以下库存,所有行均使用同一原始 1000 tick 请求序列或其精确前缀/分段:
| 行组 | 固定执行内容 | 新执行调用数/overlay |
|---|---|---:|
| Fresh 对照 | 完整 1000 tick reference r0、light r0、light r1 | 3 |
| 连续链 | reference、light 各一条,段长 `[5, 13, 1, 481, 500]` | 10 |
| 切点前缀 | 从初态独立运行 reference 至绝对 tick 数 18、19、500 | 3 |
合计 32 次短执行,11,074 个全脑 tick,即累计 1.1074 生物秒;不是 68,000 个训练 epoch。两条链的第一段均为 fresh 5 tick,其中 reference 第一段兼作第一个切点的前缀参照,不重复运行。切点为累计 tick 数 5、18、19、500,终点为 1000:覆盖短于 18 tick 的父/子段、在绝对 tick 18 的首次合法延迟到达、19-slot 环的相位,以及较晚的活动边界。不得把部分库存完成称作完整门通过。
固定参数:166,700 neurons、25,582,938 edges、`dt_ms=0.1`、18 tick synaptic delay、19 行 spike ring、18 行 graded-APL ring、LIF float64 SI 状态与旧 APL float32 操作。保留 14,232 mask、5845 fast-APL 出边置零规则及原 APL 全形矩阵。`static25` 是 AM 原 float32-mV 候选基值的固定 `-0.25 * base` overlay,经原 SI bridge 安装;不是声称 float64 原权重每项恰减 25%。两条 overlay 分别与自己的 AM 参照比较,不能要求 zero 与 static25 相等。
禁止:weight updates、eligibility 推进/应用、eta 改动、DAN/US 新请求、odor/seed 新抽样、动力学调参、训练、额外 tail ticks。这里的连续链只检验工程续接,不检验记忆保留。
## 2. 当前真实接口与输入派生
以下是底层 Python API;本门的独立 coordinator 使用它们,不改旧模块 globals:
```python
runtime_light.run_light(case_dir, out_dir, build_root, overlay_file, overlay_manifest)
runtime_light.run_reference(case_dir, out_dir, build_root, overlay_file, overlay_manifest)
runtime_continuation.seal_checkpoint(checkpoint_dir)
runtime_continuation.load_checkpoint(checkpoint_dir, expected_sha)
runtime_continuation.run_continued(
case_dir, out_dir, build_root, overlay_file, overlay_manifest,
checkpoint_dir, mode='light' # 或 'reference'
)
```
本门的 comparator、有限库存 coordinator、预注册/执行 receipt 已在独立 engineering 模块实现;必须在服务器通过纯合成测试、完成只读代码审查并锁定全部源码后,才能注册执行。现有 AM comparator 对 AM adapter/provenance 和完整 1000 tick 有硬 schema 约束,不应直接冒充 AN comparator 或修改其旧 globals。
在新的独占输入目录派生 spec,不改 AK/AM 原文件。完整 fresh case 保留原 `name`、model、APL、图、系数和请求;增加显式 `max_steps=1000`,仅调整新的输出预算根及已列明的非科学标签。要求 `record_history=False`。新旧 spec 的逐字段差异必须列入 receipt,不能用“大致相同”或整类忽略代替。
前缀保留原请求中 `tick < end` 的行。分段 `[start, stop)` 只保留该区间原请求,局部 tick 精确减 `start`;全局 neuron_idx 不变。每段 `steps=stop-start`、`duration_ms=steps*0.1`、`max_steps=1000`。将各段请求恢复绝对 tick 后拼接,必须与原请求两个 int32 数组逐位相同、顺序相同、无丢失/重复;不能重抽随机输入。NPZ 容器 SHA 可以因新文件而不同,但数组语义必须 exact,并同时保留文件 SHA。
第一段完成后立即 `seal_checkpoint`;外部 coordinator 把返回的实际 manifest SHA 绑定到下一段新 spec 的 `parent_checkpoint_manifest_sha256`,可同时写准确 `global_tick_offset`。此 SHA 是运行后产出的数据依赖,预注册应锁定派生规则而不是预造值。后续 `run_continued` 自身会产生并封存 checkpoint;父 SHA 仍须外部绑定。fresh seal 是 `state.npz` 的首次新增封存绑定,不能声称它追溯证明 seal 前文件未改:应在同一受控流程中立即比较、封存,保存前后 inventory,期间不允许第三方写入。
AN overlay manifest 使用现有 `phase4an-overlay-v1` 精确键:`schema, overlay_name, overlay_sha256, mask_path, mask_sha256, base_candidate_path, base_candidate_sha256, aj_model_dir, am_overlay_path, am_overlay_sha256`。复用 AM 原 overlay 数组字节及实际 SHA,仅新建 manifest;不能把 AM 原 manifest 的额外字段直接塞给 AN,也不使用 `learned` 模式。
## 3. 预注册必须完整绑定的证据
在任何本门 GPU 执行前,保存不可覆盖的 input/source inventory 与其外部 SHA。至少包括:
- AM 最终状态、model/experiment seal、`protocol/preregistered.json`、`prepared-hashes.json`、所选原 case spec/requests,以及 `gpu/zero/<name>-r0`、`gpu/static25/<name>-r0` 的完整 `output-hashes.json`、comparison、adapter provenance 与全部待比较 artifact。核验完整 AM seal,而非只相信单个 `pass_all`。
- 图的 `nodes/edge_id/src/dst/ptr/signed_raw/weights.npy` 全部文件;APL `input_matrix/output_matrix/kc_globals/apl_globals.npy`;AF 原 `coefficients.json`;AM `protocol/base-candidate.float32-mV.npy`、两个 overlay NPZ/manifest、原 H mask。图和 bundle 的所有 SHA 来自外部锚定的 AM prereg inputs,不是仅由 AN 本次读入后自报。
- AN `runtime_light.py`、`runtime_continuation.py`、对应 tests、即将实现的本门 coordinator/comparator/tests,以及本草案最终注册版。固定其实际 SHA 和源 AST 审计;不得在跑到一半后修改源代码继续使用同一 prereg。
- `runtime_light.PINS` 的整个依赖链:AJ GPU/timing queue、AG-r2 GPU/legacy APL worker、AE backend/C++/CUDA、AF queue helper,以及 `AM_OVERLAY_SHA` 对应原 `overlay.py`。记录 `runtime_continuation.LIGHT_SHA` 与实际 light 文件一致。运行前后都重验旧来源与静态输入,不依赖时间戳缓存来替代字节哈希。
- 新输入每份 spec/requests/manifest 的 SHA,原始记录到前缀/分段的确定性映射,以及库版本、科学 Python 原 venv 入口、GPU UUID/driver/CUDA/Torch/NumPy、编译标志和 extension 来源。Python 用 `-B`;不可通过 resolve venv executable 改变包环境。Torch import 前完成 GPU/CUDA 环境设置;仅记录并核对原 TF32/CuBLAS/确定性配置,不为追求一致或速度偷偷改默认值。
AM 已完成封存并取得小证据下载包;该下载包不含全部大型轨迹,因此预注册仍必须在服务器完整校验 AM 两个 seal。实际 case 名与 seal SHA 依照上述确定规则写入预注册,不以本文代填。缺失 authority、旧输入变化或 AM 不完整时停止,不进入 GPU 门。
## 4. Exact 比较,不把摘要布尔当证明
所有浮点比较按原 dtype 的连续 little-endian raw bits,包括正负零;禁止 tolerance、round、只比均值或把压缩 NPZ 文件字节相等误作数组相等。所有状态要求 finite,事件、计数和请求库存必须完整。
### Fresh light/reference ↔ 同 overlay AM
逐数组比 `state.npz` 六键:`step/neuron_idx/counts/ref` 为 int32,`v/g` 为 float64;全部 N 个末态和全部稀疏事件相同,events 自身排序唯一且 bincount 等于 counts。light 与 reference 都保留 `requests.csv`,其五字段 `tick, neuron_idx, attempted, accepted, rejected` 必须与 AM 原输出逐行 exact;不能只比较请求总数。reference 另比每 tick `v_sha256/g_sha256/ref_sha256`、全部 APL drive/state/generated inhibition/applied h float32、pending tail 和每 tick queue。来源标签、新 spec SHA、记录范围和计时等确有差异的字段须显式白名单,不能整份忽略 provenance。
AN 新 `terminal-queues.npz` 没有对应 AM 原文件时,只从 AM 已记录事实独立重建:1000 tick 最后 19 tick 的全局 sparse events 决定 raw spike ring;末行 `apl-state.float32.npy` 决定 APL state;原 `apl-pending-tail.float32.npy` 决定 chronological pending。比较全部数组及 `next_tick=1000`,不补跑 AM。
light 没有全 APL trace、逐 tick V/g/ref hash 或 queue.jsonl;不能把这些缺文件当空记录、不能称 light 已做这些逐 tick 对照。但 `ReportingOnly.visit_With` 只移除 queue/hash 的两个 context item,留下 `request_output`,也保留 `writer.writerows` 和请求接受判定的 GPU→CPU 拷贝。因此 light 的门包括完整 events/counts/终态、逐行五字段请求接受记录、APL state、完整两种 pending queue、有限值覆盖、总到达/接受计数及固定权重身份与 reference exact;不能把它描述为完全无逐请求记录或完全无逐 tick CPU 同步。reference 承担完整逐 tick 状态/队列观测。light r1 要与 r0 的科学数组、请求 CSV、终态队列及有限值/总量一致,计时不是 exact 字段。
### Monolithic ↔ segmented
每个 overlay、每种模式分别比自己的完整 1000 tick fresh 基线,不跨 overlay 借结果。拼接所有段的 sparse events,并按 receipt offset 恢复绝对 tick;逐段 counts 用 int64 求和、验证范围后与单体 counts exact。末态 V/g/ref、最终 APL state/pending、effective weights raw SHA 必须一致;每个中间切点还与相同绝对长度的 fresh reference 前缀末态对照。切点 5 使用已记录的首段 reference,切点 18/19/500 使用预定前缀,终点使用完整基线。
spike ring 不能直接比较物理行号:各自按 `terminal.next_tick % 19` 转成“下一写 slot 起”的 19 行顺序后逐位比较;continued 的 `next_tick` 是本段局部相位,绝对时间来自 receipt。APL pending 已按下一次应用顺序保存,直接逐位比较;不得把它当未来脑活动或继续衰减的完整尾迹。短段未覆盖的 spike slots 和未消费 APL prefix 还须精确等于父 carry。
reference 的全部逐 tick 状态 hashes 和 APL 四种 trace 按全局时间拼接后与 monolithic 对照;不重复/跳过边界 tick。`queue.jsonl` 的 `joint_sha256` 编码了局部 tick,不能直接跨段相等比较:按全局 tick 对齐,精确比较 `count, accepted_count, ordered_edge_sha256, accepted_mask_sha256`;用完整事件、原 CSR、原 ref 规则独立重建原 source-order queue,并分别验证每份文件自己的 local-header joint hash。不能对 queue 重新按 target 排序后才声称原队列相同。**light 和 reference 两条链的请求 CSV 均须**按 offset 还原 tick、保持其余四字段和行序,拼接后与各自 monolithic CSV 逐行 exact;同时与预定请求库存对应,不能漏掉拒收请求。
上述门可证明 reference 完整记录和 light 已记录状态的一致;**不能据此声称不可观测的 light 每 tick 内部 V/g bits 已被直接逐项比较。** 若正式注册要求这一更强结论,须先单独批准新只读观测器和相应 source pin,而不是运行后扩大结论。
## 5. 防真空和故障门
每个 overlay 都必须从当前 reference 实际记录证明:事件非空;generated 与 applied APL h 至少有非零样本;至少一个预定内部切点同时出现非空 spike carry 和非零 APL pending。还须保存至少一个真实跨边界 synaptic witness:源 spike 在边界之前、`arrival_tick=source_spike_tick+18` 位于边界之后且小于 1000;其真实 canonical edge 权重非零、到达被接受,并包含在原队列哈希重建中。记录 boundary、source/target、edge_id、spike/arrival tick、接受标志及权重,不把这个工程 witness 解释为单一生物学原因。
这些仅是“确实测试过 live carry”的覆盖条件,不是调参阈值。不得强迫所有边界都有活动,也不得用空队列 exact 当作 live-carry PASS。固定 case/切点不满足时,结果为 coverage INCOMPLETE,保存全部输出;不可临时加 pulse、选另一个效果强的 case 或后移边界救场。任何扩展需新草案修订及新 prereg,明确看过既有结果。
故障检查以纯临时合成测试完成:错误外部 SHA、祖先 state 篡改、局部/绝对时钟错配、未写 ring slot 或未消费 APL prefix 改值、输入数组 dtype/NaN/Inf、重复/遗漏事件、拒收被错算接受、输出/编译目录越入任一祖先或冻结输入、已有目录重用。不要故意破坏 AM 或 GPU 正式输出来造负例。
## 6. 计时、压缩与磁盘预算
建议注册一个独立新实验输出根,新增 artifact/build 合计硬上限 **8 GiB**,启动前另保留 **10 GiB** 空闲余量;这是待批准的工程资源上限,不是神经科学门。所有图/APL/旧输出仅只读引用,不复制大型图或全脑 dense history。每次运行前后核实已写字节和剩余空间;不足则停止并保留 partial,不删证据续跑,也不缩短注册 tick 数。
32 次调用共享新的 extension build cache,但不共享可变神经/APL状态,只有声明的父 checkpoint 传状态。记录每次 wall time、已有 `diagnostic_simulation_seconds`、build/cache 时间、完整校验/压缩在 wall 中的费用及峰值 VRAM。reference r0 先行、light r0/r1 随后;明确首次编译/文件缓存顺序造成的影响,不能把一次差值称为普适性能 benchmark。可报告实测 100 ms trial 的耗时/压缩字节,不以此承诺 conditioning 大库存的总成本。
压缩格式按实际来源逐项核对:当前固定 SHA 的 AJ `gpu_timing.py::run_gpu` 写 `state.npz` 的语句是 `np.savez_compressed(target, **state)`,不是 `np.savez`;AN 的 light/reference AST 派生没有替换该语句。`terminal-queues.npz` 则由 AN `_Runtime.finish` 明确使用 `np.savez_compressed`。记录各数组 uncompressed `nbytes`、这两种 NPZ 的实际压缩 bytes、全部 JSONL/CSV/NPY 及 build bytes,并在执行 receipt 核验各 NPZ entry 的实际压缩方式;不得仅凭 `.npz` 扩展名判断。不能只按 NPZ 小体积漏掉 reference APL 原始 `.npy`。reference 四种 APL trace 的主体约为每 tick 32,528 bytes;完整 1000 tick 约 32.528 MB,尚未含事件/末态/队列/头部。所有估计与最终实际 inventory 同报,不预称已节省多少磁盘;不改原输出格式或移除证据以追求预算。
## 7. 执行 receipt 与停止结论
新 coordinator 须 exclusive 写每行执行 receipt 和全门 inventory:注册 SHA、行 ID/overlay/mode/repeat/区间、case/requests/代码输入 SHA、进程退出码、完整 stdout/stderr、开始/结束记录、实际 completed ticks、原始 output inventory SHA、独立 comparator 结果/输入 SHA、finite coverage、weight identity、父 checkpoint 路径/外部 SHA、全局时钟、计时及磁盘字节。单独的 `summary.complete/pass_all` 不足:它可能先于 wrapper 后置核验失败写出;必须有完整 runtime provenance、无 error/failure marker,并重新验证 input/old integrity。
checkpoint manifest、continuation receipt 和外部执行 inventory 各自按现有排除自身的规则封存,不能自哈希循环。新比较报告和 coordinator receipt 放在被封存 checkpoint 目录之外,避免为加报告改坏父链。恢复只可跳过经完整库存/输入/独立比较重验的已完成行;存在 partial、故障 marker 或不完整祖先则停止,不覆盖、不自动科学重试。
全部库存、exact 比较、live-carry 覆盖和前后 integrity 均通过时,最高结论仅为“固定 zero/static25 的 GPU 记录模式与连续状态工程等价门通过”;`learning_pass_claimed=False`、`candidate_generated=False`、`weights_updated_during_run=False`。数值不一致为 FAIL;缺证据、预算中断或 live-carry 未覆盖为 INCOMPLETE;不把未执行项记 PASS。任何结果都不自动授权 conditioning,后续训练协议、独立输入/种子/控制与科学门仍需另行审定。
## 8. 未来可另行注册的补充,不属于当前库存
以下来自独立审查者的工程提案,**不自动追加到本门、不授权执行,也不是当前 32 次调用的 PASS 条件**。需要分别固定新输入、代码/比较器、父 checkpoint 来源和资源预算后重新审定;不可把本草案悄然扩为 90 次调用。
第一项是独立全图 synthetic live-carry 短矩阵。可从冻结 mask 的最低 canonical KC source 及其最低 global edge ID 的真实 MBON target 定义 K、M;从冻结 APL input matrix 的非零列按预定 canonical 规则选 Q,不按新输出选人。拟定 80 tick 的两种请求表:K 在 0/1/22/44/66;必要时 Q 在 0;另一表额外 M 在 18,以覆盖到达接收/不应期拒收、child local tick 0 请求及 tick 84 尚未到期的 spike tail。Q 的实际身份和非零列须静态核验,不能仅凭前缀猜 PN/KC;预期 live/accept/reject 条件也须由实际记录证明。zero/static25 × 两表,各比 reference monolithic 与 reference/light 分段 `[5,12,1,1,18,43]`,提案合计 52 次短调用、960 tick(0.096 生物秒)。它测试人工工程请求,不是新增 odor conditioning;本门固定 AM case 的覆盖不足不能未经新注册就用该矩阵救成 PASS。
第二项是在上述另注册短矩阵通过并得到明确绑定的 active checkpoint 后,再注册较长的连续 idle 工程门:zero/static25 各比较 light 一段 100,000 tick 与连续 10,000 + 90,000 tick,均零新输入、零权重更新,提案合计 6 次 continuation 调用。比较全部 sparse events、分段 counts 总和、终态 V/g/ref/APL state 及规范化 queues;不记录 10 秒逐 tick CPU hash 时,不能宣称逐 tick 浮点状态已独立对照。1 秒 checkpoint 沿同一 idle 主链续到 10 秒;任何测试 probe 必须另 fork,不能把 probe 插入主轨迹,其调用数和预算另列。所有祖先 manifest 在 fork 前后不变。即使通过,也仅证明较长时钟/状态接口,不是记忆保持或 learning PASS。
源码依据:本目录 `runtime_light.py` 的 `validate_spec/load_overlay/derive_ast/_Runtime.finish/_run`;`runtime_continuation.py` 的 `rotate_spike_ring/validate_state/load_checkpoint/run_continued/audit_carried_sequence`;封存 AM `prepare.py/run_suite.py/adapter.py/comparison.py`;封存 AJ `gpu_timing.py` 与 AF `oracle_full.py::queue_summary`。本文只读这些来源,未运行 GPU、未注册或修改其内容。
@@ -0,0 +1,68 @@
# Explicit evaluation policy — draft implementation decision
Status: **DRAFT / NOT REGISTERED / NO SCIENTIFIC EXECUTION**.
This is a pre-outcome implementation choice by the coordinator for the existing
conditioning proposal. It does not change a frozen brain, eligibility-v2,
anatomical mask, odor identity, five-replicate matrix or scientific pass gate.
The AN engineering result remains separate; none of its 21 sources is edited.
## Probe timing and snapshots
- Every evaluation probe lasts 120 ms, with CS requests only in the half-open
interval [0, 100 ms). The remaining 20 ms is a no-CS tail. This matches the
proposed normal training CS duration; it is not the old L full-120-ms input.
- Normal, delayed and reversal curves use 0, 1, 2, 5, 10, 20 and 50 scheduled
reinforcement opportunities. Intermediate snapshots are taken after the
**complete epoch containing** that opportunity and its boundary update,
not immediately at the US tick.
- Terminal checkpoint 50 includes all planned epochs, including trailing
neutral or blank epochs. It is not an early snapshot chosen for response.
- Each snapshot executes both A and B evaluation inputs. No baseline-output
deduplication is assumed. Repeated probes use the same independent evaluation
input within a training replicate; they are paired observations, not new
training replicates. There are still five independent training replicates
per experimental cell and 80 independent A/B evaluation streams overall.
- Probe US and weight updates are disabled. A probe cannot update the training
or retention trajectory. Acquisition/reversal/delayed start from the common
declared reset state; retention forks the actual continuous saved state.
## Intermediate control alignment
For normal and delayed experiments, obtain the paired arm's complete ordered
CS prefix at the chosen epoch boundary. Match actual cue, occurrence, odor,
source random stream and full seed, not just the number of epochs.
- D=0, frozen, random reward and CS-only use that identical CS prefix.
- DAN-only uses the corresponding removed-CS/yoked epoch positions. It has
**zero CS exposure**, not equal CS exposure; this is an intended control.
- Unpaired uses the earliest epoch boundary that has consumed the same ordered
CS prefix, including US-only epochs already inserted before that boundary.
Its terminal checkpoint includes the entire 150-epoch inventory.
Unpaired intermediate US counts and elapsed time can differ from paired.
Random-reward intermediate US counts can also differ. These differences must
be reported rather than called fully matched exposure. All complete terminal
inventories remain fixed by the original proposal. The terminal gate, not the
most favorable intermediate point, decides the result.
Reversal and no-reversal use each arm's own kth reinforcement epoch boundary.
Their intermediate CS prefixes may differ because their reinforced cues differ;
their complete 100-epoch inventories remain fixed. The signed readout always
keeps the original acquisition cue orientation and original pre baseline.
## Inventory, not approval
Under these choices the pure planner is expected to enumerate 9,440 probes
(1,132.8 biological seconds), in addition to the existing 10,760 seconds of
training, controls, reversal and continuous idle. This arithmetic is not a
measured wall-time or disk estimate and excludes engineering validation,
encoding, output, regressions and reserves. Test fixtures use synthetic masters;
none is adopted as a scientific master seed.
Before any scientific run, complete the actual input/state/delta file bindings,
exact event-to-eligibility update coordinator, implemented assay gates and a
finite resource plan. Register the protocol and bounded implementation check
before observing scientific outcomes. Only after those checks may the staged
acquisition suite run. A failed acquisition is sealed as failure; it must not
trigger more pairings, parameter changes, odor reselection or control removal.
@@ -0,0 +1,84 @@
# Phase 4AN:条件化接口的生物学证据与解释边界
状态:**未注册 DRAFT;不是实验结果、科学 PASS 或学习候选声明。**
日期:2026-09-30(Asia/Shanghai)。
本文件记录为后续协议准备的一手文献核对。未修改冻结的脑、符号策略、4J eligibility-v2、mask、eta 或受体假设;未运行远端实验。文献证据不能代替新实验的完整门禁。
## 1. 本地身份、文献类型和具体个体必须分层
冻结 H 的静态身份依据是 `annotation_class`、`annotation_type`、`annotation_instance` 中精确的 compartment token,以及真实 KC→MBON 和 DAN→MBON 边;不是根据 PAM/PPL1 前缀猜测效价或 compartment。可读源码依据为 `../models/male-cns-lif/phase4h.yaml` 的 `audit`/`conditioning` 字段、`phase4h.py` 的 mask/confirmed-DAN 加载,以及 `phase4h_reinforcement_provenance.json`。
下表本地索引、body ID 和侧别是本轮协调者从冻结 H mask/metadata 的实际静态查询确认的目标集合,**不是文献记录的实验个体**。对应冻结 H 文件为 `/data/digital-drosophila/experiments/phase4h-dopamine-plasticity/plastic-mask/plastic-edge-candidates.parquet`,协调者提供的 SHA256 为 `e5075ee3e5a0edf3a4d197453847163cc295b254f21c67203159bba204b12a76`。本次文献核对没有重新下载该表。正式注册仍须将完整静态映射、来源路径及 SHA 与 mask 一同绑定;不得把本表作为绕过静态核验的硬编码替代。
| 本地类型与精确 token | 固定本地 neuron_idx | 已提供的本地 body ID | 文献类型对应 |
| --- | --- | --- | --- |
| MBON01 / `y5B'2a` | `9, 130579` | `9 → 10013`;`130579 → 520151` | MBON-01 = MBON-γ5β′2a |
| MBON12 / `y2a'1` | `130136, 131044, 3477, 3513` | `130136 → 519368 (R)`;`131044 → 521086 (R)`;`3477 → 13728 (L)`;`3513 → 13768 (L)` | MBON-12 = MBON-γ2α′1 |
文献编号与名称的对应来自 [Aso 等,2014,解剖论文 Table 1](https://elifesciences.org/articles/04577/figures)。ASCII token 与希腊字母类型名的对应是本项目跨数据集映射层的一部分,不证明每个 MaleCNS body 的生理效应、性别适用性、行为效价或受体组成。文献不能提供这些本地 neuron_idx/body ID 的身份核验。
冻结 mask 仍为 14,232 条边、四个精确 compartment;另外的 MBON07/`a1` 四个细胞和 MBON02/`B2B'2a` 两个细胞不会因为是 mask 成员而被自动赋予与下面两组相同的行为功能,也不因本次聚焦而从 mask 删除。
## 2. MBON01 与 MBON12 的一手功能证据
### MBON01:avoidance-related 类型,不等于每个本地细胞已验证
Stahl 等在同一条件化实验框架中测得,食欲条件化后 MBON-γ5β′2a 的 CS+ 相对 CS− 响应降低;论文将该 compartment 的输出联系到 avoidance-promoting MBON。该结果支持“减少这一类型的条件气味输出可能产生 appetitive-like 偏移”,不证明所有条件下它都是单一行为开关。[Stahl 等,2022,Figure 5 及 Figure 5—figure supplement 1](https://pmc.ncbi.nlm.nih.gov/articles/PMC8956283/)
较早的光遗传行为实验包含表达于多个 M4/M6 MBON 类型的驱动系;不能把多类型联合激活结果重述为每个 MBON01 个体的独立因果验证。[Aso 等,2014,行为论文 Table 2 与 Figures 2–5](https://cdn.elifesciences.org/articles/04580/elife-04580-v1.pdf)
### MBON12:approach-related 输出与 aversive-learning depression
Aso 等明确报告激活 MBON-γ2α′1 可引起趋近。Berry 等在气味与电击配对后观察到该 MBON 对 CS+ 的响应特异性抑制,未配对/气味和 US 分离的控制不产生相同结果;同一 DAN 的后续刺激还可恢复已抑制响应。因此这里有类型特异的趋近相关输出和厌恶记忆证据,但不支持“DAN 的任何刺激只会继续 LTD”。[Aso 等,2014,行为论文 pp.26–27](https://cdn.elifesciences.org/articles/04580/elife-04580-v1.pdf);[Berry 等,2018,Figures 2–4](https://pmc.ncbi.nlm.nih.gov/articles/PMC6239218/)
Stahl 等还观察到食欲条件化后 MBON-γ2α′1 的相对 CS+ 响应提高,但不能明确全部归因于 CS+ potentiation 或 CS− depression;不能用该结果论证所有相关 plasticity 都是单向 LTD。[Stahl 等,2022](https://pmc.ncbi.nlm.nih.gov/articles/PMC8956283/)
## 3. 两个人工 US 接口只具有预先声明的 like 解释
| 固定的源类型与 compartment 接口 | 类型层面的预期 | 不可升级成的结论 |
| --- | --- | --- |
| PAM15 → `y5B'2a` / MBON01 | 若目标气味的相关 KC 输入发生 LTD,可预期 avoidance-related 输出相对减弱,即 appetitive-like 输出偏移 | PAM15 全体天然奖励、所有 PAM 为正效价、这些本地细胞已经证明奖赏学习 |
| PPL103 → `y2a'1` / MBON12 | 若目标气味的相关 KC 输入发生 LTD,可预期 approach-related 输出相对减弱,即 aversive-like 输出偏移 | 所有 PPL1 为负效价、DAN 任意刺激均是惩罚、这些本地细胞已经证明厌恶学习 |
**PAM15 不是 PAM01 的同义词。** Li 等把 PAM15(γ5β′2a)列为相对既有分类新增的一类;PAM01 对应 γ5,PPL103 对应 γ2α′1。旧文献中 PAM-γ5/PAM01 的全部奖励结果不能不经核对就转记为 PAM15 特异性证据。[Li 等,2020,DAN 分类、Figure 6 与 Figures 35–36](https://pmc.ncbi.nlm.nih.gov/articles/PMC7909955/);[同文图表页](https://elifesciences.org/articles/62576/figures)
这里的 D 仍是冻结规则中的人工标量命令,不是测得的 dopamine 浓度,也不是由实际 DAN spike/接受刺激自动推定的生物学门控。命令、刺激是否被接受、实际 DAN events、选定 compartment 与权重更新必须分别记录。两个接口共用既定 polarity = −1;不因为接口名字不同而翻转 D、eta、base sign 或下游权重。
## 4. 顺序和时间窗证据不等于冻结参数的数值标定
- Hige 等在 γ1pedc 测得正向配对后的抑制;气味改为在最后一束 DAN 激活光后 0.5 秒开始的 backward protocol 未产生同样变化。这是特定 compartment 和刺激协议的结果,不是所有类型的通用截止时间。[Hige 等,2015,Figure 1](https://pmc.ncbi.nlm.nih.gov/articles/PMC4674068/)
- Aso–Rubin 显示,单一 DAN 类型可因相对气味的刺激时序不同而写入、削弱或改变记忆效价;不同 compartment 的学习和更新规则并不相同。[Aso–Rubin,2016](https://pmc.ncbi.nlm.nih.gov/articles/PMC4987137/)
- Handler 等在其协议中测得 forward pairing 对应 depression、backward pairing 对应 potentiation,并观察到 PAM/PPL 群均可随时序改变行为偏移方向。这阻止我们把 PAM/PPL 名称当作固定效价开关;本项目不据此加入新受体、LTP 或重设参数。[Handler 等,2019](https://pmc.ncbi.nlm.nih.gov/articles/PMC9012144/)
出版年复核:该文正式题名为 *Distinct Dopamine Receptor Pathways Underlie the Temporal Sensitivity of Associative Learning*,刊于 **Cell,2019-06-27,178(1):60–75.e19**,online publication 为 2019-06-20,DOI 为 [10.1016/j.cell.2019.05.040](https://doi.org/10.1016/j.cell.2019.05.040)。[PubMed 原文书目记录 PMID 31230716](https://pubmed.ncbi.nlm.nih.gov/31230716/) 明确同时关联 PMC9012144。因此本草稿保留 2019;旧材料中的 2022 不是本论文正式出版年,本次不修改旧冻结材料,也不猜测该旧标注的来源。
冻结 4J eligibility-v2 的 τ = 100 ms、trace cap = 1、eta = 0.00275 mV、polarity = −1、最多 25% depression 仍是**既定 causal-LTD 工程假说**,不是上述论文为当前全脑提供的定量标定。负时序零更新和 +1000 ms/零延迟比例门只验证冻结规则的一致性;不能据此声称复现了生物学 relief learning、受体机制、bidirectional plasticity 或遗忘。4J 中另列的 bidirectional branch 也不是本接口可偷偷启用的替代规则。
## 5. 固定 B/S:新的离线分析约定,不是 decoder
首要输出应保留每个目标细胞的原始 spike/count/rate、训练前后变化,以及两个类型各自的结果。可供正式协议审议的固定类型对比为:
```text
R12(o) = mean of the fixed four MBON12 cells' rates for odor o
R01(o) = mean of the fixed two MBON01 cells' rates for odor o
B(o) = R12(o) - R01(o)
DeltaB(o) = B_post(o) - B_pre(o)
S = DeltaB(CS+) - DeltaB(CS-)
```
所有项使用预先锁定的同一评估窗,分别按类型取均值,避免四个 versus 两个细胞的数量差异直接决定符号。CS+/CS− 的角色须事先指定并 counterbalance,不能根据结果挑选气味方向、细胞或 seed。
这是**新的、需要预注册的离线分析约定**;不是文献给出的定量行为公式,不进入动力学、不拟合权重或阈值、不构建选择策略、classifier 或 decoder,也不替换旧封存指标。它不要求未训练 B = 0,不把 spike rate 等同于真实选择概率。
两个接口永远使用同一个 B 方向:PAM15 分支的假说是 S > 0,PPL103 分支的假说是 S < 0。不得按接口乘效价符号把所有结果改成“正值即成功”,不得取绝对值掩盖方向错误。若反馈使方向相反,应报告为该假说的反证,不改读出或重命名接口。还须与实际运行的预注册控制比较;单独 S 的方向、MBON separation 增大或权重改变都不构成完整 learning PASS。
## 6. 正式协议不可省略的解释门
- 文献类型功能、本地精确身份、人工 US 命令、实际 DAN 激活和实际 KC→MBON 更新须为不同证据字段;任何一层未知都不得用另一层填补。
- 冻结脑与 v2 不变;不得新设 receptor sign、改变 eta、扩 mask、选择最有利 compartment、挑训练 checkpoint 或把 direct/static25 诊断作为条件化本身。
- paired、unpaired、zero-D、frozen、CS-only、DAN-only 应为真实执行控制,不是 summary 中的标签;独立评估 seeds 和预先锁定的 cue 角色不能在看到结果后改变。
- 必须区分 weight update、实际 MBON 响应变化、类型对比方向和下游功能传递。纯 separation 可能在错误方向上增大;“appetitive-like/aversive-like”不能在没有对应证据时升级为行为学习结论。
- 本文件不新增 success threshold;完整通过条件应由尚待注册的协议明示,并保留失败、无效应和反向结果。没有训练结果就不生成 learned candidate。
浏览核对仅采用上列原始研究的期刊、官方 PDF 或 PMC 全文。没有把综述、搜索摘要的二次推断、Google FlyBody 或其他身体运动项目作为本阶段学习机制证据。
@@ -0,0 +1,469 @@
# DRAFT — fixed-v2 olfactory conditioning in the independent D18 branch
**Status: DRAFT / NOT REGISTERED / NOT APPROVED FOR EXECUTION.**
This document is a proposal for review. It does not authorize a run, record a
successful gate, freeze a protocol, or generate a learned candidate. It was
written from frozen H/J/L source and the declared AM interface contract, not
from selecting favorable AM outcomes. The coordinator must approve the final
scope, budget, source/input inventory, and exact gate implementation before
registration. No training or remote action is performed by this document.
## 1. Purpose and entry conditions
The next scientific endpoint should be actual acquisition under the unchanged
TemporalEligibility-v2 rule, followed, if its registered acquisition gate passes,
by fixed reversal, delayed-reward, and real-time retention assays. File validity,
static 25% sensitivity, or a change in delta alone is not learning.
Entry requires the **complete** AM whole-brain interface gate, including its
registered zero-overlay comparisons, nonzero internal/repeat validation, old
reference regression, and frozen integrity checks. A partially completed zero
suite is not this gate. Continue to use the independent D18 branch; do not edit
or overwrite Phase 1–AM artifacts. The original base CSR, anatomical counts,
transmitter signs, W_syn, LIF coefficients, APL calibration, D18 timing,
14,232-edge candidate mask, and eligibility-v2 remain unchanged.
New long-run/continuation plumbing must preserve the exact existing timestep
operation order. Validate zero-delta and fixed-delta continuation, including a
split across pending synaptic/APL arrivals, against an unsplit run before using
it for retention. This is an implementation prerequisite, not another tunable
scientific endpoint. No game, decoder, RL, visual branch, or brain-gain search
is in scope.
## 2. Fixed population and replicate matrix
Use the original externally selected 4L odors without re-selection:
* ODOR_A: **4-ethylphenol**.
* ODOR_B: **3-hexanone**.
* Both frozen encoder modes: `physiological` and `matched_energy`.
* Both reciprocal assignments: A reinforced/B neutral, and B reinforced/A
neutral.
* Both exact DAN/type-compartment branches below.
* Five independent training replicates in each cell.
Thus there are **2 modes × 2 DAN branches × 2 assignments × 5 replicates =
40 blocks**, or eight experimental cells with five training replicates each.
| Branch identity | Exact raw compartment string | Proposed signed prediction |
| --- | --- | --- |
| PAM15 | `y5B'2a` | S > 0 |
| PPL103 | `y2a'1` | S < 0 |
These identities are drawn from old H/J configuration, not from a rule that all
PAM is appetitive or all PPL1 is aversive. Preparation must materialize and hash
the actual DAN body IDs/neuron indices, exact annotation values, confirmed
DAN→MBON/compartment mapping, and affected existing plastic edges from the
sealed H mapping. Missing, ambiguous, or unmatched mappings remain `unknown`;
they cannot be completed by cell-name inference.
The type-level signed predictions above are **new, falsifiable engineering
hypotheses**. Evidence specifically establishing PAM15's proposed functional
direction is limited. Correspondence of type labels across fly datasets is not
proof that these individual MaleCNS bodies have been functionally validated.
Neither branch is labeled a biologically demonstrated reward/aversion channel.
### Seed inventory
Before outcomes, generate and save the complete RNG lineage and explicit seed
table for replicate, cell, arm, epoch, CS events, US ordering, and evaluation.
Use distinct replicate streams, for example an explicitly registered
`SeedSequence` hierarchy with a newly audited master seed. Test expanded seed
identities for collisions. Merely choosing five different base integers is
insufficient: H/J/L used consecutive bases plus the trial number, producing
overlapping nominal replicate streams.
Matched counterfactual arms may deliberately reuse the same input event table;
list each such common-random-number alias. They are paired controls, not extra
independent replicates. All evaluation RNG streams must be disjoint from all
training/US/control RNG streams. Within a replicate use the same registered
evaluation events for pre/post/control comparisons. If more than one evaluation
stream is used per replicate, report/aggregate it within that replicate;
**five evaluation streams never substitute for five independent training
replicates**. The minimal budget below uses one independent A/B evaluation
stream pair per training replicate, reused for paired contrasts.
## 3. Frozen update and explicit US interface
Use the exact frozen J/L v2 implementation and operation order:
* KC trace float32, tau = 100 ms, cap = 1.
* Every timestep: separate multiply, add actual KC spikes, then clamp.
* At the registered US tick, sample after that tick's multiply/add/clamp;
include its KC spikes. Do not insert an anatomical-delay shift into eligibility.
* eta = 0.00275 mV, compartment polarity = −1.
* Additive update only on the already-confirmed compartment mask, with the
old float32 floor `−0.25 × base_candidate_mV`.
* Update once at the end of the training epoch; it affects subsequent epochs,
not an earlier part of the same epoch. No recovery, potentiation, or new mask.
* Install via the verified AM bridge: immutable base SI float64 plus the
float32 delta widened to float64 and converted from mV to V. Preserve all
untouched weights and fast-APL handling exactly.
The first proposal preserves the old **scheduled dimensionless D = 1** artificial
unconditioned-signal interface. It is neither measured dopamine concentration
nor an actual-DAN-spike count. At the same scheduled tick, request spikes in the
fixed matched DAN population. Record requested, accepted, rejected, and actual
DAN spikes, including zero-delivery epochs, separately from D. Do not resample,
boost, or silently discard failed delivery to obtain a preferred result.
Multiplying D by actual acceptance, changing D to a spike count, or adding a
measured dopamine kinetic model would be a distinct, explicitly registered
interface hypothesis; none is hidden in this draft. If neural DAN delivery is
absent, any update still demonstrates scheduled artificial gating, not that
actual DAN activity caused plasticity. This limitation must remain visible even
if an engineering output gate passes.
## 4. Epochs, timing, and all actual controls
Training epochs retain the old trial-boundary reset policy: start V/g/ref,
spike-delay queues, APL state/queue and KC trace from their declared initial
values; carry only plastic_delta between epochs. Log the state policy. This is
not continuous natural behavior. Retention deliberately uses the different,
explicit continuous-state policy in section 7.
Normal epochs last 120 ms. CS occupies 0–100 ms, and a paired US is at 90 ms.
Delayed epochs last 180 ms; the identical CS occupies 0–100 ms, and US is at
150 ms, i.e. **50 ms after CS offset** and 60 ms later than the normal US.
All delayed controls use that same 180 ms duration and 150 ms US slot, including
the neutral epochs; do not reuse 90 ms controls to attribute delayed effects.
The frozen L encoder mapping/rates and original A/B identity are retained.
However, old `phase4l.py::run_rates` generated PN events throughout its passed
120 ms duration. The present 100 ms CS plus a no-CS tail is an explicit new
experimental timing choice aligned with validated AK input duration, **not** a
claim of bitwise identity to old L's full training input. `matched_energy`
matches the frozen encoder's expected total rate, not an artificially edited
actual Poisson event count; report actual requests and acceptance by condition.
Within each arm, present 50 reinforced-cue epochs and 50 neutral-cue epochs in
a pre-generated balanced order. The **50 reinforcement opportunities** come
from L's fixed formal conditioning count, not from a new magnitude search.
| Arm | Actual execution in each normal or delayed suite |
| --- | --- |
| paired | 50 CS+ with its matched DAN/US and 50 CS− without US |
| D=0 | Identical input events and US schedule to paired; D=0 in updates |
| frozen | Identical input events and US schedule; no delta updates |
| random reward | Same 50 A/50 B and 50 US opportunities; exactly 25 A and 25 B get US, selected by a pre-generated random balanced schedule independent of responses |
| unpaired | Same 100 CS epochs and same 50 US events in 50 additional separate, no-CS epochs; preserve total event inventory, declare 150 rather than 100 epochs |
| CS-only | Same 100 CS epochs, no DAN/US; no scheduled D |
| DAN-only | Same 50 US epochs and 50 matched blank epochs, no CS |
Every control is genuinely executed; neither identity reasoning nor a text label
replaces its trajectory. The unpaired arm is specifically **separate-epoch,
reset-trace unpairing**, not a claim about a long continuous biological
interstimulus interval. Actual DAN-only network activity may recruit KC and
produce eligibility; it must be measured, not assumed zero. Blank and US-only
epochs retain the corresponding 90/150 ms timing slot and 120/180 ms duration.
## 5. Offline signed readout — do not discard contrary signs
Preparation defines complete, exact-annotation MBON12 and MBON01 populations
from MaleCNS metadata, with all body/index membership and side information. No
best responding neuron, hemisphere, compartment subset, or evaluation pair is
selected after outcomes. Missing either population makes this specific readout
`NOT-TESTABLE`; do not substitute another type or silently use an empty mean.
For an evaluation stimulus X and a fixed 0–120 ms probe window, define:
`B(X) = mean_spike_count(MBON12, X) − mean_spike_count(MBON01, X)`.
This is **offline analysis only**, not an action decoder, a feedback signal,
or proof of approach/avoidance. Also save both population means separately,
their sizes and counts, all-MBON output, genuine downstream output, and latency.
Keep two distinct downstream inventories. The frozen H actual-edge table's
complete target union is `legacyDownstream` (11,342 neurons in the verified
local data), and contains **all 97 MBONs**. Do not relabel or silently edit that
old population. Define `nonMBONDownstream` as the exact set difference of that
union and the complete MBON population (11,245 here). Report both. Requiring
an effect outside the MBON population is an additional pre-outcome hypothesis,
not an already inherited H gate; it prevents a purely MBON-local change from
being counted twice as evidence of broader propagation.
With C denoting that block's reinforced cue and N its neutral cue:
`S_arm = [B_arm(C) − B_arm(N)] − [B_pre(C) − B_pre(N)]`.
Always emit the raw signed B and S, pre/post components and control S. For PAM15
the proposed relation is **S > 0**; for PPL103 it is **S < 0**. Never replace raw
S in the report with `abs(S)`, `polarity × S`, a magnitude-only score, or a
renamed “positive learning score.” An opposite sign is a counterexample to the
prediction, even if a magnitude or unsigned separation statistic increases.
The expected-control relation is equally explicit:
* PAM15: `S_paired > S_control` for each registered control.
* PPL103: `S_paired < S_control` for each registered control.
Use matched evaluation inputs and each arm's matched pre baseline. All probe
updates and US inputs are off. Evaluation begins from the same frozen initial
state for acquisition/reversal/delayed comparisons; retention uses its saved
continuation state and time-matched frozen control as specified below.
## 6. Acquisition and strict reversal gates
### 6.1 Acquisition — newly proposed signed gate
For each of the eight cells, a successful training replicate must satisfy all:
1. The raw S has the registered direction (PAM15 >0; PPL103 <0).
2. It lies in that direction beyond **each of the six actually executed
timing-matched controls**, not just the easiest control.
3. The inherited L functional requirement holds: conditioned-cue MBON
`Delta_g_max_abs` exceeds neutral-cue `Delta_g_max_abs`, and the conditioned
cue has a nonzero MBON spike-count or first-latency change versus its own pre
probe. Undefined latency is separately reported, not silently zero-filled
into proof of a latency effect.
4. The inherited H MBON and complete `legacyDownstream` A/B Euclidean
separation gains are positive. Independently require positive separation
gain in `nonMBONDownstream` as a newly declared specificity/transfer gate.
Report all three underlying count vectors and gains, never select whichever
downstream definition is more favorable after outcomes.
Require **at least 3 of the 5 independent training replicates to satisfy the
entire conjunction in the same replicate**. Also calculate the inherited H
mean-control rule separately. This exact new combined signed gate is a proposal,
not a claim that H/L already implemented it.
Record acquisition at fixed update counts 0, 1, 2, 5, 10, 20 and 50. Only the
registered terminal count 50 determines its gate. Do not stop at the best
earlier count or add pairings after failure.
### 6.2 Reversal — new hypothesis, not H's permissive sign test
After normal acquisition, fork its exact delta into two arms:
* **Reversal:** 50 pairings of the former neutral cue with the same specific DAN
branch, and 50 formerly reinforced cue epochs without US.
* **No-reversal continuation:** 50 further original-cue pairings and 50 original
neutral epochs, with matched epoch totals and registered independent input
streams. This is an additional real control, included in the budget.
Never reset the learned delta between acquisition and reversal; do not restore
depressed old-cue edges. Keep the old epoch-reset neural state policy in both
arms and evaluate from common fresh states. The pure-LTD/no-recovery rule may
prevent reversal; that is a falsifiable model limitation, not permission to add
potentiation, recovery, or more trials.
For strict reversal, retain the original C−N orientation when calculating
`S_acquisition`, `S_reversal`, and `S_continuation` against the **same original
pre-training baseline**. At the fixed terminal reversal count 50 require:
| DAN branch | Acquisition | Reversal | No-reversal continuation |
| --- | --- | --- | --- |
| PAM15 | `S_acquisition > 0` | `S_reversal < 0` | `S_continuation > 0` |
| PPL103 | `S_acquisition < 0` | `S_reversal > 0` | `S_continuation < 0` |
Additionally require a nonzero change in the former neutral cue's MBON spike
count or defined first latency between acquisition and reversal, and a nonzero
change in its actual-downstream event/count output. Save both cue vectors;
counts alone can conceal a shift confined to the original cue. The same
replicate must satisfy all conditions, with a 3/5 majority per cell. These are
**new terminal reversal hypotheses**, fixed before the learning results.
Zero is failure of a strict sign-crossing gate, not reversal. Report H's original
`sign(acquisition) != sign(reversal)` separately as a legacy statistic; its
nonzero→zero success case does not satisfy this new gate. Do not claim that
these engineering sign changes establish reversal of real appetitive/aversive
behavior.
## 7. Delayed reward and genuine retention gates
### Delayed-reward test — new hypothesis
Run delayed acquisition from zero delta, using the same fixed 50 reinforcement
opportunities and all six controls in the **180 ms / US-at-150 ms** suite. Sample
the unchanged trace at 150 ms after that tick's update. Do not extend tau or
redefine the trace to force delayed learning.
At count 50, require the same signed, functional, downstream and control
conjunction as section 6.1, with at least 3/5 independent training replicates per
cell and raw S direction intact. This explicitly tests whether the unchanged
rule supports this particular delayed schedule. H did not supply an independent
delayed-reward success threshold; this is a new pre-outcome gate.
Report delayed and immediate signed S side by side and their per-replicate
difference. Do not require a invented retained percentage, impose a monotonic
amplitude ratio, pool their seeds, or call an opposite delayed sign a successful
larger absolute response. Delayed controls cannot be substituted by normal ones.
### Retention — new continuous-state implementation and gate
Fork the terminal normal-acquisition checkpoint and the matched terminal normal
frozen-control checkpoint. Each includes V/g/ref, pending spike-delay state,
APL state, the complete D18 APL queue, absolute tick, delta, and relevant RNG
state. Run real no-input/no-update dynamics continuously to 1 s and 10 s; the 1 s
checkpoint is on the same idle trajectory used to reach 10 s. Do not reset,
discard pending events, analytically decay only delta, or fabricate elapsed time
by changing a seed.
At 0, 1, 10 s, fork independent A/B probes from each checkpoint; probes do not feed
back into the idle continuation. Use the same registered evaluation streams
for trained/frozen comparisons at every elapsed time. The probe lasts 120 ms,
without US or updates. Keep delta constant throughout idle; the model does not
contain molecular consolidation or recovery, and a pass cannot prove either.
Define signed `S_trained(t)` and `S_frozen(t)` using the original C−N orientation
and their registered pre baselines. Require, in the **same replicate at both 1 s
and 10 s**:
* PAM15: `S_trained(t) > 0` and `S_trained(t) > S_frozen(t)`.
* PPL103: `S_trained(t) < 0` and `S_trained(t) < S_frozen(t)`.
* Conditioned-cue MBON output has a nonzero spike-count or defined first-latency
difference from the time-matched frozen probe; actual-downstream event/count
output also differs. Save neutral-cue differences and the complete signed
specificity contrasts so a generic global response is not presented as memory.
Require 3/5 independent training replicates per cell, without a fitted amplitude
tolerance or an arbitrarily chosen “percent retained.” This is a new functional
retention hypothesis, not the old H reset-and-retest result. Report an optional
predeclared fresh-state/persistent-delta probe separately as a weight-storage
diagnostic; it cannot satisfy the real-time gate.
## 8. Legacy gates, new hypotheses, and candidate limits
Retain the following old predicates verbatim as separately labeled audit fields
where their data definitions apply:
* H: minimum seed majority 3; positive MBON and downstream Euclidean separation
gains; the smaller mean gain of its two DAN branches exceeds the maximum
mean gain among D0/random/frozen controls.
* H: legacy reversal sign-inequality and last-retention separation>pre. Disclose
the old retention had no actual idle dynamics and is not direct validation of
the new real-time protocol.
* L: conditioned `Delta_g_max_abs`>neutral plus a functional spike/latency change,
in at least 3/5 seeds. Preserve the exact old implementation's legacy result as
well as the new explicitly defined handling of missing latencies.
There was **no completed inherited delayed-reward gate**, no proven signed
MBON12−MBON01 valence mapping for these bodies, and no complete L reversal or
retention assay. The strict signed gates in sections 6–7 are new proposals, not
historical passes and not thresholds obtained from AL/AM responses.
For the strongest proposed engineering candidate claim, all eight cells must
pass their registered acquisition and subsequent assay gates, old required
integrity/control gates, and numerical safety checks. This all-cell conjunction
is itself an explicit proposed scope of claim; it is not a pre-existing H/L
aggregate gate. Emit a complete 8-cell×assay table, including contrary signs,
failures, `NOT-RUN`, `NOT-TESTABLE`, and actual delivery limitations. Do not pool
one mode/compartment/assignment to rescue another, or silently retain only
successful cells as the candidate.
Suggested staged execution: run normal acquisition and all six normal controls
for all 40 blocks first. If the approved overall acquisition gate fails, seal a
real acquisition failure and stop, recording subsequent assays as `NOT-RUN`.
Otherwise execute the already registered complete delayed, reversal,
continuation and retention suites. No gate changes, tuning or new pair choices
occur between stages. A scientific failure is not a reason to delete evidence;
an integrity/numerical/resource failure stops execution without a scientific
success claim.
Even a full engineering pass would not establish odor preference behavior,
motor learning, natural dopamine concentration, appetitive/aversive valence,
long-term memory, or readiness for a game. Delta change alone, a file/checksum
pass, static-overlay sensitivity, or signed output readout alone is insufficient.
## 9. Transparent worst-case runtime and data budget
The original estimate of 38,000 epochs omitted the new no-reversal continuation
and a full timing-matched delayed-control suite. They cannot be silently added
without revising the budget.
| Per block | Epochs | Biological seconds |
| --- | ---: | ---: |
| Normal paired + five 100-epoch controls + 150-epoch unpaired | 750 | 90 |
| Delayed paired + five 100-epoch controls + 150-epoch unpaired | 750 | 135 |
| Reversal | 100 | 12 |
| No-reversal continuation | 100 | 12 |
| Total training/control/continuation | 1700 | 249 |
Across 40 blocks this is **68,000 epochs and 9,960 biological seconds**. Real 10 s
idle progression of trained and frozen retention branches adds 800 s, totaling
**at least 10,760 s biological time** (107.6 million dt=0.1 ms steps), before
evaluation probes, implementation validation, and old regressions. Checkpoints
at 1 s are reused within the 10 s trajectory, not separate additional 10 s runs.
Relative to 38,000: add 4,000 continuation epochs and 26,000 delayed-control epochs.
All delayed neutral/control epochs are 180 ms, not shortened to 120 ms. The table
includes this duration correction. Any approved reduction of scope must be
decided and documented before registration, not after results or a resource
overrun.
Let measured AM walltime per 100 ms be `t100`. A preliminary linear conversion is
`107,600 × t100`, plus input generation, updates, evaluation, I/O and validation.
If `t100=1 s`, this base is 29.9 h; if 5 s, 149.4 h. These are arithmetic scenarios, not
claimed measured performance: learning may change spike load. Measure and
reserve a conservative actual budget before registration; enforce a resource
stop rather than changing the protocol during a run.
Full generated+applied APL traces alone use approximately
`2 × 4064 × 4 × 10000 = 325,120,000 bytes/biological-second`, hence **more than
3.18 TiB** for the above minimum. Do not silently carry the small AM experiment
disk budget into this suite. Per-epoch full neuron counts also total about 45 GB
at 68,000×166,700×4 bytes, before events or full checkpoints.
The proposed recording policy to approve before registration is: all spike
events, actual input/delivery logs, per-epoch counts, signed metrics, every delta
update and full terminal V/g/ref/APL/queue state. Every timestep retains a GPU
finite-value latch and first-failure tick, **not a CPU copy/hash of all state**.
At epoch boundaries hash the files and bind the complete chain of parent,
input, delta-before, events/state/queues, eligibility/D and delta-after. Full
APL/per-tick reference recordings are restricted to a fixed pre-outcome
validation inventory, not a result-selected subset. That subset's exact
comparison must not be described as per-tick regression of all 68,000 epochs.
Compute capacity from measured event/log
sizes plus a stated safety reserve, and register an exclusive-output storage
cap. Do not publish a guessed final compressed size or omit encoder/update/I/O
latency from walltime accounting.
The frozen L/J metric called `MBON_g_max_abs` is the maximum **across neurons
of the terminal post-minus-pre g difference**, not a temporal maximum. Save
`MBON_final_delta_g_max_abs` with explicit SI units and the legacy alias when
needed. Complete terminal g and sparse events suffice to reconstruct that
metric; adding a dense time-by-neuron recording is not required by its name.
Even without full histories, all 68,000 terminal state/ring payloads would use
roughly 507 GB uncompressed, before events and evaluation. Use lossless
compression and measure real complete-episode outputs before setting the hard
storage cap. Do not infer final compressed usage from sparse activity alone.
## 10. Evidence, provenance, and unresolved biology
Local source basis (read-only, relative to the project):
* `23-ai-gpu/source/models/male-cns-lif/phase4h.yaml` and `phase4h.py`:
exact DAN configuration, old control/majority predicates, old nominal retention.
* `23-ai-gpu/source/models/male-cns-lif/phase4j.yaml` and `phase4j.py`:
frozen float32 trace order and `apply_v2`.
* `23-ai-gpu/source/models/male-cns-lif/phase4l.yaml` and `phase4l.py`:
original odor mapping, 50 formal pairings, input timing, and acquisition gate.
* `23-ai-gpu/source/wholebrain-plastic-interface-v1/PROTOCOL.md`:
unchanged-base bridge, scheduled-US semantics and later-learning boundary.
Relevant primary evidence, with narrow use:
* [Hige et al., Neuron 2015, author institution](https://www.janelia.org/publication/heterosynaptic-plasticity-underlies-aversive-olfactory-learning-drosophila):
compartmental KC→MBON plasticity and odor-specific effects motivate local LTD;
this does not measure this MaleCNS model's eta, tau or DAN concentration.
* [Aso and Rubin, eLife 2016, author institution](https://www.janelia.org/publication/dopaminergic-neurons-write-and-update-memories-cell-type-specific-rules):
cell-type-specific DAN learning/update rules oppose a universal cluster-valence
assumption. They do not functionally identify every MaleCNS body here.
* [Aso et al., eLife 2014](https://elifesciences.org/articles/04580):
MBON population outputs motivate population-level readout. The exact
MBON12−MBON01 signed hypothesis and the two predictions in this draft remain
explicit cross-dataset/model assumptions, not a demonstrated behavioral scale.
The exact scalar D interface, fixed trace/eta/floor, absence of recovery,
trial-boundary resetting, simplified signed readout, and 100/120/180 ms timings
are declared model/engineering choices. In particular, the 1.8 ms anatomical
synaptic delay is not an eligibility time constant. Physiological applicability,
PAM15 functional direction in these bodies, continuous intertrial learning,
behavioral valence and long-term retention remain unresolved.
Before formal registration the coordinator must approve and hash the completed
population/seed/input inventory, new readout/gate code, exact continuation
semantics, storage policy/cap, resource budget and final candidate wording.
Until then this remains **DRAFT — NOT REGISTERED — NO EXECUTION AUTHORITY**.
@@ -0,0 +1,118 @@
# Olfactory conditioning lab — unregistered preparation
This directory is an independent, unfinished model-branch implementation.
Its 21 engineering sources are registered and sealed by Phase 4AN; those files
must not be edited. Phase 4AM and the finite Phase 4AN engineering gate passed.
The scientific conditioning proposal and additional planning modules remain
unregistered, have not run training, and have not produced a learned candidate.
Engineering PASS is not learning PASS.
Engineering evidence: `registry/phase4an-engineering-validation` in the project
repository, commit `aeccb7d0ccdcf6788e0a8de60e7ab96f70dbe734`. Its exact
21-source inventory is in `source-identity.json`; this README is not one of
those registered sources. The experiment manifest SHA-256 is
`e8952a48cd3ea6a93c811a1183a1f7b79e19646299924e7688e5f393737a9aa2`.
## Existing code
- `runtime_light.py`: derives a fixed-weight trial runner from the SHA-pinned
AJ source. Its full-recording reference retains the entire original tick
loop. The light variant removes an explicit diagnostics whitelist while
preserving the ordered mathematical AST. It retains sparse events, stimulus
acceptance, terminal neural state and delay queues. It supports an explicit
longer admission bound, not hidden time truncation or approximate delays.
- `seed_registry.py`: explicitly separates training, evaluation and schedule
design random streams. Declared same-split aliases can give controls common
random inputs; aliases do not count as independent replicates. Duplicate
keys, cross-split aliases and seed collisions stop preparation. A full
manifest is independently reconstructible.
- `runtime_continuation.py`: carries full terminal state, spike/APL delay
queues and the absolute clock across fixed-weight segments. It protects
every ancestor checkpoint against output/build writes. The explicit
arrival-admission change and reporting hooks are AST-audited. The finite AN
gate passed 32 actual GPU calls and 74 exact comparisons, including pending
arrivals across a split. This does not verify every future 120/180 ms
training epoch or the unexecuted 1/10 s retention protocol.
- `biological_contract.py`: binds observed canonical identity, exact DAN/MBON
compartment tokens and real graph edges without filling unknown relations.
- `trial_plan.py`: constructs and independently validates the proposed complete
40-block training/control matrix and explicit random-stream aliases. It has
not selected a formal master seed or generated scientific brain inputs.
Complete-protocol measured storage and runtime budgets remain unfinished.
- `evaluation_plan.py`: requires explicit evaluation timing, reversal points
and per-control snapshot bindings. It reconstructs the parent inventory and
retains 80 independent A/B evaluation streams, paired across snapshots within
each training replicate. It does not bind actual state files or authorize a run.
- `evaluation_policy.py`: supplies the explicit draft policy in
`EVALUATION_POLICY_DRAFT.md`: 100 ms CS, 120 ms probes, seven curve points,
actual ordered-CS-prefix control alignment and separate US/time accounting.
Its conditional inventory contains 9,440 probes, not executed trials.
- `conditioning_inputs.py`: consumes a full 63-bit semantic seed in the original
discrete float32 Bernoulli CUDA sampler, with explicit device/draw budget and
CS window. CPU/mock tests do not establish CUDA random-sequence equivalence or
the influence of high seed bits. It never substitutes a CPU random generator.
- `conditioning_update.py`: uses the SHA-pinned AM/J bridge to compute a single
epoch-end delta from supplied actual sparse events. It never installs weights;
scheduled D, actual DAN activity, attempted increments and actual changed
edges remain distinct. Synthetic validation does not establish a trained brain.
- `metrics.py`: pure, explicitly draft functional gates; keeps the historical
downstream population and the non-MBON difference as distinct readouts.
Caller-bound receipts and anatomical membership are still required.
- `test_*.py`: CPU synthetic/structural checks. These cannot establish real
GPU equivalence, biological validity or successful learning.
- `EVIDENCE_DRAFT.md` and `PROTOCOL_DRAFT.md`: evidence and protocol proposals;
neither is an approved or frozen experiment.
## Required before registration or training
1. Preserve completed AM/AN engineering evidence. AM completed its 144 zero
and 144 fixed-overlay runs, five old frozen reference trials, and integrity
checks. AN's 32-call continuation gate passed its finite exact comparisons
and 28-root integrity checks; it did not rerun the five old reference trials.
Neither gate executed conditioning.
2. Review and lock the complete experimental matrix, true executed controls,
separate training/evaluation stream table, data budget and stop rules.
Report any additional gates as new model hypotheses, not inherited facts.
The generic seed registry alone does not prove protocol coverage or semantic
independence: preparation must require every planned stream and permit
aliases only within the same training replicate and input role, for named
matched controls/pre-post comparisons. Never alias two training replicates
or CS and US roles. Save generator/version, target ordering, rates, timing,
consumed stream ID and actual requests hash. Do not truncate a 63-bit seed
to 32 bits or derive a new seed by adding the epoch number downstream.
3. Bind **all** graph arrays, APL bundle, coefficient JSON, base-candidate
vector, full 14,232-edge mask, DAN/MBON identity tables and input files to
the registered manifest. A runtime's self-consistent spec or source hash
is not sufficient authority for these data.
4. Preserve the completed finite real-GPU light/reference comparisons, including exact
ordered events, per-neuron counts, terminal V/g/ref, external-request
acceptance, APL terminal state and both pending delay queues. No tolerance
may excuse a spike divergence. Benchmark encoding, loading, simulation,
validation and output time separately and together.
5. Validate the actual retention protocol before claiming retention. The
separate continuation module passed the finite AN GPU gate. V/g/ref,
spike-ring phase, APL state/pending phase and global tick must survive;
segmented and monolithic executions must agree. A checkpoint seal binds
fresh state bytes when created; it cannot prove pre-seal immutability.
6. Verify every learned snapshot's parent chain, input events and exact
frozen eligibility-v2 update. The single-trial runtime verifies the
bounded overlay and parent-hash declaration, but does not prove the
coordinator actually computed that update. Within each neural trial the
delta is fixed. The original base is never written.
7. Record the realized wall-time/output budget before the full acquisition
suite. A failed acquisition gate must not be rescued by tuning eta,
thresholds, APL gain, trial count, mask membership or selected odors.
## Interpretation limits
PAM15 and PPL103 are specific, statically mapped artificial US interfaces;
PAM/PPL cluster names are not general valence labels. The fixed eligibility
rule is a single-direction causal-LTD engineering hypothesis, not a complete
model of dopamine, bidirectional plasticity, forgetting or consolidation.
The proposed fixed MBON type contrast is offline analysis, not a decoder.
No old sealed artifact, raw connectome, transmitter sign, eligibility-v2 or
base weight is modified here. No game, reinforcement-learning policy,
classifier, action decoder, visual encoder, web service or 3D interface is
introduced. A future learning PASS requires functional stimulus-specific
responses and executed controls, never merely a changed delta array.
@@ -0,0 +1,205 @@
"""Read-only exact-table contracts for the proposed conditioning branch.
No body identity, compartment or behavioral function is inferred from a name.
The specific two interfaces are explicit protocol hypotheses checked against
the frozen H maps and actual graph edges. No training or file write occurs.
"""
from __future__ import annotations
import hashlib
from pathlib import Path
import numpy as np
import pandas as pd
INTERFACES = {"PAM15": ("y5B'2a", "MBON01"), "PPL103": ("y2a'1", "MBON12")}
N = 166700
E = 25582938
INPUT_PINS = {
"datasets/male-cns-v1.0/processed/neuron-only/neuron-index/unified-metadata.parquet":
"922af16846a065c3dbfca7c11146b83ca217fcbd7392edc745f444a04eae8c63",
"experiments/phase4h-dopamine-plasticity/plastic-mask/plastic-edge-candidates.parquet":
"e5075ee3e5a0edf3a4d197453847163cc295b254f21c67203159bba204b12a76",
"experiments/phase4h-dopamine-plasticity/mappings/dan-to-compartment.parquet":
"a16f72c3121ed0ec638c1a2b249ace446e6ccee020b49679818eb42589d00662",
"experiments/phase4h-dopamine-plasticity/mappings/mbon-to-compartment.parquet":
"54f371f7e35b9def44becd1ba9bd2872861f4876b255b2c8ec46feee6473b70c",
"experiments/phase4h-dopamine-plasticity/mappings/dan-to-mbon-confirmed.parquet":
"730d000f25667b59181d2ee6703c64591229a8beab0856197ed71e68d799d191",
}
def require(ok, message):
if not ok:
raise ValueError(message)
def digest(path):
h = hashlib.sha256()
with Path(path).open("rb") as stream:
for block in iter(lambda: stream.read(8 << 20), b""):
h.update(block)
return h.hexdigest()
def _columns(frame, names, label):
require(isinstance(frame, pd.DataFrame) and set(names) <= set(frame.columns),
"Missing observed schema columns: " + label)
def _identity(frame, meta, label):
_columns(frame, ("neuron_idx", "body_id", "annotation_type", "annotation_instance"), label)
require(frame.neuron_idx.dtype == np.int32 and frame.body_id.dtype == np.uint64,
"Canonical identity dtypes changed: " + label)
ids = frame.neuron_idx.to_numpy()
require(len(np.unique(ids)) == len(ids) and np.all((ids >= 0) & (ids < len(meta))),
"Ambiguous/out-of-range identity: " + label)
for field in ("body_id", "annotation_type", "annotation_instance"):
actual = frame[field].reset_index(drop=True)
expected = meta.iloc[ids][field].reset_index(drop=True)
require(actual.equals(expected), "Identity field mismatch: " + label + "/" + field)
def _edges(frame, meta, src, dst, signed_raw, label):
names = ("edge_id", "source_neuron_idx", "source_body_id", "target_neuron_idx", "target_body_id",
"raw_synapse_count", "source_annotation_type", "source_annotation_instance",
"target_annotation_type", "target_annotation_instance")
_columns(frame, names, label)
require(frame.edge_id.dtype == np.int64 and frame.edge_id.is_unique, "Invalid edge IDs: " + label)
ids = frame.edge_id.to_numpy()
require(np.all((ids >= 0) & (ids < len(src))), "Edge outside graph: " + label)
require(frame.raw_synapse_count.dtype == np.int64 and (frame.raw_synapse_count > 0).all(),
"Invalid anatomical contact count: " + label)
require(np.array_equal(np.abs(signed_raw[ids]), frame.raw_synapse_count.to_numpy()),
"Anatomical edge weight mismatch: " + label)
for side, expected_idx in (("source", src[ids]), ("target", dst[ids])):
require(frame[side + "_neuron_idx"].dtype == np.int32 and
frame[side + "_body_id"].dtype == np.uint64,
"Canonical edge identity dtype mismatch: " + label)
require(np.array_equal(frame[side + "_neuron_idx"].to_numpy(), expected_idx),
"Actual graph endpoint mismatch: " + label)
for field in ("body_id", "annotation_type", "annotation_instance"):
actual = frame[side + "_" + field].reset_index(drop=True)
expected = meta.iloc[expected_idx][field].reset_index(drop=True)
require(actual.equals(expected), "Actual edge identity mismatch: " + label + "/" + side + "/" + field)
def audit_interface_tables(meta, dan, mbon, confirmed, candidates, src, dst, signed_raw):
"""Pure table audit; small synthetic graphs are allowed in unit tests only.
The production loader below separately requires the complete N/E identity.
Supplied annotation population membership is never filled from a prefix.
"""
_columns(meta, ("neuron_idx", "body_id", "annotation_class", "annotation_type", "annotation_instance"), "metadata")
require(meta.neuron_idx.dtype == np.int32 and meta.body_id.dtype == np.uint64,
"Canonical metadata dtype mismatch")
require(np.array_equal(meta.neuron_idx.to_numpy(), np.arange(len(meta), dtype=np.int32)) and meta.body_id.is_unique,
"Metadata must have canonical complete ordered identities")
require(src.dtype == dst.dtype == np.int32 and signed_raw.dtype == np.int64 and
src.ndim == dst.ndim == signed_raw.ndim == 1 and len(src) == len(dst) == len(signed_raw),
"Graph endpoint/weight schema mismatch")
require(np.all((src >= 0) & (src < len(meta))) and np.all((dst >= 0) & (dst < len(meta))),
"Graph endpoint out of range")
_identity(dan, meta, "DAN map")
_identity(mbon, meta, "MBON map")
for table, klass, label in ((dan, "DAN", "DAN"), (mbon, "MBON", "MBON")):
require(np.array_equal(np.sort(table.neuron_idx.to_numpy()), np.flatnonzero(meta.annotation_class.eq(klass))),
"Mapping must include the complete exact-annotation population: " + label)
_columns(table, ("mb_compartment_verbatim", "mb_compartment_mapping_status"), label)
_edges(confirmed, meta, src, dst, signed_raw, "confirmed DAN-MBON")
_edges(candidates, meta, src, dst, signed_raw, "plastic candidates")
_columns(confirmed, ("source_compartment_verbatim", "target_compartment_verbatim", "exact_compartment_match"), "confirmed DAN-MBON")
_columns(candidates, ("target_compartment_verbatim", "plasticity_polarity"), "plastic candidates")
require(confirmed.exact_compartment_match.dtype == np.bool_ and confirmed.exact_compartment_match.all(),
"Confirmed table contains an unknown compartment relation")
require(confirmed.source_compartment_verbatim.notna().all() and
confirmed.source_compartment_verbatim.equals(confirmed.target_compartment_verbatim),
"Confirmed exact compartment relation differs")
require(meta.iloc[candidates.source_neuron_idx.to_numpy()].annotation_class.eq("Kenyon_Cell").all() and
meta.iloc[candidates.target_neuron_idx.to_numpy()].annotation_class.eq("MBON").all(),
"Candidate is not a real KC-MBON edge")
require((signed_raw[candidates.edge_id.to_numpy()] > 0).all() and candidates.plasticity_polarity.eq(-1).all(),
"Frozen KC base sign/polarity changed")
pop = {klass: np.flatnonzero(meta.annotation_class.eq(klass)).astype(np.int32)
for klass in ("Kenyon_Cell", "MBON", "DAN")}
reports = {}
for dan_type, (compartment, mbon_type) in INTERFACES.items():
dd = dan[dan.annotation_type.eq(dan_type)].sort_values("neuron_idx")
mm = mbon[mbon.annotation_type.eq(mbon_type)].sort_values("neuron_idx")
require(len(dd) > 0 and len(mm) > 0, "Missing exact type: " + dan_type)
for table in (dd, mm):
require(table.mb_compartment_mapping_status.eq("explicit_single_parenthesized_token").all() and
table.mb_compartment_verbatim.eq(compartment).all(),
"Unknown or mismatched compartment: " + dan_type)
edges = confirmed[confirmed.source_neuron_idx.isin(dd.neuron_idx)]
require(len(edges) > 0 and edges.source_compartment_verbatim.eq(compartment).all() and
edges.target_compartment_verbatim.eq(compartment).all() and
edges.target_neuron_idx.isin(mm.neuron_idx).all(),
"Specific DAN interface has unknown or mismatched actual target")
require(set(edges.source_neuron_idx) == set(dd.neuron_idx) and
set(edges.target_neuron_idx) == set(mm.neuron_idx),
"Incomplete actual DAN/MBON type coverage")
plastic = candidates[candidates.target_compartment_verbatim.eq(compartment)]
require(len(plastic) > 0 and plastic.target_neuron_idx.isin(mm.neuron_idx).all() and
set(plastic.target_neuron_idx) == set(mm.neuron_idx),
"Frozen plastic subset is not the exact mapped target population")
pop[dan_type] = dd.neuron_idx.to_numpy(dtype=np.int32)
pop[mbon_type] = mm.neuron_idx.to_numpy(dtype=np.int32)
def members(frame):
return frame[["neuron_idx", "body_id", "annotation_type", "annotation_instance", "mb_compartment_verbatim"]].to_dict(orient="records")
reports[dan_type] = dict(compartment=compartment, MBON_type=mbon_type,
DAN_members=members(dd), MBON_members=members(mm),
confirmed_DAN_MBON_edge_ids=edges.edge_id.tolist(),
confirmed_DAN_MBON_edges=len(edges), confirmed_DAN_MBON_contacts=int(edges.raw_synapse_count.sum()),
candidate_edges=len(plastic), candidate_contacts=int(plastic.raw_synapse_count.sum()),
unknown_selected_interface_relations=0,
functional_valence="cross-dataset type-level hypothesis, not measured MaleCNS body function",
artificial_US=True, D_equals_measured_dopamine=False)
return pop, dict(interfaces=reports, whole_DAN_mapping_unknown=int(dan.mb_compartment_verbatim.isna().sum()),
whole_MBON_mapping_unknown=int(mbon.mb_compartment_verbatim.isna().sum()),
all_candidate_edges_retained=len(candidates), populations={k: len(v) for k, v in pop.items()},
global_unknowns_filled=False, training=False, new_identity_inferred=False)
def _complete_graph_identity(meta, graph, n=N, e=E):
"""Exact full-graph dimensions; small explicit sizes are for unit tests."""
require(len(meta) == n and graph["nodes"].dtype == np.int32 and
np.array_equal(graph["nodes"], np.arange(n, dtype=np.int32)), "Complete canonical neuron set required")
require(graph["edge_id"].dtype == np.int64 and
np.array_equal(graph["edge_id"], np.arange(e, dtype=np.int64)), "Complete canonical edge set required")
require(all(graph[name].shape == (e,) for name in ("src", "dst", "signed_raw")),
"Complete graph endpoint/weight lengths required")
def load_bound_tables(root, graph_manifest):
"""Require external, registered graph hashes, not self-authorized paths.
graph_manifest maps the absolute graph paths to their authorized SHA-256.
The coordinator must bind this dictionary to the preregistration/parent.
"""
root = Path(root).resolve()
observed = {}
tables = []
for relative, sha in INPUT_PINS.items():
path = root / relative
require(digest(path) == sha, "Frozen table SHA mismatch: " + relative)
observed[str(path)] = sha
tables.append(pd.read_parquet(path))
meta, candidates, dan, mbon, confirmed = tables
graph_dir = root / "experiments/phase4ag-author-apl-r2/graphs/full"
graph = {}
for name in ("nodes", "edge_id", "src", "dst", "signed_raw"):
path = graph_dir / (name + ".npy")
require(str(path) in graph_manifest and digest(path) == graph_manifest[str(path)],
"Unbound/changed complete graph input: " + name)
observed[str(path)] = graph_manifest[str(path)]
graph[name] = np.load(path, mmap_mode="r", allow_pickle=False)
_complete_graph_identity(meta, graph)
require(len(candidates) == 14232, "Complete frozen mask required")
populations, audit = audit_interface_tables(meta, dan, mbon, confirmed, candidates,
graph["src"], graph["dst"], graph["signed_raw"])
require(all(digest(path) == sha for path, sha in observed.items()), "Input bytes changed during audit")
return populations, dict(audit, input_sha256=observed, schema={
name: table.dtypes.astype(str).to_dict() for name, table in
(("metadata", meta), ("candidates", candidates), ("DAN", dan), ("MBON", mbon), ("confirmed", confirmed))})
@@ -0,0 +1,312 @@
"""Exclusive postprocessing of one externally authorized sealed fresh epoch.
No brain execution, source selection, input generation or weight installation.
The external caller must derive and pin authority from its registered row and
successful process receipt; a SHA supplied to this helper is not self-proving
registration authority. Production uses the actual pinned C/R checkpoint and
overlay readers. Small synthetic tests exercise only the private core.
"""
from __future__ import annotations
import hashlib
import json
import math
from pathlib import Path
import sys
from types import ModuleType
import numpy as np
SCHEMA = 'phase4an-epoch-postprocess-authority-v1'
SOURCE_PINS = {
'runtime_light_source': '522b8c31ea938890fe15364fccd25ccc5bded54101754909f255532e7ad01aae',
'runtime_continuation_source': '0b8e8e0a6a23a079dc62a88355ce355d53b3ae22f6665519a8f6aef2dcfc5aba',
'conditioning_update_source': '96efff678cff910654eb08e86f1a17789e4e5589e340345f419c7027a7d547e2',
'eligibility_bridge_source': '174fb8d14619189e74b095e77a6db18b14c93046dc431c0b55710a5aa2dd2e61',
'phase4j_source': 'abb4574da85f00bd9ef61024c65993ef81d42651ff8d8d63c15dd69c6222471e',
'am_overlay_source': '7f891d0ba79b393c96e75fd2dee2eddc51e57d8213eaac2779703bc2c18712b4',
}
ROLES = frozenset(('spec', 'requests', 'overlay_before', 'overlay_manifest_before', 'state',
'kc', 'canonical_kc', 'candidate_map', 'mask', 'base_candidate', *SOURCE_PINS))
GRAPH_FILES = ('nodes.npy', 'edge_id.npy', 'src.npy', 'dst.npy', 'signed_raw.npy', 'ptr.npy', 'weights.npy')
APL_FILES = ('input_matrix.npy', 'output_matrix.npy', 'kc_globals.npy', 'apl_globals.npy')
DIMENSIONS = (166700, 25582938, 4064, 14232)
_MODULES = {}
def _require(ok, message):
if not ok:
raise ValueError(message)
def _sha(value):
return type(value) is str and len(value) == 64 and all(c in '0123456789abcdef' for c in value)
def _digest(path):
h = hashlib.sha256()
with Path(path).open('rb') as stream:
for chunk in iter(lambda: stream.read(8 << 20), b''):
h.update(chunk)
return h.hexdigest()
def _read(path):
def pairs(items):
value = {}
for key, item in items:
_require(key not in value, 'Duplicate JSON key: ' + key)
value[key] = item
return value
return json.loads(Path(path).read_text(encoding='utf-8'), object_pairs_hook=pairs,
parse_constant=lambda value: (_ for _ in ()).throw(ValueError('Nonfinite JSON: ' + value)))
def _save(path, value):
with Path(path).open('x', encoding='utf-8') as stream:
json.dump(value, stream, sort_keys=True, indent=2, allow_nan=False)
stream.write('\n')
def _path(value):
_require(type(value) is str and Path(value).is_absolute(), 'Input paths must be explicit and absolute')
path = Path(value)
_require(path.is_file() and not path.is_symlink(), 'Input must be an existing regular, unlinked file')
return path.resolve(strict=True)
def _check_files(pins):
for path, expected in pins.items():
_require(_sha(expected) and _digest(path) == expected, 'Input/source SHA mismatch: ' + str(path))
def _authority(path, expected):
path = _path(str(path))
_require(_sha(expected) and _digest(path) == expected, 'Authority differs from external SHA')
value = _read(path)
keys = {'schema', 'epoch_id', 'diagnostic_mode', 'command', 'checkpoint_dir',
'checkpoint_manifest_sha256', 'files', 'frozen_inputs'}
_require(type(value) is dict and set(value) == keys and value['schema'] == SCHEMA, 'Invalid authority fields/schema')
_require(type(value['epoch_id']) is str and value['epoch_id'] and value['epoch_id'].strip() == value['epoch_id'], 'Invalid epoch identity')
_require(value['diagnostic_mode'] in ('light', 'reference'), 'Invalid diagnostic mode')
command = value['command']
_require(type(command) is dict and set(command) == {'steps', 'compartment', 'us_tick', 'd_command', 'frozen', 'trace_backend'}, 'Invalid command fields')
_require(type(command['steps']) is int and command['steps'] > 0, 'Invalid command steps')
_require(command['us_tick'] is None or (type(command['us_tick']) is int and 0 <= command['us_tick'] < command['steps']), 'Invalid US tick')
_require(type(command['d_command']) in (int, float) and math.isfinite(command['d_command']) and command['d_command'] in (0, 1), 'Invalid scheduled D')
_require(command['us_tick'] is not None or command['d_command'] == 0, 'No US requires D=0')
_require(type(command['frozen']) is bool and command['trace_backend'] in ('numpy', 'torch'), 'Explicit frozen/backend required')
_require(command['compartment'] in ("B2B'2a", 'a1', "y2a'1", "y5B'2a"), 'Explicit frozen compartment required')
_require(type(value['files']) is dict and set(value['files']) == ROLES, 'Missing/unknown authority file roles')
paths, pins = {}, {path: expected}
for role, entry in value['files'].items():
_require(type(entry) is dict and set(entry) == {'path', 'sha256'} and _sha(entry['sha256']), 'Invalid file authority: ' + role)
location = _path(entry['path'])
_require(location not in pins or pins[location] == entry['sha256'], 'Conflicting hashes for the same file')
paths[role], pins[location] = location, entry['sha256']
_require(type(value['frozen_inputs']) is dict and value['frozen_inputs'], 'Full frozen input pins are required')
for name, sha in value['frozen_inputs'].items():
location = _path(name)
_require(_sha(sha) and (location not in pins or pins[location] == sha), 'Invalid/conflicting frozen input pin')
pins[location] = sha
checkpoint = Path(value['checkpoint_dir'])
_require(checkpoint.is_absolute() and checkpoint.is_dir() and not checkpoint.is_symlink(), 'Explicit existing checkpoint directory required')
checkpoint = checkpoint.resolve(strict=True)
manifest = checkpoint / 'checkpoint-manifest.json'
_require(_sha(value['checkpoint_manifest_sha256']), 'External checkpoint manifest SHA required')
_require(manifest not in pins or pins[manifest] == value['checkpoint_manifest_sha256'], 'Conflicting checkpoint manifest pins')
pins[manifest] = value['checkpoint_manifest_sha256']
_require(paths['state'] == checkpoint / 'state.npz', 'Events must come from this checkpoint state.npz')
_check_files(pins)
return value, paths, pins, checkpoint, path
def _load_source(path, expected, role):
raw = path.read_bytes()
_require(hashlib.sha256(raw).hexdigest() == expected, 'Pinned source changed: ' + role)
name = '_conditioning_epoch_' + role
if name in _MODULES:
saved_path, saved_sha, module = _MODULES[name]
_require(saved_path == path and saved_sha == expected and sys.modules.get(name) is module,
'Source module identity collision: ' + role)
return module
_require(name not in sys.modules, 'Unverified source module collision: ' + role)
module = ModuleType(name)
module.__file__ = str(path)
sys.modules[name] = module
try:
exec(compile(raw, str(path), 'exec'), module.__dict__)
except BaseException:
sys.modules.pop(name, None)
raise
_MODULES[name] = (path, expected, module)
return module
def _production_dependencies(paths, pins):
for role, expected in SOURCE_PINS.items():
_require(pins[paths[role]] == expected, 'Authority cannot replace frozen source: ' + role)
_require(paths['runtime_continuation_source'].parent / 'runtime_light.py' == paths['runtime_light_source'],
'Continuation must load its declared adjacent frozen runtime')
c = _load_source(paths['runtime_continuation_source'], SOURCE_PINS['runtime_continuation_source'], 'continuation')
_require(Path(c.R.__file__).resolve() == paths['runtime_light_source'] and
_digest(c.R.__file__) == SOURCE_PINS['runtime_light_source'], 'Loaded actual runtime differs from authority')
u = _load_source(paths['conditioning_update_source'], SOURCE_PINS['conditioning_update_source'], 'update')
return c, c.R, u
def _npz(path, keys):
with np.load(path, allow_pickle=False) as archive:
_require(len(archive.files) == len(keys) and set(archive.files) == set(keys), 'Unexpected NPZ schema: ' + str(path))
return {key: archive[key].copy() for key in keys}
def _same(a, b):
return a.dtype == b.dtype and a.shape == b.shape and a.tobytes() == b.tobytes()
def _vector(a, dtype, count, label):
_require(a.dtype == np.dtype(dtype) and a.shape == (count,), 'Invalid dtype/shape: ' + label)
def _coordinate(value, paths, pins, checkpoint, authority_path, out_dir, dependencies, dimensions):
"""Private core: explicit tiny-fixture dependencies only in synthetic tests."""
c, r, u = dependencies
n, edges, kc_count, candidates = dimensions
out = Path(out_dir).resolve(strict=False)
protected = {checkpoint, *(path.parent for path in pins)}
_require(not out.exists() and not any(out.is_relative_to(root) for root in protected),
'Output must be new and outside checkpoint/source/data/input parent directories')
_require(out.parent.is_dir(), 'Output parent must already exist')
source_sha = _digest(__file__)
spec = r.validate_spec(_read(paths['spec']), paths['spec'].parent)
command = value['command']
_require(type(spec['steps']) is int and spec['steps'] == command['steps'], 'Command steps differ from authorized spec')
_require(paths['spec'].name == 'spec.json' and paths['requests'] == paths['spec'].parent / 'requests.npz', 'Actual case input filenames differ')
graph = Path(spec['graph_dir']).resolve(strict=True)
apl = Path(spec['apl_bundle_dir']).resolve(strict=True)
required_inputs = [graph / name for name in GRAPH_FILES] + [apl / name for name in APL_FILES] + [Path(spec['coefficients_path']).resolve(strict=True)]
_require(all(path in pins for path in required_inputs), 'Full graph/APL/coefficient authority is incomplete')
_require(paths['canonical_kc'] == apl / 'kc_globals.npy', 'Canonical KC must be the declared frozen APL/KC membership')
old_meta = _read(paths['overlay_manifest_before'])
for role, prefix in (('mask', 'mask'), ('base_candidate', 'base_candidate'), ('am_overlay_source', 'am_overlay')):
_require(Path(old_meta[prefix + '_path']).resolve(strict=True) == paths[role] and
old_meta[prefix + '_sha256'] == pins[paths[role]], 'Overlay authority differs: ' + role)
aj = Path(old_meta['aj_model_dir']).resolve(strict=True)
for relative, expected in r.PINS.items():
path = (aj.parent / relative).resolve(strict=True)
_require(pins.get(path) == expected, 'Missing exact frozen dependency pin: ' + relative)
record = c.load_checkpoint(checkpoint, value['checkpoint_manifest_sha256'])
_require(record['fresh'] is True and record['global_tick_offset'] == 0 and record['steps'] == command['steps'], 'Only complete fresh epochs may produce an update')
summary, provenance = record['summary'], record['provenance']
_require(summary['nodes'] == n and summary['edges'] == edges and summary['diagnostic_mode'] == value['diagnostic_mode'], 'Runtime population/mode differs from authority')
_require(record['manifest']['files_sha256']['state.npz'] == pins[paths['state']], 'State/events are not bound by the expected checkpoint seal')
for role in ('spec', 'requests', 'overlay_before', 'overlay_manifest_before'):
_require(provenance['input_sha256'].get(str(paths[role])) == pins[paths[role]], 'Runtime did not bind authorized ' + role)
_require(_digest(checkpoint / 'overlay-input.npz') == pins[paths['overlay_before']] and
_digest(checkpoint / 'overlay-manifest.json') == pins[paths['overlay_manifest_before']], 'Sealed runtime used a different overlay or manifest')
helper, before, checked_meta, _ = r.load_overlay(paths['overlay_before'], paths['overlay_manifest_before'])
_require(checked_meta == old_meta, 'Overlay manifest changed during validation')
mapping = _npz(paths['candidate_map'], ('edge_id', 'source_neuron_idx', 'target_neuron_idx', 'target_compartment_verbatim'))
mask = np.load(paths['mask'], allow_pickle=False)
kc = np.load(paths['kc'], allow_pickle=False)
canonical_kc = np.load(paths['canonical_kc'], allow_pickle=False)
base = np.load(paths['base_candidate'], allow_pickle=False)
_vector(mask, 'int64', candidates, 'complete mask')
_vector(kc, 'int32', kc_count, 'KC')
_vector(canonical_kc, 'int32', kc_count, 'canonical KC')
_vector(base, 'float32', candidates, 'original base mV')
_require(_same(kc, canonical_kc), 'Supplied KC differs from canonical membership bytes')
_require(_same(mapping['edge_id'], mask) and _same(before['edge_id'], mask), 'Candidate mapping/overlay differ from exact mask')
_require(_same(before['base_candidate_mv'], base), 'Overlay used different original base bytes')
for key in ('source_neuron_idx', 'target_neuron_idx'):
_vector(mapping[key], 'int32', candidates, key)
comps = mapping['target_compartment_verbatim']
_require(comps.shape == (candidates,) and comps.dtype.kind == 'U', 'Compartment mapping must be an exact Unicode vector')
maps = {}
try:
for filename, dtype, length in (('nodes.npy', 'int32', n), ('edge_id.npy', 'int64', edges),
('src.npy', 'int32', edges), ('dst.npy', 'int32', edges)):
array = np.load(graph / filename, allow_pickle=False, mmap_mode='r')
maps[filename] = array
_vector(array, dtype, length, 'full graph ' + filename)
_require(np.array_equal(maps['nodes.npy'], np.arange(n, dtype=np.int32)), 'Global nodes are not canonical')
_require(np.all((mask >= 0) & (mask < edges)) and np.all(mask[1:] > mask[:-1]), 'Invalid canonical edge mask')
_require(_same(maps['edge_id.npy'][mask], mask), 'Mask is not indexed by original global edge ID')
_require(_same(mapping['source_neuron_idx'], maps['src.npy'][mask]) and
_same(mapping['target_neuron_idx'], maps['dst.npy'][mask]), 'Candidate endpoints differ from actual graph')
finally:
for array in maps.values():
array._mmap.close()
state = _npz(paths['state'], ('step', 'neuron_idx', 'counts', 'v', 'g', 'ref'))
_require(all(_same(state[key], record['state'][key]) for key in state), 'Event/state arrays differ from actual sealed reader')
_require(record['overlay']['delta_raw_sha256'] == r._raw(before['plastic_delta_mv']) and
record['overlay']['effective_weights_raw_sha256'] == summary['effective_weights_raw_sha256'], 'Runtime weight installation is not bound to delta_before')
args = dict(step=state['step'], neuron_idx=state['neuron_idx'], kc=kc,
candidate_edge_id=mask, candidate_sources=mapping['source_neuron_idx'],
candidate_targets=mapping['target_neuron_idx'], candidate_compartments=comps,
delta_before=before['plastic_delta_mv'], base_candidate=base, steps=command['steps'], n_neurons=n,
compartment=command['compartment'], us_tick=command['us_tick'], d_command=command['d_command'],
frozen=command['frozen'], bridge_source=paths['eligibility_bridge_source'],
phase4j_source=paths['phase4j_source'], trace_backend=command['trace_backend'])
result = u.update_epoch(**args)
_check_files(pins)
# The directory creation is the exclusive claim. A later failure remains a
# partial directory and can never silently be retried or overwritten.
out.mkdir()
try:
with (out / 'delta-after.npz').open('xb') as stream:
np.savez_compressed(stream, edge_id=mask, plastic_delta_mv=result.delta_after)
metadata = {key: old_meta[key] for key in ('schema', 'mask_path', 'mask_sha256',
'base_candidate_path', 'base_candidate_sha256', 'aj_model_dir', 'am_overlay_path', 'am_overlay_sha256')}
metadata.update(overlay_name='learned', overlay_sha256=_digest(out / 'delta-after.npz'),
parent_snapshot_sha256=pins[paths['overlay_before']], eligibility_v2_source_sha256=SOURCE_PINS['phase4j_source'])
_save(out / 'overlay-manifest.json', metadata)
_save(out / 'update-receipt.json', result.receipt)
_, after, _, _ = r.load_overlay(out / 'delta-after.npz', out / 'overlay-manifest.json')
_require(_same(after['plastic_delta_mv'], result.delta_after) and _same(after['edge_id'], mask), 'Written learned snapshot failed actual runtime loader')
# Re-read the sealed neural state after writing, without altering its
# seal or appending coordinator output inside the runtime directory.
c.load_checkpoint(checkpoint, value['checkpoint_manifest_sha256'])
_check_files(pins)
_require(_digest(__file__) == source_sha, 'Coordinator source changed')
output_hashes = {name: _digest(out / name) for name in ('delta-after.npz', 'overlay-manifest.json', 'update-receipt.json')}
chain = dict(schema='phase4an-epoch-postprocess-chain-v1', complete=True,
epoch_id=value['epoch_id'], authority_file=str(authority_path), authority_sha256=pins[authority_path],
coordinator_source_sha256=source_sha, command=dict(command), diagnostic_mode=value['diagnostic_mode'],
before_overlay_file_sha256=pins[paths['overlay_before']],
before_overlay_manifest_sha256=pins[paths['overlay_manifest_before']],
after_overlay_file_sha256=output_hashes['delta-after.npz'],
after_overlay_manifest_sha256=output_hashes['overlay-manifest.json'],
parent_snapshot_sha256_semantics='delta-before overlay NPZ file SHA256, not neural checkpoint or manifest SHA',
source_neural_checkpoint_dir=str(checkpoint),
source_neural_checkpoint_manifest_sha256=value['checkpoint_manifest_sha256'],
source_state_file_sha256=pins[paths['state']], source_spec_sha256=pins[paths['spec']],
source_requests_sha256=pins[paths['requests']],
runtime_delta_before_raw_sha256=record['overlay']['delta_raw_sha256'],
runtime_effective_weights_raw_sha256=summary['effective_weights_raw_sha256'],
input_files_sha256={str(path): sha for path, sha in sorted(pins.items(), key=lambda item: str(item[0]))},
output_files_sha256=output_hashes,
update_receipt_payload_sha256=result.receipt['payload_sha256'],
runtime_learned_manifest_loader_verified=True, input_and_checkpoint_bytes_reverified=True,
candidate_endpoints_equal_graph=True, KC_equals_supplied_canonical_membership=True,
authority_scope='caller must bind this externally pinned authority to its preregistered row, successful process, anatomical metadata and complete mask audit',
registration_authority_independently_proven=False, anatomical_identity_inferred_from_lengths=False,
neural_checkpoint_and_plastic_parent_are_distinct=True,
weights_installed=False, brain_executed=False, learning_pass_claimed=False, candidate_generated=False)
_save(out / 'chain-receipt.json', chain)
return chain
except BaseException as error:
if not (out / 'postprocess-error.json').exists():
_save(out / 'postprocess-error.json', dict(complete=False, error_type=type(error).__name__,
message=str(error), partial_preserved=True, retry_forbidden=True, learning_pass_claimed=False))
raise
def postprocess_epoch(authority_json, expected_authority_sha256, out_dir):
"""Production entry: full 166700/25582938/4064/14232, no small-mode flag."""
_require(sys.dont_write_bytecode, 'Use python -B; frozen sources must not receive bytecode writes')
value, paths, pins, checkpoint, authority_path = _authority(authority_json, expected_authority_sha256)
dependencies = _production_dependencies(paths, pins)
return _coordinate(value, paths, pins, checkpoint, authority_path, out_dir, dependencies, DIMENSIONS)
@@ -0,0 +1,221 @@
"""UNREGISTERED input consumer; no I/O, brain execution, or training.
The frozen L/AK interface is discrete float32 Bernoulli (one request per PN
and tick), not a continuous-time Poisson sampler. PN/rate mapping is supplied
and externally authenticated by the caller; this module never selects odors,
rebuilds mappings, rescales rates, clips probabilities, or resets brain RNGs.
generate_cs_requests lazily uses the original CUDA Torch random operation.
CPU mocks test its contract only, not CUDA sequence equivalence. A full
(steps, PN-columns) matrix is drawn once, then the explicit half-open CS
window masks requests. A 100 ms CS in a 120 ms epoch is NOT the full old L
120 ms input. Different draw shapes need not share identical random prefixes.
The caller must register source/rate authority, device/environment, budgets,
stream aliases, and a separate bounded CUDA equivalence test before real use.
Returned request/source arrays are owned read-only copies, not immutable
capabilities; the consumer must recheck their recorded hashes before use.
"""
from __future__ import annotations
import hashlib
import re
import numpy as np
N_NEURONS = 166700
L_SOURCE_SHA = 'e97d58d8c1f44920d0e953f8f5831ddc86fa0198784e86460e315de4116ac553'
AK_ENCODER_SHA = '0061689abb1832e963718bd836bc8a50009ee38b258143a753b4d59e63414925'
def require(value, message):
if not value:
raise ValueError(message)
def array_identity(value):
return dict(dtype=value.dtype.str, shape=list(value.shape), order='C-logical-bytes',
sha256=hashlib.sha256(value.tobytes(order='C')).hexdigest())
def _vector(value, dtype, name):
require(isinstance(value, np.ndarray) and value.dtype == np.dtype(dtype) and value.ndim == 1,
name + ' must be an exact one-dimensional ' + np.dtype(dtype).name + ' array')
def _targets(value):
_vector(value, np.int32, 'targets')
require(np.all((value >= 0) & (value < N_NEURONS)), 'Target outside canonical global neuron range')
require(len(np.unique(value)) == len(value), 'Duplicate PN columns rejected; never merge rates or deduplicate RNG columns')
def _clock(steps, start, stop):
require(type(steps) is int and 0 < steps <= np.iinfo(np.int32).max, 'steps must be a positive int32-range integer')
require(type(start) is type(stop) is int and 0 <= start <= stop <= steps, 'Invalid explicit half-open CS tick window')
def probabilities_f32(targets, rates_hz, *, dt_ms=0.1):
"""Preserve the L expression's separate float32 multiply then divide."""
_targets(targets); _vector(rates_hz, np.float32, 'rates_hz')
require(rates_hz.shape == targets.shape and np.all(np.isfinite(rates_hz)) and np.all(rates_hz >= 0), 'Invalid PN rate vector')
require(type(dt_ms) in (int, float) and dt_ms == 0.1, 'Only the frozen dt_ms=0.1 contract is supported')
probability = np.divide(np.multiply(rates_hz, np.float32(dt_ms), dtype=np.float32), np.float32(1000.0), dtype=np.float32)
original = rates_hz * dt_ms / 1000.0
require(original.dtype == np.float32 and original.tobytes() == probability.tobytes(), 'Installed NumPy changed the frozen rate expression')
require(np.all(np.isfinite(probability)) and np.all((probability >= 0) & (probability <= 1)), 'Probability outside [0,1]; no clipping or rate adjustment')
return probability
def forced_from_uniforms(uniforms, probability):
"""Pure arithmetic fixture helper, NOT a replacement CUDA generator."""
_vector(probability, np.float32, 'probability')
require(isinstance(uniforms, np.ndarray) and uniforms.dtype == np.float32 and uniforms.ndim == 2 and
uniforms.shape[1] == len(probability), 'Uniform fixture requires exact float32 [ticks,columns]')
require(np.all(np.isfinite(uniforms)) and np.all((uniforms >= 0) & (uniforms < 1)) and
np.all(np.isfinite(probability)) and np.all((probability >= 0) & (probability <= 1)), 'Invalid uniform/probability fixture')
return uniforms < probability[None, :]
def requests_from_forced(forced, targets, *, cs_start_tick, cs_stop_tick):
"""Sort resulting requests only; never reorder the pre-sampling columns."""
_targets(targets)
require(isinstance(forced, np.ndarray) and forced.dtype == np.bool_ and forced.ndim == 2 and
forced.shape[1] == len(targets), 'Forced samples must be Boolean [steps,original PN columns]')
steps = forced.shape[0]; _clock(steps, cs_start_tick, cs_stop_tick)
ticks, columns = np.nonzero(forced[cs_start_tick:cs_stop_tick])
ticks = ticks.astype(np.int32) + np.int32(cs_start_tick)
neurons = targets[columns].copy()
order = np.lexsort((neurons, ticks))
requests = {'tick': ticks[order], 'neuron_idx': neurons[order]}
validate_requests(requests, steps)
for value in requests.values(): value.setflags(write=False)
generated = int(np.count_nonzero(forced)); retained = len(ticks)
return dict(requests=requests, audit=dict(steps=steps, CS_window=[cs_start_tick, cs_stop_tick],
full_matrix_boolean_hits=generated, retained_CS_requests=retained,
outside_CS_boolean_hits_masked=generated - retained,
original_PN_column_order_preserved=True, sorted_only_after_sampling=True,
duplicate_requests_removed=0, probabilities_clipped=False,
returned_requests_read_only=True, consumer_must_recheck_array_hashes=True,
forced_identity=array_identity(forced), targets_identity=array_identity(targets),
requests_identity={k: array_identity(v) for k, v in requests.items()}))
def _load_torch():
import torch
return torch
def generate_cs_requests(targets, rates_hz, *, seed, steps, cs_start_tick, cs_stop_tick,
device, max_uniform_draws, dt_ms=0.1):
"""Future explicit CUDA input generation; no implicit CPU fallback.
``seed`` is the caller's full semantic seed (Python int in [0,2**63)).
The generator is private to this call. No offset/modulo/truncation or
global/manual brain seed is used. Aliases must reuse the same complete
sampling descriptor, including steps/column order/window, not seed alone.
The caller owns authorization, resource admission, and external rate pins.
``device`` and ``max_uniform_draws`` must be explicitly supplied. The
latter bounds steps * PN columns, not total process/GPU memory. Retaining
initial_seed proves only integer delivery, not CUDA high-bit influence.
"""
require(type(seed) is int and 0 <= seed < 2**63, 'Exact full 63-bit Python integer seed required')
_clock(steps, cs_start_tick, cs_stop_tick)
probability = probabilities_f32(targets, rates_hz, dt_ms=dt_ms)
require(type(device) is str and re.fullmatch('cuda:[0-9]+', device), 'Explicit indexed CUDA device required; no CPU fallback')
require(type(max_uniform_draws) is int and max_uniform_draws > 0, 'Explicit positive integer uniform-draw budget required')
uniform_draws = steps * len(targets)
require(uniform_draws <= max_uniform_draws, 'Full matrix exceeds caller uniform-draw budget; no chunking or clipping')
torch = _load_torch()
require(torch.cuda.is_available() and torch.get_default_dtype() == torch.float32, 'Original CUDA/default-float32 sampler environment required')
actual_device = torch.device(device)
generator = torch.Generator(device=actual_device)
generator.manual_seed(seed)
require(generator.initial_seed() == seed, 'Generator did not retain the complete semantic seed')
p_gpu = torch.as_tensor(probability, dtype=torch.float32, device=actual_device)
# Exactly one complete draw. No chunking, per-PN loop, or tail-dependent redraw.
uniforms = torch.rand((steps, len(targets)), generator=generator, device=actual_device)
require(uniforms.dtype == torch.float32 and str(uniforms.device) == str(actual_device), 'Sampler dtype/device changed')
forced = (uniforms < p_gpu[None, :]).cpu().numpy()
result = requests_from_forced(forced, targets, cs_start_tick=cs_start_tick, cs_stop_tick=cs_stop_tick)
# Optional identifiers are recorded only when the installed Torch exposes
# them. An indexed device name is not presented as a UUID or external pin.
identity = {'uuid': None, 'pci_bus_id': None, 'query_error': None}
try:
properties = torch.cuda.get_device_properties(actual_device)
for key in ('uuid', 'pci_bus_id'):
value = getattr(properties, key, None)
if value is not None: identity[key] = str(value)
except (AttributeError, RuntimeError) as exc:
identity['query_error'] = type(exc).__name__ + ': ' + str(exc)
result['audit'].update(schema='phase4an-conditioning-input-v1-unregistered',
seed=seed, seed_bits=63, seed_hex=f'{seed:016x}', generator_initial_seed=generator.initial_seed(),
seed_conversion='exact integer; no truncation/modulo/offset', private_generator=True,
sampler='Torch CUDA rand default float32; strict U < float32 probability',
generator_type=type(generator).__module__ + '.' + type(generator).__qualname__,
numpy_version=str(np.__version__), torch_version=str(torch.__version__), torch_cuda_version=torch.version.cuda,
execution_device=str(uniforms.device), device_name=str(torch.cuda.get_device_name(actual_device)),
device_capability=list(torch.cuda.get_device_capability(actual_device)),
device_identity=identity, device_identity_verified_against_external_pin=False,
matrix_draw_calls=1, uniform_matrix_shape=[steps, len(targets)], uniform_scalar_draws=uniform_draws,
max_uniform_draws=max_uniform_draws, uniform_matrix_bytes=uniform_draws * np.dtype(np.float32).itemsize,
budget_scope='Uniform scalar count only, not total process or GPU memory',
sampled_before_window_mask=True, dt_ms=dt_ms, dt_float32_identity=array_identity(np.asarray([dt_ms], np.float32)),
rate_units='Hz', probability_identity=array_identity(probability), rates_identity=array_identity(rates_hz),
input_mapping_authority_verified_by_this_module=False,
reference_L_source_sha256=L_SOURCE_SHA, reference_AK_encoder_source_sha256=AK_ENCODER_SHA,
full_old_L_input_identity_claimed=False, CUDA_sequence_equivalence_tested_here=False,
CUDA_high_seed_bit_sequence_influence_tested_here=False,
timing_scope='Full epoch matrix drawn once, then explicit CS window; old L supplied PN requests over its full duration',
RNG_prefix_invariance_across_shapes_claimed=False, continuous_time_Poisson_claimed=False,
brain_reset_called=False, brain_executed=False, training_executed=False, protocol_registered=False)
return result
def validate_requests(requests, steps):
_clock(steps, 0, steps)
require(type(requests) is dict and set(requests) == {'tick', 'neuron_idx'}, 'Exact sparse request schema required')
for key, value in requests.items(): _vector(value, np.int32, key)
tick, neuron = requests['tick'], requests['neuron_idx']
require(tick.shape == neuron.shape and np.all((tick >= 0) & (tick < steps)) and
np.all((neuron >= 0) & (neuron < N_NEURONS)), 'Requests outside local clock/global neuron range')
keys = tick.astype(np.int64) * N_NEURONS + neuron
require(len(keys) < 2 or np.all(keys[1:] > keys[:-1]), 'Requests must already be sorted unique; no silent sorting/deduplication')
return keys
class RequestConflictError(ValueError):
def __init__(self, audit):
super().__init__('CS/US request overlap rejected; explicit boolean_union is required to combine it')
self.audit = audit
def merge_requests(cs, us, *, steps, conflict_policy='error'):
"""Compose two valid streams, recording every cross-source collision.
boolean_union explicitly reproduces a single Boolean request per neuron /
tick, not two deliveries. Separate source attempts are retained in audit
and uint8 source_mask (CS=1, US=2, overlap=3). This is not refractory
acceptance, actual spiking, an artificial D value, or an update rule.
"""
require(conflict_policy in ('error', 'boolean_union'), 'Unknown explicit conflict policy')
cs_keys, us_keys = validate_requests(cs, steps), validate_requests(us, steps)
overlap = np.intersect1d(cs_keys, us_keys, assume_unique=True)
conflicts = [dict(tick=int(k // N_NEURONS), neuron_idx=int(k % N_NEURONS), CS_attempts=1, US_attempts=1,
combined_boolean_requests=1 if conflict_policy == 'boolean_union' else None) for k in overlap]
audit = dict(schema='phase4an-request-composition-v1-unregistered', policy=conflict_policy,
CS_attempts=len(cs_keys), US_attempts=len(us_keys), cross_source_conflict_count=len(overlap), conflicts=conflicts,
duplicate_within_source_allowed=False, refractory_acceptance_evaluated=False, actual_spikes_evaluated=False,
input_identity={name: {k: array_identity(v) for k, v in values.items()} for name, values in (('CS', cs), ('US', us))})
if len(overlap) and conflict_policy == 'error': raise RequestConflictError(audit)
keys = np.union1d(cs_keys, us_keys)
sources = np.zeros(len(keys), np.uint8)
sources[np.searchsorted(keys, cs_keys)] |= np.uint8(1)
sources[np.searchsorted(keys, us_keys)] |= np.uint8(2)
requests = {'tick': (keys // N_NEURONS).astype(np.int32), 'neuron_idx': (keys % N_NEURONS).astype(np.int32)}
validate_requests(requests, steps)
for value in (*requests.values(), sources): value.setflags(write=False)
audit.update(output_requests=len(keys), explicit_boolean_overlap_collapses=len(overlap),
requests_identity={k: array_identity(v) for k, v in requests.items()}, source_mask_identity=array_identity(sources),
returned_arrays_read_only=True, consumer_must_recheck_array_hashes=True,
learning_or_D_command_defined=False, brain_executed=False, protocol_registered=False)
return dict(requests=requests, source_mask=sources, audit=audit)
@@ -0,0 +1,247 @@
"""Pure CPU comparisons for a separately authorized, finite input/update test.
No files, checkpoint seals, source pins, parent trees or GPU are read here.
The coordinator must first verify subprocess receipts, externally pinned
checkpoint manifests, registered source/input files and learned-parent chains.
It must independently recompute update receipts with conditioning_update's
verify_receipt. These functions bind supplied arrays and declared metadata;
they cannot replace those external checks or prove functional learning.
The frozen engineering comparator is reused unchanged. Reference diagnostics
must be complete. Light has events, final state and queues, not per-tick V/g
or APL histories; missing light observations are never fabricated as exact.
"""
from __future__ import annotations
import hashlib
import json
import re
import numpy as np
import engineering_compare as C
N, E, N_KC, CANDIDATE_COUNT = 166700, 25582938, 4064, 14232
COMPARATOR_SHA256 = "5a2403f11bc08d65a9d242ac2cf9686b850e0c3680d65659f785e9b6b50c0238"
PHASE4J_SHA256 = "abb4574da85f00bd9ef61024c65993ef81d42651ff8d8d63c15dd69c6222471e"
ELIGIBILITY_BRIDGE_SHA256 = "174fb8d14619189e74b095e77a6db18b14c93046dc431c0b55710a5aa2dd2e61"
MASK_SHA256 = "8b0fb6eb2e6c31da4d17dc20d890aa3f367e514f05642de5bad6b42a1611c05b"
AM_OVERLAY_SHA256 = "7f891d0ba79b393c96e75fd2dee2eddc51e57d8213eaac2779703bc2c18712b4"
MANIFEST_FIELDS = frozenset(("schema", "overlay_name", "overlay_sha256", "mask_path", "mask_sha256",
"base_candidate_path", "base_candidate_sha256", "aj_model_dir", "am_overlay_path", "am_overlay_sha256",
"parent_snapshot_sha256", "eligibility_v2_source_sha256"))
def _require(condition, message):
if not condition:
raise ValueError(message)
def _canonical(value):
return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False, allow_nan=False)
def _sha(value):
return hashlib.sha256(_canonical(value).encode("utf-8")).hexdigest()
def _valid_sha(value):
return type(value) is str and re.fullmatch("[0-9a-f]{64}", value) is not None
def _evidence(value):
return dict(dtype=value.dtype.str, shape=list(value.shape), nbytes=int(value.nbytes),
raw_c_order_sha256=hashlib.sha256(value.tobytes(order="C")).hexdigest())
def _integer(mapping, key, expected):
return type(mapping.get(key)) is int and mapping[key] == expected
def _request_input(value, steps):
_require(isinstance(value, np.ndarray) and value.dtype == np.int32 and value.ndim == 2 and value.shape[1] == 2,
"Expected input requests must be exact int32 [M,2] (local tick, global neuron)")
_require(np.all((value[:, 0] >= 0) & (value[:, 0] < steps)) and
np.all((value[:, 1] >= 0) & (value[:, 1] < N)), "Expected input request is out of range")
keys = value[:, 0].astype(np.int64) * N + value[:, 1]
_require(len(keys) < 2 or np.all(keys[1:] > keys[:-1]), "Expected requests must already be sorted unique")
def compare_epoch_pair(reference_record, light_record, *, expected_requests, expected_steps,
expected_effective_weight_sha256):
"""Compare fresh reference/light records, not paths or self-asserted seals.
Inputs are dictionaries returned by frozen engineering_compare.load_run.
The coordinator supplies the original registered request array and the
independently expected effective-weight raw SHA, not a value copied from
the summary being checked. Corrupt array schemas raise ValueError. Valid
but different records return numerical_exact/pass_all=False. pass_all is
ONLY the supplied-record identity check, not a learning or activity gate.
"""
_require(type(expected_steps) is int and 0 < expected_steps <= np.iinfo(np.int32).max,
"Explicit expected epoch steps required")
_require(_valid_sha(expected_effective_weight_sha256), "Expected effective-weight raw SHA is required")
_request_input(expected_requests, expected_steps)
_require(type(reference_record) is dict and type(light_record) is dict,
"Supply already authenticated loaded records, not paths")
C.validate_run(reference_record)
C.validate_run(light_record)
inherited = C.compare_fresh(light_record, reference_record)
checks = dict(inherited["checks"])
for label, record, mode in (("reference", reference_record, "reference"), ("light", light_record, "light")):
summary = record.get("summary")
_require(type(summary) is dict, "Missing runtime summary")
checks[label + ".full_population"] = len(record["state"]["v"]) == N and _integer(summary, "nodes", N) and _integer(summary, "edges", E)
checks[label + ".expected_clock"] = (record["steps"] == expected_steps and record["global_tick_offset"] == 0 and
_integer(summary, "steps", expected_steps) and _integer(summary, "completed_ticks", expected_steps) and
_integer(summary, "explicit_steps_executed", expected_steps))
checks[label + ".fresh_fixed_weight_scope"] = (summary.get("checkpoint_continuation_used") is False and
summary.get("within_trial_weight_updates") is False and summary.get("hidden_early_stop") is False and
summary.get("entire_neural_trial_executed") is True and summary.get("record_history") is False and
summary.get("parent_checkpoint_manifest_sha256") is None)
checks[label + ".mode_precision_delay"] = (summary.get("diagnostic_mode") == mode and
summary.get("precision") == "float64" and summary.get("apl_precision") == "float32" and
_integer(summary, "apl_delay_steps", 18) and summary.get("state_units") == "SI volts")
checks[label + ".complete_reported_finite"] = (summary.get("complete") is True and summary.get("pass_all") is True and
summary.get("all_finite") is True and _integer(summary, "first_failure_tick", -1))
checks[label + ".effective_weight_authority"] = summary.get("effective_weights_raw_sha256") == expected_effective_weight_sha256
checks[label + ".input_request_array"] = np.array_equal(record["requests"][:, :2], expected_requests.astype(np.int64))
checks[label + ".request_and_spike_accounting"] = (summary.get("counts_match_sparse_events") is True and
_integer(summary, "total_spikes", len(record["state"]["step"])) and
_integer(summary, "requested", len(expected_requests)) and
_integer(summary, "accepted_requests", int(record["requests"][:, 3].sum(dtype=np.int64))) and
_integer(summary, "rejected_requests", int(record["requests"][:, 4].sum(dtype=np.int64))))
checks[label + ".recorded_full_APL_terminal"] = (record.get("terminal_origin") == "recorded_terminal_queues" and
record["terminal"]["apl_pending"].shape == (18, N_KC))
full = reference_record
full_summary = full["summary"]
checks["reference.full_diagnostics"] = (set(full["traces"]) == set(C.TRACE_FILES) and
full.get("state_hashes") is not None and full.get("queues") is not None and
all(full_summary.get(key) is True for key in
("full_APL_history_recorded", "full_tick_state_hashes_recorded", "full_queue_hashes_recorded", "full_sequence_bitwise_audit_performed")))
if full.get("queues") is not None:
checks["reference.queue_summary_accounting"] = (
_integer(full_summary, "arriving_edges", sum(row["count"] for row in full["queues"])) and
_integer(full_summary, "accepted_arriving_edges", sum(row["accepted_count"] for row in full["queues"])))
else:
checks["reference.queue_summary_accounting"] = False
light_summary = light_record["summary"]
checks["light.declared_recording_scope"] = (not light_record["traces"] and
light_record.get("state_hashes") is None and light_record.get("queues") is None and
all(light_summary.get(key) is False for key in
("full_APL_history_recorded", "full_tick_state_hashes_recorded", "full_queue_hashes_recorded", "full_sequence_bitwise_audit_performed")))
for label, summary in (("reference", full_summary), ("light", light_summary)):
inputs = summary.get("input_sha256")
checks[label + ".input_hash_schema"] = (type(inputs) is dict and
set(inputs) == {"base_weights.npy", "coefficients.json", "requests.npz", "spec.json"} and
all(_valid_sha(value) for value in inputs.values()))
for name in ("base_weights.npy", "coefficients.json"):
left = full_summary.get("input_sha256", {}).get(name)
right = light_summary.get("input_sha256", {}).get(name)
checks["pair.input." + name] = _valid_sha(left) and left == right
for name in ("arriving_edges", "accepted_arriving_edges"):
checks["pair.summary." + name] = (type(full_summary.get(name)) is int and
type(light_summary.get(name)) is int and full_summary[name] == light_summary[name])
checks = {key: bool(value) for key, value in checks.items()}
exact = all(checks.values())
return dict(schema="conditioning-preflight-epoch-comparison-v1", numerical_exact=exact, pass_all=exact,
status="EXACT-RECORDED-EPOCH-PAIR-NOT-LEARNING" if exact else "FAIL-EPOCH-COMPARISON",
checks=checks, failed_checks=[key for key, ok in checks.items() if not ok],
expected_steps=expected_steps, expected_requests=_evidence(expected_requests),
expected_effective_weight_raw_sha256=expected_effective_weight_sha256,
recorded_pair_availability=inherited["reference_comparisons_available"],
activity=inherited["activity"], nonvacuous_activity=inherited["nonvacuous_activity"],
activity_is_descriptive_not_a_new_gate=True,
full_reference_diagnostics_required=True, light_intermediate_states_observed=False,
light_per_tick_V_g_ref_identity_claimed=False, independent_finite_latch_replay_performed=False,
independent_CSR_queue_reconstruction_performed=False,
independent_wholebrain_CPU_oracle_performed=False,
expected_comparator_source_sha256=COMPARATOR_SHA256, source_pin_verified_here=False,
external_file_seals_process_and_input_authority_verified_here=False,
input_generator_or_parent_delta_chain_verified_here=False,
learning_pass_claimed=False, candidate_generated=False,
scope="Supplied fresh epoch records and original requests only; prerequisite external checkpoint/source/parent verification belongs to coordinator")
def _delta(value, name):
_require(isinstance(value, np.ndarray) and value.dtype == np.float32 and value.shape == (CANDIDATE_COUNT,) and
np.isfinite(value).all(), name + " must be the full finite float32 candidate delta vector")
def audit_delta_binding(delta_before, delta_after, installed_delta, *, update_receipt,
parent_snapshot_sha256, child_manifest):
"""Bind already computed/installed arrays to receipt and learned metadata.
parent_snapshot_sha256 means the actual PARENT OVERLAY NPZ file hash,
externally established by the coordinator, NOT a checkpoint manifest hash
and NOT a raw delta-array hash. installed_delta must come from the child
overlay subsequently checked by frozen runtime_light.load_overlay.
This function does not read those files, authenticate anatomical IDs,
recompute eligibility-v2, establish clipping bounds, or prove a causal
behavioral effect. Call conditioning_update.verify_receipt separately
against actual sealed epoch events and complete pinned mappings.
"""
for name, value in (("delta_before", delta_before), ("delta_after", delta_after), ("installed_delta", installed_delta)):
_delta(value, name)
_require(_valid_sha(parent_snapshot_sha256), "Externally established parent overlay FILE SHA is required")
_require(type(update_receipt) is dict and type(child_manifest) is dict, "Receipt and manifest must be dictionaries")
_require(type(update_receipt.get("input_arrays")) is dict and type(update_receipt.get("outputs")) is dict and
type(update_receipt.get("source_pins")) is dict and type(update_receipt.get("command")) is dict and
type(update_receipt.get("semantics")) is dict and type(update_receipt.get("counts")) is dict,
"Incomplete epoch update receipt schema")
before, after, installed = (_evidence(value) for value in (delta_before, delta_after, installed_delta))
command = update_receipt["command"]
counts = update_receipt["counts"]
checks = {
"receipt.self_hash": update_receipt.get("payload_sha256") == _sha({k: v for k, v in update_receipt.items() if k != "payload_sha256"}),
"receipt.schema": update_receipt.get("schema") == "phase4an-conditioning-update-draft-v1" and
update_receipt.get("status") == "DRAFT-COMPUTED-EPOCH-UPDATE-NOT-INSTALLED",
"receipt.original_source_pins": update_receipt["source_pins"] == {
"eligibility_bridge_sha256": ELIGIBILITY_BRIDGE_SHA256, "phase4j_sha256": PHASE4J_SHA256},
"receipt.delta_before": _canonical(update_receipt["input_arrays"].get("delta_before")) == _canonical(before),
"receipt.delta_after": _canonical(update_receipt["outputs"].get("delta_after")) == _canonical(after),
"computed_installed_delta_bitwise": C.bit_equal(delta_after, installed_delta),
"delta.nonpositive": bool(np.all(delta_before <= 0) and np.all(delta_after <= 0)),
"delta.no_potentiation": bool(np.all(delta_after <= delta_before)),
"receipt.candidate_count": _integer(counts, "candidates", CANDIDATE_COUNT),
"receipt.changed_bit_count": _integer(counts, "actual_bitwise_changed_edges", int(np.count_nonzero(delta_before.view(np.uint32) != delta_after.view(np.uint32)))),
"receipt.changed_value_count": _integer(counts, "actual_value_changed_edges", int(np.count_nonzero(delta_before != delta_after))),
"receipt.explicit_scheduled_D": type(command.get("D_scheduled")) is float and command["D_scheduled"] in (0.0, 1.0) and
type(command.get("frozen")) is bool,
"receipt.epoch_boundary_only": update_receipt["semantics"].get("application") == "epoch_boundary_after_all_recorded_ticks; affects subsequent epoch only" and
update_receipt["semantics"].get("delta_dtype") == "float32" and update_receipt["semantics"].get("delta_units") == "mV",
"receipt.not_a_DAN_measurement": update_receipt["semantics"].get("D_definition") == "artificial scheduled scalar" and
update_receipt["semantics"].get("actual_DAN_delivery_observed") is False and
update_receipt["semantics"].get("actual_DAN_delivery_used_to_gate") is False,
"manifest.exact_fields": set(child_manifest) == MANIFEST_FIELDS,
"manifest.learned_schema": child_manifest.get("schema") == "phase4an-overlay-v1" and child_manifest.get("overlay_name") == "learned",
"manifest.parent_overlay_file_sha": child_manifest.get("parent_snapshot_sha256") == parent_snapshot_sha256,
"manifest.frozen_rule_mask_helper": child_manifest.get("eligibility_v2_source_sha256") == PHASE4J_SHA256 and
child_manifest.get("mask_sha256") == MASK_SHA256 and child_manifest.get("am_overlay_sha256") == AM_OVERLAY_SHA256,
"manifest.declared_file_hashes": all(_valid_sha(child_manifest.get(key)) for key in ("overlay_sha256", "base_candidate_sha256")),
"manifest.declared_paths": all(type(child_manifest.get(key)) is str and bool(child_manifest[key])
for key in ("mask_path", "base_candidate_path", "aj_model_dir", "am_overlay_path")),
}
no_update = command.get("frozen") is True or command.get("D_scheduled") == 0.0
checks["zero_D_or_frozen_identity"] = not no_update or C.bit_equal(delta_before, delta_after)
for name in ("weights_installed", "base_modified", "brain_dynamics_executed", "scientific_learning_pass_claimed", "candidate_generated"):
checks["receipt.scope." + name] = update_receipt.get(name) is False
checks = {key: bool(value) for key, value in checks.items()}
exact = all(checks.values())
return dict(schema="conditioning-preflight-delta-binding-v1", pass_all=exact,
status="EXACT-DELTA-ARRAY-BINDING-NOT-LEARNING" if exact else "FAIL-DELTA-BINDING",
checks=checks, failed_checks=[key for key, ok in checks.items() if not ok],
delta_before=before, computed_delta_after=after, installed_delta=installed,
parent_overlay_file_sha256=parent_snapshot_sha256,
declared_child_overlay_file_sha256=child_manifest.get("overlay_sha256"),
update_receipt_payload_sha256=update_receipt.get("payload_sha256"),
actual_changed_edges=int(np.count_nonzero(delta_before != delta_after)),
actual_bitwise_changed_edges=int(np.count_nonzero(delta_before.view(np.uint32) != delta_after.view(np.uint32))),
delta_nonzero_or_change_is_not_learning_evidence=True,
eligibility_v2_independently_recomputed_here=False, full_mask_identity_verified_here=False,
candidate_order_or_compartment_mapping_verified_here=False, floor_bound_independently_verified_here=False,
file_bytes_and_parent_snapshot_authority_verified_here=False, installation_in_runtime_verified_here=False,
actual_DAN_delivery_verified_here=False, learning_pass_claimed=False, candidate_generated=False,
scope="Supplied float32 mV arrays and declared parent/update metadata only; coordinator must verify real files, v2 recomputation and next-epoch installation")
@@ -0,0 +1,288 @@
"""UNREGISTERED bounded preflight input planning, not a conditioning runner.
Only the explicitly called generate_epoch_inputs performs random sampling; it
delegates to conditioning_inputs' single full-shape CUDA draw. The plan and
static rate loader do no sampling, neural execution, plasticity, or I/O writes.
The coordinator owns registration, full seals, graph/overlay authority, device
UUID admission, and every epoch update/checkpoint receipt.
"""
from __future__ import annotations
import ast
import copy
import hashlib
import io
import json
from pathlib import Path, PurePosixPath
from typing import Any
import numpy as np
import conditioning_inputs as inputs
import trial_plan
MASTER = 20260930
BLOCK = 'physiological__PAM15__reinforce_A__rep01'
SCHEMA = 'phase4an-conditioning-preflight-inputs-v1-unregistered'
STATIC = 'experiments/phase4l-olfactory-grounding/'
SOURCE_REL = 'models/male-cns-lif/phase4l.py'
CONFIG_REL = 'models/male-cns-lif/olfactory-grounding-v1/phase4l.yaml'
SOURCE_PINS = {
SOURCE_REL: 'e97d58d8c1f44920d0e953f8f5831ddc86fa0198784e86460e315de4116ac553',
CONFIG_REL: '9b8bcb7e031bf9063033a1854a439a7cd0bf6730301c2d2d02ca99a76f098d6e',
STATIC + 'odor-encoder/frozen-odor-panel.parquet': '907f16a56e024a8e2112b5751e53b0e6037b329fa110098af3d31b119c10a1a0',
STATIC + 'odor-encoder/mapped-glomerular-vectors.npz': 'f057563152e0fa22060080af13019f319e7f36563a775eff829bbc1929c451b0',
STATIC + 'odor-encoder/selection.json': '1b832becf9d08935716fe9ba5e3cd3e619e7d208dfb43c4fa197c969d095f113',
STATIC + 'receptor-mapping/olfactory-mapping.parquet': 'a01d27000533e9844458a4c559cac084f1c34ca20a1104e97e6a0f72b0cc93b1',
}
def require(ok, message):
if not ok:
raise ValueError(message)
def _canonical(value):
return json.dumps(value, sort_keys=True, separators=(',', ':'), ensure_ascii=False, allow_nan=False)
def _sha(value):
return hashlib.sha256(_canonical(value).encode('utf-8')).hexdigest()
def build_preflight_plan():
"""Rebuild the whole semantic plan, then select by rule, never outcomes.
D1/D0/frozen share the two paired input epochs. Every neural epoch starts
fresh; only the externally verified plastic delta can cross its boundary.
This inventory does not enable the complete 68,000-epoch protocol.
"""
full = trial_plan.build_trial_plan(MASTER)
trial_plan.validate_trial_plan(full)
block = full['plan']['blocks'][0]
require(block['block_id'] == BLOCK and block['mode'] == 'physiological' and
block['dan_type'] == 'PAM15' and block['compartment_verbatim'] == "y5B'2a", 'First fixed block changed')
suite = next(s for s in block['suites'] if s['assay'] == 'normal')
paired = next(a for a in suite['arms'] if a['arm'] == 'paired')
first = next(i for i, epoch in enumerate(paired['epochs']) if epoch['us']['requested'])
require(first + 1 < len(paired['epochs']), 'Requested US lacks its following complete epoch')
epochs = copy.deepcopy(paired['epochs'][first:first + 2])
calls = []
for arm in ('D1', 'D0', 'frozen'):
for mode in ('reference', 'light'):
previous = None
for position, epoch in enumerate(epochs):
_validate_epoch(epoch)
row_id = f'{arm}--{mode}--epoch-{position:03d}'
calls.append(dict(row_id=row_id, arm=arm, diagnostic_mode=mode,
input_epoch_index=position, epoch_id=epoch['epoch_id'],
full_plan_epoch_index=epoch['epoch_index'], steps=1200,
d_command=0.0 if arm == 'D0' else epoch['D_command'],
frozen=arm == 'frozen', previous_delta_row_id=previous,
neural_state='fresh_rest_zero_ref_zero_spike_queue_zero_APL_queue',
input_sharing='same saved CS and US arrays across arms and diagnostic modes'))
previous = row_id
result = dict(schema=SCHEMA, master=MASTER, block_id=BLOCK, mode='physiological',
dan_type='PAM15', compartment_verbatim="y5B'2a", full_trial_plan=full,
selected_epochs=epochs, calls=calls, total_calls=12, total_ticks=14400,
input_generation_calls=2, input_generation_executed=False, brain_runs_executed=0,
complete_conditioning_authorized=False, training_result_claimed=False,
selection_rule='first normal paired requested US epoch in first fixed block, plus its immediate successor')
result['payload_sha256'] = _sha(result)
return result
def validate_preflight_plan(value):
require(type(value) is dict and _canonical(value) == _canonical(build_preflight_plan()),
'Preflight inventory differs from full deterministic reconstruction')
return True
def _validate_epoch(epoch):
require(type(epoch) is dict and epoch.get('steps') == 1200 and epoch.get('duration_ms') == 120 and
epoch.get('initial_neural_state') == 'frozen_reset_including_KC_trace_and_both_delay_queues',
'Only complete fresh normal 120 ms epochs are in this preflight')
cs, us = epoch.get('cs'), epoch.get('us')
require(type(cs) is dict and cs.get('cue') in ('A', 'B') and cs.get('odor') == trial_plan.ODORS[cs['cue']] and
type(cs.get('seed')) is int and 0 <= cs['seed'] < 2**63 and cs.get('seed_bits') == 63 and
cs.get('start_tick') == 0 and cs.get('stop_tick_exclusive') == 1000, 'CS semantic descriptor changed')
require(type(us) is dict and type(us.get('requested')) is bool and us.get('nominal_slot_tick') == 900 and
us.get('rng_seed') is None and us.get('rng_draws') == 0, 'US must remain a deterministic scheduled pulse')
require((us['requested'] and us.get('tick') == 900 and us.get('request_rule') == 'fixed_single_pulse_all_bound_DAN_targets') or
(not us['requested'] and us.get('tick') is None and us.get('request_rule') == 'none'), 'US timing/request rule changed')
def _extract_drive(raw, filename):
"""Compile only the SHA-pinned pure pn_drive AST; never import old L."""
import pandas as pd
require(hashlib.sha256(raw).hexdigest() == SOURCE_PINS[SOURCE_REL], 'Frozen L source SHA mismatch')
nodes = [n for n in ast.parse(raw, filename=filename).body if isinstance(n, ast.FunctionDef) and n.name == 'pn_drive']
require(len(nodes) == 1 and not nodes[0].decorator_list, 'Unique undecorated pn_drive required')
namespace = {'np': np, 'pd': pd, 'Any': Any}
exec(compile(ast.Module(body=nodes, type_ignores=[]), filename, 'exec'), namespace)
return namespace['pn_drive'], hashlib.sha256(ast.dump(nodes[0], include_attributes=False).encode()).hexdigest()
def _rate_tables(selection, panel, vectors, mapping, config, drive):
"""Pure static-table derivation, preserving every original PN column."""
require(selection.get('selected_before_KC_simulation') is True and len(panel) == 12 and
panel.odor_id.is_unique and panel.split.tolist() == ['calibration'] * 6 + ['heldout'] * 6,
'Frozen static odor panel schema changed')
gloms = selection['glomeruli_order']
require(type(gloms) is list and len(gloms) == len(set(gloms)) and len(gloms) > 0 and
set(vectors) == set(panel.odor_id), 'Frozen vector/glomerulus inventory mismatch')
encoder = config['odor_encoder']
require(encoder['max_total_rate_per_glomerulus_hz'] == 400.0 and
encoder['modes'] == ['physiological', 'matched_energy'], 'Frozen rate transform changed')
median = float(np.median(panel.mapped_response_sum))
require(np.isfinite(median) and median > 0, 'Invalid panel response median')
tables = {}
for cue, name in trial_plan.ODORS.items():
selected = selection['ODOR_' + cue]
require(selected['odor_name'] == name, 'Selected frozen A/B odor name changed')
odor_id = selected['odor_id']
positions = np.flatnonzero(panel.odor_id.to_numpy() == odor_id)
require(len(positions) == 1, 'Selected odor ID not unique in frozen panel')
panel_row = panel.iloc[int(positions[0])]
require(panel_row.odor_name == name and panel_row.formal_label == 'ODOR_' + cue, 'Panel/selection label mismatch')
vector = vectors[odor_id]
require(isinstance(vector, np.ndarray) and vector.dtype == np.float32 and vector.shape == (len(gloms),) and
np.isfinite(vector).all() and np.all(vector >= 0), 'Invalid frozen glomerular vector; never clip')
targets, rates, frame = drive(odor_id, vector, mapping, gloms, 'physiological', median, config)
inputs.probabilities_f32(targets, rates)
require(len(targets) > 0 and np.all(rates > 0), 'Empty or nonpositive selected PN drive')
require(frame.neuron_idx.to_numpy(np.int32).tobytes() == targets.tobytes() and
frame.rate_hz.to_numpy(np.float32).tobytes() == rates.tobytes(), 'Original frame/array ordering mismatch')
targets, rates = targets.copy(), rates.copy()
targets.setflags(write=False); rates.setflags(write=False)
tables[cue] = dict(cue=cue, odor_id=str(odor_id), odor_name=name, mode='physiological',
source_panel_row=int(positions[0]), source_panel_split=str(panel_row.split),
targets=targets, rates_hz=rates, identities={
'targets': inputs.array_identity(targets), 'rates_hz': inputs.array_identity(rates)},
glomeruli_order=copy.deepcopy(gloms), mapped_vector_identity=inputs.array_identity(vector))
return tables, median
def load_rate_tables(root):
"""Read only the six exact static/source files pinned by frozen AK.
This does not read any historical neural result or rerun odor selection.
Source files are checked before parsing; the parsed bytes are the same
checked snapshots. Full prior seal/registration authority remains external.
"""
import pandas as pd
import yaml
root = Path(root)
raw = {rel: (root / rel).read_bytes() for rel in SOURCE_PINS}
require(all(hashlib.sha256(value).hexdigest() == SOURCE_PINS[rel] for rel, value in raw.items()),
'Frozen static encoder/source SHA mismatch')
source, ast_sha = _extract_drive(raw[SOURCE_REL], str(root / SOURCE_REL))
selection = json.loads(raw[STATIC + 'odor-encoder/selection.json'])
panel = pd.read_parquet(io.BytesIO(raw[STATIC + 'odor-encoder/frozen-odor-panel.parquet']))
mapping = pd.read_parquet(io.BytesIO(raw[STATIC + 'receptor-mapping/olfactory-mapping.parquet']))
config = yaml.safe_load(raw[CONFIG_REL])
with np.load(io.BytesIO(raw[STATIC + 'odor-encoder/mapped-glomerular-vectors.npz']), allow_pickle=False) as values:
require(set(values.files) == set(panel.odor_id) | {'glomeruli'} and
values['glomeruli'].tolist() == selection['glomeruli_order'], 'Frozen NPZ inventory/order mismatch')
vectors = {str(oid): values[oid].copy() for oid in panel.odor_id}
tables, median = _rate_tables(selection, panel, vectors, mapping, config, source)
return dict(tables=tables, audit=dict(schema=SCHEMA, static_sources=copy.deepcopy(SOURCE_PINS),
original_pn_drive_AST_sha256=ast_sha, historical_dynamic_results_read=False,
original_function_called_without_PN_reordering=True, mode='physiological',
median_mapped_response_sum=median, full_seal_verification_owned_by_coordinator=True,
random_draws=0, brain_runs=0, numpy_version=np.__version__, pandas_version=pd.__version__))
def _validate_table(table, cue):
require(type(table) is dict and table.get('cue') == cue and table.get('odor_name') == trial_plan.ODORS[cue] and
table.get('mode') == 'physiological' and type(table.get('odor_id')) is str and table['odor_id'], 'Rate-table cue identity mismatch')
inputs.probabilities_f32(table['targets'], table['rates_hz'])
require(table['identities'] == {key: inputs.array_identity(table[key]) for key in ('targets', 'rates_hz')},
'Rate-table arrays changed after authority binding')
def generate_epoch_inputs(epoch, tables, dan_targets, *, device, max_uniform_draws, conflict_policy='error'):
"""Future explicit sampler, called once per selected input epoch, not arm.
No brain reset is called. The scheduled US request is identical for D1,
D0, and frozen; D-command and update policy belong to the epoch authority.
The caller must preserve and share these returned bytes across all arms.
"""
_validate_epoch(epoch)
cs = epoch['cs']; table = tables[cs['cue']]; _validate_table(table, cs['cue'])
require(isinstance(dan_targets, np.ndarray) and dan_targets.dtype == np.int32 and dan_targets.ndim == 1 and
len(dan_targets) > 0 and np.all((dan_targets >= 0) & (dan_targets < inputs.N_NEURONS)) and
(len(dan_targets) < 2 or np.all(dan_targets[1:] > dan_targets[:-1])), 'Externally bound DAN globals must be sorted unique int32')
require(conflict_policy in ('error', 'boolean_union'), 'Unknown explicit request composition policy')
cs_result = inputs.generate_cs_requests(table['targets'], table['rates_hz'], seed=cs['seed'], steps=1200,
cs_start_tick=0, cs_stop_tick=1000, device=device, max_uniform_draws=max_uniform_draws)
nodes = dan_targets.copy() if epoch['us']['requested'] else np.empty(0, np.int32)
us = {'tick': np.full(len(nodes), 900, np.int32), 'neuron_idx': nodes}
for value in us.values(): value.setflags(write=False)
composed = inputs.merge_requests(cs_result['requests'], us, steps=1200, conflict_policy=conflict_policy)
return dict(requests=composed['requests'], cs_requests=cs_result['requests'], us_requests=us,
source_mask=composed['source_mask'], audit=dict(schema=SCHEMA, epoch_id=epoch['epoch_id'],
CS_sampler=cs_result['audit'], composition=composed['audit'],
DAN_targets_identity=inputs.array_identity(dan_targets), requested_US=epoch['us']['requested'],
scheduled_US_tick=epoch['us']['tick'], CS_shared_across_arms=True, US_shared_across_arms=True,
US_rng_draws=0, semantic_seed_never_passed_to_brain_reset=True, brain_reset_called=False,
source_authority_and_saved_array_rehash_owned_by_coordinator=True))
def derive_case_spec(original, row, epoch, rate_table, *, brain_reset_seed, budget_root, output_budget_bytes):
"""Copy an externally authenticated wholegraph spec with explicit changes.
``seed`` is an independent conservative 31-bit reset-control value, not a
conversion of the semantic input seed. The inherited deterministic AJ
runtime consumes saved requests, not this input RNG. No checkpoint carry
is admitted: the next epoch carries only its separately verified delta.
"""
_validate_epoch(epoch); _validate_table(rate_table, epoch['cs']['cue'])
require(type(original) is dict and original.get('kind') == 'fullgraph' and original.get('dt_ms') == .1 and
original.get('apl_delay_steps') == 18 and original.get('precision') == 'float64' and
original.get('apl_precision') == 'float32' and original.get('record_history') is False and
original.get('within_trial_weight_updates', False) is False, 'Anchored fresh D18 fullgraph spec required')
require(not any('checkpoint' in str(k).lower() or k in ('initial_state', 'resume', 'continuation', 'global_tick_offset') for k in original),
'No neural checkpoint continuation in this two-epoch preflight')
require(type(brain_reset_seed) is int and 0 <= brain_reset_seed < 2**31, 'Independent conservative 31-bit brain reset seed required')
require(type(row) is dict and row.get('arm') in ('D1', 'D0', 'frozen') and row.get('diagnostic_mode') in ('reference', 'light') and
row.get('epoch_id') == epoch['epoch_id'] and row.get('steps') == 1200 and
row.get('frozen') is (row['arm'] == 'frozen') and
row.get('d_command') == (0.0 if row['arm'] == 'D0' else epoch['D_command']), 'Row/epoch command mismatch')
require(type(row.get('input_epoch_index')) is int and row['input_epoch_index'] in (0, 1) and
row.get('row_id') == f"{row['arm']}--{row['diagnostic_mode']}--epoch-{row['input_epoch_index']:03d}", 'Unsafe row identity')
require(type(budget_root) is str and budget_root.startswith('/') and not budget_root.startswith('//') and
'..' not in PurePosixPath(budget_root).parts and
type(output_budget_bytes) is int and output_budget_bytes > 0, 'Explicit absolute output budget required')
old = PurePosixPath(original['budget_root']); new = PurePosixPath(budget_root)
require(old.is_absolute() and not new.is_relative_to(old) and not old.is_relative_to(new), 'Budget cannot overlap old experiment')
result = copy.deepcopy(original)
removed = {key: result.pop(key) for key in ('validation_cohort', 'odor_index', 'expected_external_events', 'training') if key in result}
cs = epoch['cs']; rates = rate_table['rates_hz']
result.update(name=row['row_id'], steps=1200, duration_ms=120.0, max_steps=1200,
budget_root=budget_root, output_budget_bytes=output_budget_bytes, seed=brain_reset_seed,
cs_input_seed=cs['seed'], cs_input_seed_bits=63,
seed_scope='seed is separate reset control; full cs_input_seed used only by saved-input sampler',
mode='physiological', odor_id=rate_table['odor_id'], odor_name=rate_table['odor_name'],
source_panel_row=rate_table['source_panel_row'], source_panel_split=rate_table['source_panel_split'],
rate_sum_hz=float(rates.sum()), target_count=len(rate_table['targets']),
expected_CS_requests=float(np.sum(inputs.probabilities_f32(rate_table['targets'], rates), dtype=np.float64) * 1000),
CS_window=[0, 1000], within_trial_weight_updates=False,
outer_protocol='conditioning_interface_preflight_not_full_conditioning',
outer_epoch_id=epoch['epoch_id'], outer_arm=row['arm'])
changes = {k: dict(present_before=k in original, before=copy.deepcopy(original.get(k)), after=copy.deepcopy(v))
for k, v in result.items() if k not in original or _canonical(v) != _canonical(original[k])}
allowed = {'name', 'steps', 'duration_ms', 'max_steps', 'budget_root', 'output_budget_bytes', 'seed',
'cs_input_seed', 'cs_input_seed_bits', 'seed_scope', 'mode', 'odor_id', 'odor_name', 'source_panel_row',
'source_panel_split', 'rate_sum_hz', 'target_count', 'expected_CS_requests', 'CS_window',
'within_trial_weight_updates', 'outer_protocol', 'outer_epoch_id', 'outer_arm'}
require(set(changes) <= allowed and all(_canonical(result[k]) == _canonical(v) for k, v in original.items()
if k not in allowed and k not in removed), 'Unexpected frozen spec change')
return result, dict(schema=SCHEMA, explicit_field_changes=changes, removed_stale_input_metadata=removed,
original_spec_sha256=_sha(original), derived_spec_sha256=_sha(result),
graph_APL_model_coefficients_and_other_fields_unchanged=True,
independent_brain_reset_seed=brain_reset_seed, semantic_input_seed=cs['seed'],
neural_state_between_epochs='fresh reset; only plastic delta may carry via separate verified authority',
D_and_update_commands_owned_by_epoch_authority=True, random_draws=0, brain_runs=0,
outer_learning_pass_claimed=False, full_conditioning_authorized=False)
@@ -0,0 +1,197 @@
"""Exclusive CPU preparation for the finite AO conditioning-interface gate.
This is a two-epoch installation test, not acquisition, retention or a learning
candidate. Source authority and the complete 12-call plan precede CUDA sampling.
Generated requests and learned snapshots receive separate execution receipts;
their not-yet-existing hashes are never invented in the registration.
"""
from __future__ import annotations
import json
from pathlib import Path
import shutil
import subprocess
import sys
import numpy as np
import engineering_prepare as P
import runtime_light as R
import biological_contract as B
import conditioning_preflight_inputs as I
ROOT = Path('/data/digital-drosophila')
MODEL = Path(__file__).resolve().parent
EXP = ROOT / 'experiments/phase4ao-conditioning-interface'
AN = ROOT / 'experiments/phase4an-engineering-validation'
AN_REG = '7317441126f191ccc6e6d28438870b3921537ebe282a6acb163716a690057945'
AN_MANIFEST = 'e8952a48cd3ea6a93c811a1183a1f7b79e19646299924e7688e5f393737a9aa2'
AN_FINAL = '16630d0d666a2467c56b647d021cbdb627a553795f6afcd7f7eb496a2737d815'
PREP_COMMIT = 'c187d7d5a6047f183467e3d860e2529575dcda97'
DRAFT_INVENTORY_SHA = '2c72d47d09b4ad1949dc6cfdc170f64847ff75f95ded5d20b421fb5a75e0eb84'
BUDGET = dict(output_budget_bytes=4 * 1024**3, minimum_free_reserve_bytes=10 * 1024**3,
max_uniform_draws=2_000_000, child_timeout_seconds=1800)
require, read, save, digest = P.require, P.read, P.save, P.digest
def same(a, b):
return json.dumps(a, sort_keys=True, allow_nan=False) == json.dumps(b, sort_keys=True, allow_nan=False)
def verify_pins(pins):
for name, sha in pins.items():
path = Path(name)
require(path.is_absolute() and path.is_file() and not path.is_symlink() and
digest(path) == sha, 'Pinned file changed: ' + name)
def save_npz(path, **arrays):
with Path(path).open('xb') as stream:
np.savez_compressed(stream, **arrays)
def source_inventory():
# Explicitly snapshot the current lab's text sources, including this new
# coordinator and its tests. Existing 21/20 files are never rewritten.
paths = sorted([*MODEL.glob('*.py'), *MODEL.glob('*.md')])
required = ('conditioning_epoch.py', 'conditioning_preflight_inputs.py',
'conditioning_preflight_compare.py', 'conditioning_preflight_prepare.py',
'conditioning_preflight_workflow.py', 'test_conditioning_preflight_prepare.py',
'CONDITIONING_INTERFACE_PROTOCOL.md')
require(all(MODEL / n in paths for n in required), 'Missing complete coordinator/test/protocol source')
require(not any(p.is_symlink() for p in paths), 'Linked source is forbidden')
return {str(p): digest(p) for p in paths}
def plan_descriptor(full):
return {k: v for k, v in full.items() if k not in ('full_trial_plan', 'payload_sha256')}
def prepare():
require(sys.dont_write_bytecode, 'Use scientific Python -B')
require(EXP.is_dir() and not any(EXP.iterdir()) and not EXP.is_symlink(),
'Preparation requires the explicitly created empty AO directory; never overwrite')
require(digest(AN / 'manifest.json') == AN_MANIFEST and digest(AN / 'audit/final-status.json') == AN_FINAL,
'Externally anchored AN controls changed')
an = P.read_registration(AN_REG)
helper = P.load_module('phase4ao_original_seal_helper', ROOT / P.AI_HELPER_REL, P.AI_HELPER_SHA)
an_seal = helper.verify_seal(AN)
am_seal = P.verify_authority(an['anchors'])
old = P.verify_old_integrity(an)
full = I.build_preflight_plan()
rates = I.load_rate_tables(ROOT)
populations, biology = B.load_bound_tables(ROOT, an['input_sha256'])
pins = dict(an['input_sha256'])
pins.update({str(ROOT / k): v for k, v in I.SOURCE_PINS.items()})
pins.update(biology['input_sha256'])
pins.update({str(AN / 'manifest.json'): AN_MANIFEST, str(AN / 'audit/final-status.json'): AN_FINAL,
str(AN / 'protocol/preregistered.json'): AN_REG})
bridge = ROOT / 'models/male-cns-lif/wholebrain-plastic-interface-v1/eligibility_bridge.py'
phasej = ROOT / 'models/male-cns-lif/mb-physiology-v1/phase4j.py'
pins[str(bridge)] = '174fb8d14619189e74b095e77a6db18b14c93046dc431c0b55710a5aa2dd2e61'
pins[str(phasej)] = 'abb4574da85f00bd9ef61024c65993ef81d42651ff8d8d63c15dd69c6222471e'
original_path = Path(an['selected_case']['original_dir']) / 'spec.json'
require(str(original_path) in pins, 'Original graph spec not anchored by AN')
original = read(original_path)
previous = MODEL / 'draft-source-inventory.json'
require(digest(previous) == DRAFT_INVENTORY_SHA, 'Previously delivered 20-file source inventory changed')
inventory = read(previous)
require(len(inventory['files']) == 20, 'Previous scientific snapshot must contain 20 files')
pins[str(previous)] = DRAFT_INVENTORY_SHA
for entry in inventory['files']:
require(Path(entry['path']).name == entry['path'], 'Nonlocal scientific source inventory entry')
pins[str(MODEL / entry['path'])] = entry['sha256']
zero = an['overlays']['zero']
_, overlay, metadata, _ = R.load_overlay(zero['overlay_file'], zero['overlay_manifest'])
require(not np.any(overlay['plastic_delta_mv'].view(np.uint32)), 'Initial delta must be exact positive zero')
kc = np.load(Path(original['apl_bundle_dir']) / 'kc_globals.npy', allow_pickle=False)
require(np.array_equal(kc, populations['Kenyon_Cell']), 'Canonical metadata and APL KC memberships disagree')
import pandas as pd
candidate = pd.read_parquet(ROOT / 'experiments/phase4h-dopamine-plasticity/plastic-mask/plastic-edge-candidates.parquet')
candidate = candidate.set_index('edge_id').loc[overlay['edge_id']].reset_index()
require(candidate.edge_id.to_numpy().tobytes() == overlay['edge_id'].tobytes() and
candidate.target_compartment_verbatim.notna().all(), 'Complete mask compartment mapping is not exact')
verify_pins(pins)
sources = source_inventory()
code_commit = subprocess.run(['git', '-C', str(ROOT), 'rev-parse', 'HEAD'], check=True,
capture_output=True, text=True).stdout.strip()
require(P.sha(code_commit + '0' * 24), 'Invalid Git code version')
for path, sha in sources.items():
relative = Path(path).relative_to(ROOT).as_posix()
blob = subprocess.run(['git', '-C', str(ROOT), 'show', code_commit + ':' + relative],
check=True, capture_output=True).stdout
require(__import__('hashlib').sha256(blob).hexdigest() == sha, 'Registered source is not the committed Git blob: ' + relative)
for name in ('protocol/source-snapshot', 'inputs/static', 'cases', 'runs', 'updates', 'receipts', 'logs', 'audit', 'regression'):
(EXP / name).mkdir(parents=True)
for path in sources:
shutil.copyfile(path, EXP / 'protocol/source-snapshot' / Path(path).name)
save(EXP / 'regression/AN-seal-before.json', an_seal)
save(EXP / 'regression/AM-seals-before.json', am_seal)
save(EXP / 'regression/old-integrity-before.json', old)
save(EXP / 'inputs/static/biology-audit.json', biology)
save(EXP / 'inputs/static/rates-audit.json', rates['audit'])
save_npz(EXP / 'inputs/static/populations.npz', **populations)
save_npz(EXP / 'inputs/static/candidate-map.npz', edge_id=candidate.edge_id.to_numpy(np.int64),
source_neuron_idx=candidate.source_neuron_idx.to_numpy(np.int32),
target_neuron_idx=candidate.target_neuron_idx.to_numpy(np.int32),
target_compartment_verbatim=candidate.target_compartment_verbatim.to_numpy(dtype=str))
table_metadata = {}
for cue, table in rates['tables'].items():
save_npz(EXP / f'inputs/static/{cue}-rates.npz', targets=table['targets'], rates_hz=table['rates_hz'])
table_metadata[cue] = {k: v for k, v in table.items() if k not in ('targets', 'rates_hz')}
save(EXP / 'inputs/static/rate-tables.json', table_metadata)
save(EXP / 'protocol/full-trial-plan.json', full['full_trial_plan'])
descriptor = plan_descriptor(full)
cases = {}
reset_seed = original['seed']
for row in descriptor['calls']:
epoch = descriptor['selected_epochs'][row['input_epoch_index']]
case = EXP / 'cases' / row['row_id']
case.mkdir()
spec, derivation = I.derive_case_spec(original, row, epoch, rates['tables'][epoch['cs']['cue']],
brain_reset_seed=reset_seed, budget_root=str(EXP), output_budget_bytes=BUDGET['output_budget_bytes'])
R.validate_spec(spec, case)
save(case / 'spec.json', spec)
save(case / 'derivation.json', derivation)
cases[row['row_id']] = str(case)
derived = {str(p): digest(p) for parent in ('inputs/static', 'cases', 'protocol/source-snapshot', 'regression')
for p in (EXP / parent).rglob('*') if p.is_file()}
derived[str(EXP / 'protocol/full-trial-plan.json')] = digest(EXP / 'protocol/full-trial-plan.json')
reg = dict(schema='phase4ao-conditioning-interface-registration-v1', plan=descriptor,
full_plan_payload_sha256=full['payload_sha256'], full_trial_plan_file_sha256=derived[str(EXP / 'protocol/full-trial-plan.json')],
source_sha256=sources, input_sha256=pins, prepared_file_sha256=derived, cases=cases,
initial_overlay=zero, original_spec=str(original_path), brain_reset_seed=reset_seed,
bridge_source=str(bridge), phase4j_source=str(phasej), budget=BUDGET,
git_parent_commit=PREP_COMMIT, git_code_commit=code_commit, committed_source_blobs_verified=len(sources), AN_registration_sha256=AN_REG,
generation=dict(device='cuda:0', GPU_UUID='GPU-79f3a85d-adce-7833-7003-6488140ca596',
conflict_policy='error', highbit_probe_seed=descriptor['selected_epochs'][0]['cs']['seed'],
highbit_comparison='full63bit versus low32bit, one fixed 1200xPN matrix each; never a seed search',
original_L_sampling_AST='only rate tensor, private generator and full-shape forced-spike expression',
trace_backend='torch', actual_input_files='created only after external registration SHA recorded'),
scope=dict(epoch_boundary_updates=True, within_trial_updates=False, full_conditioning=False,
candidate_generated=False, learning_success_claimed=False, neural_state_carried=False,
delta_carried=True, independent_model_branch=True),
success_gate='12 complete calls; exact reference/light events and final state/queues; exact D0/frozen identity; real D1 delta installed in epoch 2; nonzero D1 update coverage required',
missing_nonzero_coverage='INCOMPLETE, do not choose another input',
failure_policy='preserve all partials; no automatic run retry or result-selected seed',
retention_supported=False, future_hashes_fabricated=False, new_brain_runs_during_preparation=0)
verify_pins({**pins, **sources, **derived})
save(EXP / 'protocol/preregistered.json', reg)
return digest(EXP / 'protocol/preregistered.json')
def read_registration(expected):
path = EXP / 'protocol/preregistered.json'
require(P.sha(expected) and digest(path) == expected, 'Registration differs from external anchor')
reg = read(path)
require(reg['schema'] == 'phase4ao-conditioning-interface-registration-v1' and
reg['AN_registration_sha256'] == AN_REG and reg['budget'] == BUDGET and
reg['plan']['total_calls'] == 12 and reg['plan']['total_ticks'] == 14400,
'Unexpected finite registration')
verify_pins({**reg['source_sha256'], **reg['input_sha256'], **reg['prepared_file_sha256']})
return reg
if __name__ == '__main__':
require(sys.argv[1:] == ['prepare'], 'Only exclusive prepare is supported')
print('REGISTRATION_SHA256', prepare(), flush=True)
@@ -0,0 +1,474 @@
"""Finite AO runner: exact frozen brain, two fresh epochs, delta-only carry.
Never starts the full conditioning protocol. Partial rows are preserved. Each
runtime is sealed immediately; coordinator evidence lives outside its seal.
"""
from __future__ import annotations
import argparse
import ast
from datetime import datetime, timezone
import json
import os
from pathlib import Path
import shutil
import subprocess
import sys
import time
import traceback
from types import SimpleNamespace
import numpy as np
import conditioning_preflight_prepare as P
import conditioning_preflight_inputs as I
import conditioning_inputs as IN
import conditioning_update as U
import conditioning_epoch as E
import conditioning_preflight_compare as Q
import engineering_compare as COMP
import engineering_workflow as ENV
import runtime_light as R
import runtime_continuation as C
EXP, MODEL, ROOT = P.EXP, P.MODEL, P.ROOT
require, read, save, digest = P.require, P.read, P.save, P.digest
def npz(path):
with np.load(path, allow_pickle=False) as z:
return {k: z[k].copy() for k in z.files}
def row_by(reg, name):
matches = [r for r in reg['plan']['calls'] if r['row_id'] == name]
require(len(matches) == 1, 'Only an exact registered row may execute')
return matches[0]
def locations(name):
require(type(name) is str and name and all(c.isalnum() or c in '-_' for c in name), 'Unsafe row')
return dict(run=EXP / 'runs' / name, update=EXP / 'updates' / name,
runtime=EXP / 'receipts' / (name + '.runtime.json'), execution=EXP / 'receipts' / (name + '.process.json'),
authority=EXP / 'receipts' / (name + '.authority.json'),
stdout=EXP / 'logs' / (name + '.stdout.log'), stderr=EXP / 'logs' / (name + '.stderr.log'))
def budget(reg):
files = list(EXP.rglob('*'))
require(not any(p.is_symlink() for p in files), 'Linked output is forbidden')
used = sum(p.stat().st_size for p in files if p.is_file())
free = shutil.disk_usage(EXP).free
b = reg['budget']
require(used <= b['output_budget_bytes'] and free >= b['minimum_free_reserve_bytes'] + max(0, b['output_budget_bytes'] - used),
'Resource cap/reserve exceeded; preserve partials')
return dict(used_bytes=used, free_bytes=free, **b)
def tables():
metadata = read(EXP / 'inputs/static/rate-tables.json')
return {cue: dict(table, **npz(EXP / f'inputs/static/{cue}-rates.npz')) for cue, table in metadata.items()}
def original_sampler_nodes(raw):
"""Only the four original L sampling statements, no old brain import."""
require(__import__('hashlib').sha256(raw).hexdigest() == I.SOURCE_PINS[I.SOURCE_REL], 'L source changed')
methods = [n for n in ast.walk(ast.parse(raw)) if isinstance(n, ast.FunctionDef) and n.name == 'run_rates']
require(len(methods) == 1, 'Original sampling method is not unique')
chosen = []
for node in methods[0].body:
if isinstance(node, ast.Assign) and any(isinstance(t, ast.Name) and t.id in ('rg', 'gen', 'forced') for t in node.targets):
chosen.append(node)
elif isinstance(node, ast.Expr) and isinstance(node.value, ast.Call) and ast.unparse(node.value.func) == 'gen.manual_seed':
chosen.append(node)
require(len(chosen) == 4 and [ast.unparse(n).split('=')[0].strip() for n in chosen[:2]] == ['rg', 'gen'],
'Original sampler AST structure changed')
return chosen
def sampler_check(reg, table, generated):
import torch
seed = reg['generation']['highbit_probe_seed']
nodes = original_sampler_nodes((ROOT / I.SOURCE_REL).read_bytes())
env = dict(torch=torch, targets=table['targets'], rates_hz=table['rates_hz'], seed=seed, steps=1200,
self=SimpleNamespace(dt=.1, device=torch.device('cuda:0')))
exec(compile(ast.Module(body=nodes, type_ignores=[]), str(ROOT / I.SOURCE_REL), 'exec'), env)
original = IN.requests_from_forced(env['forced'].cpu().numpy(), table['targets'], cs_start_tick=0, cs_stop_tick=1000)
require(all(COMP.bit_equal(original['requests'][k], generated['cs_requests'][k]) for k in ('tick', 'neuron_idx')),
'Actual CUDA original-L sampling expression disagrees')
identities = []
for value in (seed, seed & 0xffffffff):
gen = torch.Generator(device='cuda:0').manual_seed(value)
matrix = torch.rand((1200, len(table['targets'])), generator=gen, device='cuda:0')
identities.append(IN.array_identity(matrix.cpu().numpy()))
require(identities[0] != identities[1], 'Actual CUDA full63bit and low32bit sequence identical; stop, no seed substitution')
return dict(original_L_sampler_AST_sha256=__import__('hashlib').sha256(ast.dump(ast.Module(body=nodes, type_ignores=[]), include_attributes=False).encode()).hexdigest(),
exact_sparse_CS_requests=True, full_seed=seed, low32_probe_seed=seed & 0xffffffff,
uniforms_identities=identities, highbit_influence_observed=True,
additional_probe_matrices=3, probe_shape=[1200, len(table['targets'])],
new_input_seed_selected=False, old_full_trial_equivalence_claimed=False)
def materialize(reg_sha):
reg = P.read_registration(reg_sha)
admission()
ENV.preflight(); budget(reg)
target = EXP / 'inputs/generated'
require(not target.exists(), 'Never rerun or overwrite actual input generation')
target.mkdir()
rate = tables()
dan = npz(EXP / 'inputs/static/populations.npz')[reg['plan']['dan_type']]
records = []
for index, epoch in enumerate(reg['plan']['selected_epochs']):
started = time.perf_counter()
value = I.generate_epoch_inputs(epoch, rate, dan, device=reg['generation']['device'],
max_uniform_draws=reg['budget']['max_uniform_draws'], conflict_policy=reg['generation']['conflict_policy'])
directory = target / f'epoch-{index:03d}'
directory.mkdir()
for kind in ('requests', 'cs_requests', 'us_requests'):
P.save_npz(directory / (kind + '.npz'), **value[kind])
P.save_npz(directory / 'source-mask.npz', source_mask=value['source_mask'])
save(directory / 'sampler-receipt.json', value['audit'])
if index == 0:
save(target / 'actual-CUDA-sampler-check.json', sampler_check(reg, rate[epoch['cs']['cue']], value))
records.append(dict(index=index, epoch_id=epoch['epoch_id'], wall_seconds=time.perf_counter() - started,
CS_requests=len(value['cs_requests']['tick']), US_requests=len(value['us_requests']['tick'])))
for row in reg['plan']['calls']:
src = target / f"epoch-{row['input_epoch_index']:03d}" / 'requests.npz'
dst = Path(reg['cases'][row['row_id']]) / 'requests.npz'
with src.open('rb') as inp, dst.open('xb') as out:
shutil.copyfileobj(inp, out)
require(digest(src) == digest(dst), 'Shared input copy differs')
pins = {str(p): digest(p) for p in target.rglob('*') if p.is_file()}
pins.update({str(Path(path) / 'requests.npz'): digest(Path(path) / 'requests.npz') for path in reg['cases'].values()})
P.read_registration(reg_sha)
save(EXP / 'receipts/input-generation.json', dict(complete=True, registration_sha256=reg_sha,
inputs_sha256=pins, rows=records, GPU_UUID=ENV.GPU, full63bit_delivery_and_influence_checked=True,
input_generation_calls=2, brain_runs=0, parameters_changed=False))
def generated_inputs(reg_sha):
receipt = read(EXP / 'receipts/input-generation.json')
require(receipt.get('complete') is True and receipt.get('registration_sha256') == reg_sha and receipt.get('brain_runs') == 0,
'Generated inputs are not bound to registration')
process = read(EXP / 'receipts/input-generation.process.json')
require(process['exit_code'] == 0 and process['registration_sha256'] == reg_sha and
process['receipt_sha256'] == digest(EXP / 'receipts/input-generation.json'), 'Input generation process missing')
for k in ('stdout', 'stderr'):
require(process[k + '_sha256'] == digest(EXP / 'logs' / ('input-generation.' + k + '.log')), 'Input generation log changed')
P.verify_pins(receipt['inputs_sha256'])
return receipt
def verify_row(reg, reg_sha, row):
paths = locations(row['row_id'])
record = read(paths['runtime']); process = read(paths['execution'])
require(record['complete'] is True and record['registration_sha256'] == reg_sha and P.same(record['row'], row),
'Runtime record differs from registered row')
require(process['exit_code'] == 0 and process['registration_sha256'] == reg_sha and
process['runtime_sha256'] == digest(paths['runtime']), 'Runtime process did not complete exactly')
for key in ('stdout', 'stderr'):
require(process[key + '_sha256'] == digest(paths[key]), 'Process log changed')
P.verify_pins(record['files_sha256'])
checkpoint = C.load_checkpoint(paths['run'], record['checkpoint_manifest_sha256'])
require(checkpoint['fresh'] and checkpoint['steps'] == row['steps'] and checkpoint['global_tick_offset'] == 0,
'Normal epoch must be fresh, not a continuation')
ENV.verify_environment(record['environment'], checkpoint['summary']['apl_component']['environment'])
chain = read(paths['update'] / 'chain-receipt.json')
require(chain['authority_sha256'] == digest(paths['authority']) and
chain['source_neural_checkpoint_manifest_sha256'] == record['checkpoint_manifest_sha256'] and
chain['epoch_id'] == row['epoch_id'], 'Delta does not derive from this actual sealed neural epoch')
P.verify_pins(chain['input_files_sha256'])
for name, sha in chain['output_files_sha256'].items():
require(digest(paths['update'] / name) == sha, 'Updated snapshot changed')
return record
def before_overlay(reg, reg_sha, row):
previous = row['previous_delta_row_id']
if previous is None:
return reg['initial_overlay']
parent = row_by(reg, previous)
require(parent['arm'] == row['arm'] and parent['diagnostic_mode'] == row['diagnostic_mode'] and
parent['input_epoch_index'] + 1 == row['input_epoch_index'], 'Invalid cross-epoch delta link')
verify_row(reg, reg_sha, parent)
directory = locations(previous)['update']
return dict(overlay_file=str(directory / 'delta-after.npz'), overlay_manifest=str(directory / 'overlay-manifest.json'))
def make_authority(reg, row, checkpoint_sha, overlay):
paths = locations(row['row_id']); case = Path(reg['cases'][row['row_id']]); spec = read(case / 'spec.json')
metadata = read(overlay['overlay_manifest'])
role_paths = dict(spec=case / 'spec.json', requests=case / 'requests.npz',
overlay_before=overlay['overlay_file'], overlay_manifest_before=overlay['overlay_manifest'],
state=paths['run'] / 'state.npz', kc=Path(spec['apl_bundle_dir']) / 'kc_globals.npy',
canonical_kc=Path(spec['apl_bundle_dir']) / 'kc_globals.npy', candidate_map=EXP / 'inputs/static/candidate-map.npz',
mask=metadata['mask_path'], base_candidate=metadata['base_candidate_path'],
phase4j_source=reg['phase4j_source'], eligibility_bridge_source=reg['bridge_source'],
conditioning_update_source=MODEL / 'conditioning_update.py', runtime_light_source=MODEL / 'runtime_light.py',
runtime_continuation_source=MODEL / 'runtime_continuation.py', am_overlay_source=metadata['am_overlay_path'])
epoch = reg['plan']['selected_epochs'][row['input_epoch_index']]
return dict(schema=E.SCHEMA, epoch_id=row['epoch_id'], diagnostic_mode=row['diagnostic_mode'],
command=dict(steps=row['steps'], compartment=reg['plan']['compartment_verbatim'],
us_tick=epoch['us']['tick'], d_command=row['d_command'], frozen=row['frozen'], trace_backend=reg['generation']['trace_backend']),
checkpoint_dir=str(paths['run']), checkpoint_manifest_sha256=checkpoint_sha,
files={role: dict(path=str(path), sha256=digest(path)) for role, path in role_paths.items()},
frozen_inputs={k: v for k, v in reg['input_sha256'].items()
if k not in (str(P.AN / 'protocol/preregistered.json'), str(P.AN / 'manifest.json'), str(P.AN / 'audit/final-status.json'))})
def replay_update(authority, receipt):
files = {k: Path(v['path']) for k, v in authority['files'].items()}
state = npz(files['state']); mapping = npz(files['candidate_map']); old = npz(files['overlay_before'])
command = authority['command']
return U.verify_receipt(receipt, step=state['step'], neuron_idx=state['neuron_idx'],
kc=np.load(files['kc'], allow_pickle=False), candidate_edge_id=mapping['edge_id'],
candidate_sources=mapping['source_neuron_idx'], candidate_targets=mapping['target_neuron_idx'],
candidate_compartments=mapping['target_compartment_verbatim'], delta_before=old['plastic_delta_mv'],
base_candidate=np.load(files['base_candidate'], allow_pickle=False), steps=command['steps'], n_neurons=166700,
compartment=command['compartment'], us_tick=command['us_tick'], d_command=command['d_command'],
frozen=command['frozen'], trace_backend=command['trace_backend'],
bridge_source=files['eligibility_bridge_source'], phase4j_source=files['phase4j_source'])
def worker(reg_sha, row_id):
reg = P.read_registration(reg_sha); generated_inputs(reg_sha)
admission()
row = row_by(reg, row_id); paths = locations(row_id)
require(not any(paths[k].exists() for k in ('run', 'update', 'runtime', 'authority')), 'Partial row cannot be retried')
ENV.preflight(); before = budget(reg)
overlay = before_overlay(reg, reg_sha, row)
case = Path(reg['cases'][row_id])
protected = {Path(p).parent for p in (*reg['input_sha256'], *reg['source_sha256'], *reg['prepared_file_sha256'])}
protected.update({EXP / 'inputs', EXP / 'updates'})
# Build/run siblings must never touch any current or earlier learned input.
R.execution_paths(case, paths['run'], EXP / 'extension-build', protected=protected)
started = time.perf_counter(); started_utc = datetime.now(timezone.utc).isoformat()
execute = R.run_reference if row['diagnostic_mode'] == 'reference' else R.run_light
execute(case, paths['run'], EXP / 'extension-build', overlay['overlay_file'], overlay['overlay_manifest'])
checkpoint_sha = C.seal_checkpoint(paths['run'])['sha256']
runtime_seconds = time.perf_counter() - started
authority = make_authority(reg, row, checkpoint_sha, overlay)
save(paths['authority'], authority)
update_started = time.perf_counter()
E.postprocess_epoch(paths['authority'], digest(paths['authority']), paths['update'])
receipt = read(paths['update'] / 'update-receipt.json')
replay_update(authority, receipt)
delta_before = npz(overlay['overlay_file'])['plastic_delta_mv']
_, after, _, _ = R.load_overlay(paths['update'] / 'delta-after.npz', paths['update'] / 'overlay-manifest.json')
binding = Q.audit_delta_binding(delta_before, after['plastic_delta_mv'], after['plastic_delta_mv'],
update_receipt=receipt, parent_snapshot_sha256=digest(overlay['overlay_file']),
child_manifest=read(paths['update'] / 'overlay-manifest.json'))
require(binding['pass_all'], 'Actual recomputed delta/child binding failed')
save(EXP / 'receipts' / (row_id + '.binding.json'), binding)
record = dict(complete=True, registration_sha256=reg_sha, row=row,
checkpoint_manifest_sha256=checkpoint_sha, environment=ENV.runtime_environment(),
runtime_and_immediate_seal_wall_seconds=runtime_seconds,
postprocess_and_recompute_wall_seconds=time.perf_counter() - update_started,
started_utc=started_utc, budget_before=before, budget_after=budget(reg),
files_sha256={str(p): digest(p) for p in (paths['authority'], *paths['update'].iterdir(),
EXP / 'receipts' / (row_id + '.binding.json'))},
before_overlay_file_sha256=digest(overlay['overlay_file']), epoch_update_recomputed=True,
learning_success_claimed=False, full_conditioning_executed=False)
ENV.verify_environment(record['environment'], read(paths['run'] / 'summary.json')['apl_component']['environment'])
P.read_registration(reg_sha)
save(paths['runtime'], record)
print('COMPLETE', row_id, receipt['counts']['actual_bitwise_changed_edges'], flush=True)
def child_process(reg, reg_sha, action, row_id=None):
if row_id is None:
out, err = [EXP / 'logs' / ('input-generation.' + k + '.log') for k in ('stdout', 'stderr')]
else:
p = locations(row_id); out, err = p['stdout'], p['stderr']
require(not out.exists() and not err.exists(), 'Partial child logs exist; never retry')
command = [sys.executable, '-B', str(Path(__file__)), action, '--registration-sha256', reg_sha]
if row_id is not None: command += ['--row-id', row_id]
started = time.perf_counter()
with out.open('xb') as stdout, err.open('xb') as stderr:
result = subprocess.run(command, cwd=MODEL, stdout=stdout, stderr=stderr,
timeout=reg['budget']['child_timeout_seconds'])
require(result.returncode == 0, 'Child failed; preserve partial files and inspect logs')
receipt = dict(exit_code=result.returncode, registration_sha256=reg_sha,
stdout_sha256=digest(out), stderr_sha256=digest(err), wall_seconds=time.perf_counter() - started)
if row_id is None:
receipt['receipt_sha256'] = digest(EXP / 'receipts/input-generation.json')
save(EXP / 'receipts/input-generation.process.json', receipt)
else:
receipt['runtime_sha256'] = digest(locations(row_id)['runtime'])
save(locations(row_id)['execution'], receipt)
def run(reg_sha):
reg = P.read_registration(reg_sha)
admission()
completed = []
try:
child_process(reg, reg_sha, 'materialize')
generated_inputs(reg_sha)
for row in reg['plan']['calls']:
print('START', row['row_id'], flush=True)
budget(reg)
child_process(reg, reg_sha, 'worker', row['row_id'])
verify_row(reg, reg_sha, row)
completed.append(row['row_id'])
print('COMPLETE', row['row_id'], flush=True)
save(EXP / 'audit/execution-complete.json', dict(registration_sha256=reg_sha, complete=True,
completed=completed, calls=12, ticks=14400, learning_success_claimed=False))
except Exception as error:
save(EXP / 'audit/stopped.json', dict(complete=False, registration_sha256=reg_sha,
error_type=type(error).__name__, error=str(error), traceback=traceback.format_exc(),
completed_rows=completed, partials_preserved=True, no_automatic_retry=True))
raise
def admission():
require(not (EXP / 'audit/stopped.json').exists() and not (EXP / 'audit/execution-complete.json').exists() and
not (EXP / 'manifest.json').exists(), 'Stopped/completed protocol cannot be automatically restarted')
def expected_weight_hash(reg, reg_sha, row):
"""Independent CPU expected installation from authorized base and delta."""
overlay = before_overlay(reg, reg_sha, row)
helper, plan, _, _ = R.load_overlay(overlay['overlay_file'], overlay['overlay_manifest'])
spec = read(Path(reg['cases'][row['row_id']]) / 'spec.json')
graph = Path(spec['graph_dir']); arrays = {}
try:
for name in ('nodes', 'weights', 'edge_id', 'src'):
arrays[name] = np.load(graph / (name + '.npy'), mmap_mode='r', allow_pickle=False)
require(np.array_equal(arrays['nodes'], np.arange(166700, dtype=np.int32)), 'Global graph node identity changed')
apl = np.load(Path(spec['apl_bundle_dir']) / 'apl_globals.npy', allow_pickle=False)
mask = np.isin(arrays['src'], apl)
require(int(mask.sum()) == 5845, 'Original fast APL mask differs')
mapping = helper.map_installation(arrays['weights'], arrays['edge_id'], mask, plan)
require(mapping['missing_candidate_count'] == 0, 'Full candidate mask not installed')
base_sha, effective_sha = helper.expected_hashes(arrays['weights'], mask, mapping)
return dict(base_raw_sha256=base_sha, effective_raw_sha256=effective_sha,
expected_from_base_and_overlay_not_run_summary=True)
finally:
for a in arrays.values(): a._mmap.close()
def analyze(reg_sha):
reg = P.read_registration(reg_sha); generated_inputs(reg_sha)
require(not (EXP / 'audit/stopped.json').exists(), 'Stopped protocol has no PASS analysis')
complete = read(EXP / 'audit/execution-complete.json')
require(complete['complete'] and complete['registration_sha256'] == reg_sha and
complete['completed'] == [r['row_id'] for r in reg['plan']['calls']], 'Missing finite calls')
records = {r['row_id']: verify_row(reg, reg_sha, r) for r in reg['plan']['calls']}
pairs, summaries, deltas, cross_arm = [], [], {}, []
for row in reg['plan']['calls']:
paths = locations(row['row_id']); authority = read(paths['authority'])
require(P.same(authority, make_authority(reg, row, records[row['row_id']]['checkpoint_manifest_sha256'],
before_overlay(reg, reg_sha, row))), 'Authority differs from registered row and actual parent')
update = read(paths['update'] / 'update-receipt.json')
replay_update(authority, update)
delta = npz(paths['update'] / 'delta-after.npz')['plastic_delta_mv']
deltas[row['row_id']] = delta
if row['arm'] in ('D0', 'frozen'):
require(not np.any(delta.view(np.uint32)), 'D0/frozen must remain exact positive zero')
summary = read(paths['run'] / 'summary.json')
population = npz(EXP / 'inputs/static/populations.npz')
state = npz(paths['run'] / 'state.npz')
run_record = COMP.load_run(paths['run'])
try:
us = npz(EXP / f"inputs/generated/epoch-{row['input_epoch_index']:03d}/us_requests.npz")
us_keys = us['tick'].astype(np.int64) * 166700 + us['neuron_idx']
request_keys = run_record['requests'][:, 0] * 166700 + run_record['requests'][:, 1]
dan_requests = run_record['requests'][np.isin(request_keys, us_keys)]
require(len(dan_requests) == len(us_keys), 'Scheduled US requests not fully accounted')
delivery = dict(scheduled_D=row['d_command'], requested=int(len(dan_requests)),
accepted=int(dan_requests[:, 3].sum()), rejected=int(dan_requests[:, 4].sum()),
actual_DAN_spikes=int(state['counts'][population[reg['plan']['dan_type']]].sum()),
actual_DAN_spikes_gate_update=False, artificial_US_not_dopamine_concentration=True)
finally:
COMP.close_run(run_record)
summaries.append(dict(row_id=row['row_id'], total_spikes=summary['total_spikes'],
active_neurons=int(np.count_nonzero(state['counts'])), all_finite=summary['all_finite'],
update_counts=update['counts'], DAN_delivery=delivery,
runtime_seconds=records[row['row_id']]['runtime_and_immediate_seal_wall_seconds'],
effective_weight_sha256=summary['effective_weights_raw_sha256']))
for arm in ('D1', 'D0', 'frozen'):
for epoch in range(2):
names = [f'{arm}--{mode}--epoch-{epoch:03d}' for mode in ('reference', 'light')]
a, b = [COMP.load_run(locations(n)['run']) for n in names]
try:
requests = npz(EXP / f'inputs/generated/epoch-{epoch:03d}/requests.npz')
expected = expected_weight_hash(reg, reg_sha, row_by(reg, names[0]))
pair = Q.compare_epoch_pair(a, b, expected_requests=np.column_stack((requests['tick'], requests['neuron_idx'])),
expected_steps=1200, expected_effective_weight_sha256=expected['effective_raw_sha256'])
require(pair['pass_all'] and COMP.bit_equal(deltas[names[0]], deltas[names[1]]), 'Reference/light exact regression failed')
pairs.append(dict(arm=arm, epoch=epoch, exact_delta_after=True, expected_weights=expected, comparison=pair))
finally:
COMP.close_run(a); COMP.close_run(b)
for mode in ('reference', 'light'):
for left_arm, right_arm, epoch in (('D0', 'frozen', 0), ('D0', 'frozen', 1), ('D1', 'D0', 0)):
left = COMP.load_run(locations(f'{left_arm}--{mode}--epoch-{epoch:03d}')['run'])
right = COMP.load_run(locations(f'{right_arm}--{mode}--epoch-{epoch:03d}')['run'])
try:
comparison = COMP.compare_fresh(left, right)
require(comparison['numerical_exact'], 'Identical-input/zero-delta cross-arm neural behavior differs')
cross_arm.append(dict(left=left_arm, right=right_arm, mode=mode, epoch=epoch, comparison=comparison))
finally:
COMP.close_run(left); COMP.close_run(right)
d1_rows = [s for s in summaries if s['row_id'].startswith('D1--')]
nonzero = all(s['update_counts']['actual_bitwise_changed_edges'] > 0 for s in d1_rows)
installed = {}
for mode in ('reference', 'light'):
first, second = (locations(f'D1--{mode}--epoch-{j:03d}') for j in range(2))
raw = R._raw(deltas[f'D1--{mode}--epoch-000'])
installation = read(second['run'] / 'overlay-audit.json')
require(installation['delta_raw_sha256'] == raw and
digest(first['update'] / 'delta-after.npz') == records[f'D1--{mode}--epoch-001']['before_overlay_file_sha256'],
'Epoch 2 did not install epoch 1 learned delta')
installed[mode] = dict(exact=True, nonzero_delta=int(np.count_nonzero(deltas[f'D1--{mode}--epoch-000'])))
status = 'PASS-EXACT-TWO-EPOCH-DELTA-INTERFACE-NOT-LEARNING' if nonzero else 'INCOMPLETE-NO-NONZERO-UPDATE-COVERAGE'
return dict(status=status, registration_sha256=reg_sha, calls=12, ticks=14400, pairs=pairs, cross_arm=cross_arm,
rows=summaries, epoch2_installations=installed, nonzero_D1_update_coverage=nonzero,
D0_and_frozen_exact_zero=True, complete_conditioning_executed=False, learned_candidate=False,
learning_success_claimed=False, retention_transition_supported=False,
scope='Two selected A epochs, one training replicate, one PAM15 compartment; no acquisition or A/B behavioral test; no general dynamics or learning claim',
light_intermediate_states_observed=False)
def finalize(reg_sha):
require(not (EXP / 'manifest.json').exists() and not (EXP / 'audit/final-status.json').exists(), 'Do not overwrite final/partial seal')
report = analyze(reg_sha)
save(EXP / 'audit/interface-analysis.json', report)
an = P.P.read_registration(P.AN_REG)
helper = P.P.load_module('phase4ao_final_seal_helper', ROOT / P.P.AI_HELPER_REL, P.P.AI_HELPER_SHA)
save(EXP / 'regression/AN-seal-after.json', helper.verify_seal(P.AN))
save(EXP / 'regression/AM-seals-after.json', P.P.verify_authority(an['anchors']))
old = P.P.verify_old_integrity(an)
save(EXP / 'regression/old-integrity-after.json', old)
P.read_registration(reg_sha)
final = dict(status=report['status'], analysis_sha256=digest(EXP / 'audit/interface-analysis.json'),
registration_sha256=reg_sha, old_integrity_pass=True, old_roots=len(old),
prior_AN_AM_seals_preserved=True, frozen_21_and_previous_draft_20_unchanged=True,
five_reference_trials='Previously sealed AM evidence retained, not rerun in this finite interface gate',
learned_candidate=False, learning_success_claimed=False, full_conditioning_executed=False,
model_directory_sealed=False, new_experiment_only=True, budget=budget(P.read_registration(reg_sha)))
save(EXP / 'audit/final-status.json', final)
paths = sorted(p for p in EXP.rglob('*') if p.is_file())
require(not any(p.is_symlink() for p in EXP.rglob('*')) and not any('__pycache__' in p.parts for p in paths), 'Unsafe seal payload')
rows = [dict(path=p.relative_to(EXP).as_posix(), bytes=p.stat().st_size, sha256=digest(p)) for p in paths]
save(EXP / 'manifest.json', dict(root=str(EXP), files=rows))
with (EXP / 'checksums.sha256').open('x', encoding='utf-8', newline='\n') as out:
out.writelines(r['sha256'] + ' ' + r['path'] + '\n' for r in rows)
out.write(digest(EXP / 'manifest.json') + ' manifest.json\n')
helper.verify_seal(EXP)
print(json.dumps(dict(status=report['status'], manifest_sha256=digest(EXP / 'manifest.json'),
final_sha256=digest(EXP / 'audit/final-status.json')), sort_keys=True), flush=True)
if __name__ == '__main__':
require(sys.dont_write_bytecode, 'Use scientific Python -B')
parser = argparse.ArgumentParser()
parser.add_argument('action', choices=('materialize', 'worker', 'run', 'finalize'))
parser.add_argument('--registration-sha256', required=True)
parser.add_argument('--row-id')
args = parser.parse_args()
if args.action == 'worker': worker(args.registration_sha256, args.row_id)
elif args.action == 'materialize': materialize(args.registration_sha256)
elif args.action == 'run': run(args.registration_sha256)
else: finalize(args.registration_sha256)
@@ -0,0 +1,205 @@
"""Pure epoch-END adapter to the frozen AM/J eligibility implementation.
Only the two SHA-pinned source files are read. No event files, brain state,
weights, base arrays or artifacts are written. The caller supplies actual sparse
events and externally authenticated COMPLETE KC/mask mappings. Hashing supplied
arrays identifies their bytes; it does not establish anatomical completeness.
The returned overlay is NOT installed. Scheduled D is not actual DAN delivery.
"""
from __future__ import annotations
from dataclasses import dataclass
import hashlib
import json
import math
from pathlib import Path
import sys
from types import ModuleType
import numpy as np
SCHEMA = "phase4an-conditioning-update-draft-v1"
BRIDGE_SHA256 = "174fb8d14619189e74b095e77a6db18b14c93046dc431c0b55710a5aa2dd2e61"
PHASE4J_SHA256 = "abb4574da85f00bd9ef61024c65993ef81d42651ff8d8d63c15dd69c6222471e"
def _require(condition, message):
if not condition:
raise ValueError(message)
def _canonical(value):
return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False, allow_nan=False)
def _sha(value):
return hashlib.sha256(_canonical(value).encode("utf-8")).hexdigest()
def _array(value, dtype, name):
_require(isinstance(value, np.ndarray) and value.ndim == 1 and value.dtype == np.dtype(dtype),
name + " requires an exact one-dimensional " + str(np.dtype(dtype)) + " array")
return value
def _evidence(array):
"""Original dtype and logical C-order raw bytes; no cast or rounding."""
return dict(dtype=array.dtype.str, shape=list(array.shape), nbytes=int(array.nbytes),
raw_c_order_sha256=hashlib.sha256(array.tobytes(order="C")).hexdigest())
def _owned(array):
result = np.array(array, copy=True, order="C")
result.setflags(write=False)
return result
def _pinned_bytes(path, expected, label):
raw = Path(path).read_bytes()
_require(hashlib.sha256(raw).hexdigest() == expected, label + " source SHA256 mismatch")
return raw
def _load_bridge(path):
raw = _pinned_bytes(path, BRIDGE_SHA256, "AM eligibility bridge")
# Execute precisely the verified bytes, not a second path-based import.
# Temporary isolated module registration supports the frozen dataclass;
# restore any prior module, rather than colliding with old globals.
name = "_conditioning_update_bridge_" + BRIDGE_SHA256
module = ModuleType(name)
module.__file__ = str(path)
previous = sys.modules.get(name)
sys.modules[name] = module
try:
exec(compile(raw, str(path), "exec"), module.__dict__)
finally:
if previous is None:
del sys.modules[name]
else:
sys.modules[name] = previous
_require(module.PHASE4J_SHA256 == PHASE4J_SHA256, "Frozen bridge/J pin disagreement")
return module
@dataclass(frozen=True)
class EpochUpdate:
delta_after: np.ndarray
sampled_trace: np.ndarray
final_trace: np.ndarray
receipt: dict
def update_epoch(*, step, neuron_idx, kc, candidate_edge_id, candidate_sources,
candidate_targets, candidate_compartments, delta_before, base_candidate,
steps, n_neurons, compartment, us_tick, d_command, frozen,
bridge_source, phase4j_source, trace_backend):
"""Return the epoch-boundary update; never apply it to the running brain.
trace_backend is mandatory: ``torch`` executes the frozen Torch CPU trace
expression; ``numpy`` remains its candidate compatibility implementation,
without a claim of per-call Torch equivalence. One declared US or no US is
supported. A US-less epoch requires D=0. Actual DAN delivery is deliberately
absent from this API and must be bound as separate coordinator evidence.
"""
_require(type(steps) is int and steps > 0, "steps must be a positive integer")
_require(type(n_neurons) is int and n_neurons > 0, "n_neurons must be a positive integer")
_require(us_tick is None or (type(us_tick) is int and 0 <= us_tick < steps), "us_tick must be None or an integer inside the epoch")
_require(type(d_command) in (int, float) and math.isfinite(d_command) and d_command in (0, 1),
"D must be the explicit scheduled scalar 0 or 1, not Boolean or measured DAN")
_require(us_tick is not None or d_command == 0, "No declared US tick requires D=0")
_require(type(frozen) is bool, "frozen must be Boolean")
_require(trace_backend in ("numpy", "torch"), "Explicit trace_backend must be numpy or torch")
arrays = dict(step=_array(step, "int32", "step"), neuron_idx=_array(neuron_idx, "int32", "neuron_idx"),
kc=_array(kc, "int32", "kc"), candidate_edge_id=_array(candidate_edge_id, "int64", "candidate_edge_id"),
candidate_sources=_array(candidate_sources, "int32", "candidate_sources"),
candidate_targets=_array(candidate_targets, "int32", "candidate_targets"),
delta_before=_array(delta_before, "float32", "delta_before"),
base_candidate=_array(base_candidate, "float32", "base_candidate"))
_require(isinstance(candidate_compartments, np.ndarray) and candidate_compartments.ndim == 1 and
candidate_compartments.dtype.kind == "U", "candidate_compartments must be a Unicode vector")
arrays["candidate_compartments"] = candidate_compartments
size = len(candidate_edge_id)
_require(size > 0 and np.all(candidate_edge_id >= 0) and np.all(candidate_edge_id[1:] > candidate_edge_id[:-1]),
"Candidate edge IDs must be nonempty, canonical sorted unique int64 IDs")
_require(all(len(arrays[name]) == size for name in ("candidate_sources", "candidate_targets",
"candidate_compartments", "delta_before", "base_candidate")), "Candidate mapping columns differ in length")
_require(np.all((candidate_targets >= 0) & (candidate_targets < n_neurons)), "Candidate target outside supplied graph size")
before_evidence = {name: _evidence(value) for name, value in arrays.items()}
# Every computation receives owned copies. Caller views/strides are accepted
# without changing their values or flags; dtype is never silently repaired.
work = {name: np.array(value, copy=True, order="C") for name, value in arrays.items()}
bridge = _load_bridge(bridge_source)
_pinned_bytes(phase4j_source, PHASE4J_SHA256, "Phase 4J")
trace = bridge.trace_from_events(work["step"], work["neuron_idx"], work["kc"], steps,
[] if us_tick is None else [us_tick], phase4j_source=phase4j_source,
backend=trace_backend, n_neurons=n_neurons)
sampled = trace.last_command_trace
after, original = bridge.apply_eligibility(sampled, work["kc"], work["candidate_sources"],
work["candidate_compartments"], compartment, work["delta_before"], work["base_candidate"],
d=d_command, frozen=frozen, phase4j_source=phase4j_source, n_neurons=n_neurons)
# Diagnostics only; the update is exclusively the frozen apply_v2 above.
selected = work["candidate_compartments"] == compartment
local = np.searchsorted(work["kc"], work["candidate_sources"])
eligible = (sampled[local] > 0) & selected
changed = after.view(np.uint32) != work["delta_before"].view(np.uint32)
value_changed = after != work["delta_before"]
floor = -bridge.FLOOR_FRACTION * work["base_candidate"]
floor_before = work["delta_before"] == floor
floor_after = after == floor
_require(bridge.bit_equal(after[~selected], work["delta_before"][~selected]), "Off-compartment identity failed")
if frozen or d_command == 0:
_require(bridge.bit_equal(after, work["delta_before"]), "Zero-D/frozen identity failed")
original_report = original["original_report"]
attempted = 0 if original_report is None else original_report["updated_edges"]
if original_report is not None:
_require(original_report["eligible_edges"] == int(eligible.sum()), "Frozen eligible-edge report disagrees with its sampled trace")
delta_after, sampled_trace, final_trace = (_owned(a) for a in (after, sampled, trace.final))
_require(all(_evidence(value) == before_evidence[name] for name, value in arrays.items()), "Caller input bytes changed")
_pinned_bytes(bridge_source, BRIDGE_SHA256, "AM eligibility bridge")
_pinned_bytes(phase4j_source, PHASE4J_SHA256, "Phase 4J")
receipt = dict(schema=SCHEMA, status="DRAFT-COMPUTED-EPOCH-UPDATE-NOT-INSTALLED",
source_pins=dict(eligibility_bridge_sha256=BRIDGE_SHA256, phase4j_sha256=PHASE4J_SHA256),
input_arrays=before_evidence,
command=dict(steps=steps, n_neurons=n_neurons, compartment=compartment, us_tick=us_tick,
D_scheduled=float(d_command), frozen=frozen, trace_backend=trace_backend),
semantics=dict(application="epoch_boundary_after_all_recorded_ticks; affects subsequent epoch only",
sampling="after_current_tick_spike", trace_initial_state="zero at epoch boundary",
no_US_sample="canonical float32 +0, not final trace", D_definition="artificial scheduled scalar",
actual_DAN_delivery_observed=False, actual_DAN_delivery_used_to_gate=False,
actual_DAN_evidence_authority="separate externally bound epoch record, not an input to this function",
original_base_units="mV", delta_units="mV", delta_dtype="float32",
candidate_target_role="identity evidence only; update uses original source/compartment mapping"),
fixed_rule=dict(dt_ms=bridge.DT_MS, trace_tau_ms=bridge.TAU_MS, trace_cap=bridge.TRACE_CAP,
eta_mv=bridge.ETA_MV, polarity=bridge.POLARITY, floor_fraction=bridge.FLOOR_FRACTION),
trace_audit=trace.audit, apply_audit=original,
counts=dict(events=len(step), actual_KC_events=trace.audit["actual_kc_events"],
candidates=size, selected_compartment_edges=int(selected.sum()), sampled_eligible_selected_edges=int(eligible.sum()),
original_updated_edges_attempted=int(attempted), original_apply_v2_executed=original["original_apply_v2_executed"],
actual_value_changed_edges=int(value_changed.sum()), actual_bitwise_changed_edges=int(changed.sum()),
actual_off_compartment_bitwise_changed_edges=int(changed[~selected].sum()),
selected_at_floor_before=int((selected & floor_before).sum()),
selected_at_floor_after=int((selected & floor_after).sum()),
selected_newly_at_floor=int((selected & floor_after & ~floor_before).sum()),
sampled_eligible_selected_already_at_floor=int((eligible & floor_before).sum())),
count_limitations="original updated_edges counts nonzero attempted increments, including saturation; floor counts do not prove why any edge was unchanged",
outputs={"delta_after": _evidence(delta_after), "sampled_trace": _evidence(sampled_trace), "final_trace": _evidence(final_trace)},
inputs_exact_unchanged=True, off_compartment_exact_unchanged=True,
numpy_torch_bitwise_comparison_performed=False,
frozen_Torch_expression_executed=trace_backend == "torch",
complete_KC_identity_verified=False, complete_mask_identity_verified=False, full_graph_authority_verified=False,
mapping_hash_scope="supplied logical array bytes only; coordinator must bind complete canonical KC and candidate mask",
brain_dynamics_executed=False, weights_installed=False, base_modified=False,
training_executed_by_this_function=False, scientific_learning_pass_claimed=False, candidate_generated=False)
receipt["payload_sha256"] = _sha(receipt)
return EpochUpdate(delta_after, sampled_trace, final_trace, receipt)
def verify_receipt(receipt, **epoch_inputs):
"""Recompute from supplied arrays/sources; never trust a self-hash alone."""
_require(type(receipt) is dict and "payload_sha256" in receipt, "Invalid epoch receipt")
_require(receipt["payload_sha256"] == _sha({k: v for k, v in receipt.items() if k != "payload_sha256"}),
"Epoch receipt payload SHA mismatch")
actual = update_epoch(**epoch_inputs)
_require(_canonical(receipt) == _canonical(actual.receipt), "Receipt differs from exact recomputation in this environment")
return True
@@ -0,0 +1,373 @@
"""Deterministic, read-only CPU analysis of the registered 32-call AN gate.
analyze(reg, reg_sha, verified_records) accepts the actual runtime receipts
already verified by engineering_workflow.verify_execution. It does not run
that workflow, publish, seal, save, retry or import a GPU runtime. All external
AM seals/checkpoint chains/process receipts remain the caller's prior duty.
This module rehashes the files it reads, executes every canonical descriptor,
and independently reconstructs the reference queues and effective SI weights.
"""
from __future__ import annotations
import hashlib
import json
import math
from pathlib import Path
import types
import numpy as np
import engineering_compare as compare
import engineering_plan as plan_module
import engineering_queue_audit as queue_audit
import engineering_scopes as scopes
N, E, K = 166700, 25582938, 4064
CANDIDATES, FAST_APL_EDGES = 14232, 5845
AM_OVERLAY_SHA = '7f891d0ba79b393c96e75fd2dee2eddc51e57d8213eaac2779703bc2c18712b4'
MASK_SHA = '8b0fb6eb2e6c31da4d17dc20d890aa3f367e514f05642de5bad6b42a1611c05b'
SUCCESS = 'PASS-EXACT-LIGHT-CONTINUATION-ENGINEERING-NOT-LEARNING'
def require(value, message):
if not value:
raise ValueError(message)
def _canonical(value):
return json.dumps(value, sort_keys=True, separators=(',', ':'), ensure_ascii=False, allow_nan=False)
def _sha_file(path):
h = hashlib.sha256()
with Path(path).open('rb') as stream:
for data in iter(lambda: stream.read(8 << 20), b''):
h.update(data)
return h.hexdigest()
class _Reader:
"""Fresh byte hashes, not stat caching; deterministic read manifest."""
def __init__(self, registered):
self.registered = registered
self.reads = {}
self.arrays = []
def bind(self, path, *, registered=True, expected=None):
path = Path(path)
require(path.is_absolute() and path.is_file() and not path.is_symlink(), 'Missing/linked/relative evidence: ' + str(path))
path = path.resolve(strict=True)
if registered:
require(str(path) in self.registered, 'Unregistered dependency: ' + str(path))
require(expected is None or expected == self.registered[str(path)], 'Conflicting external hash authority')
expected = self.registered[str(path)]
actual = _sha_file(path)
require(expected is None or (compare._sha(expected) and actual == expected), 'Evidence bytes changed: ' + str(path))
value = dict(path=str(path), bytes=path.stat().st_size, sha256=actual)
require(str(path) not in self.reads or self.reads[str(path)] == value, 'File changed during analysis')
self.reads[str(path)] = value
return path
def json(self, path, **kwargs):
return compare._json(self.bind(path, **kwargs))
def npy(self, path):
value = np.load(self.bind(path), mmap_mode='r', allow_pickle=False)
self.arrays.append(value)
return value
def npz(self, path, keys):
with np.load(self.bind(path), allow_pickle=False) as archive:
require(len(archive.files) == len(keys) and set(archive.files) == set(keys), 'Unexpected input NPZ keys')
return {name: archive[name].copy() for name in keys}
def run(self, directory, *, legacy):
directory = Path(directory)
names = ['summary.json', 'state.npz', 'apl-timing-audit.json', 'apl-pending-tail.float32.npy', 'requests.csv']
if legacy:
names.append('adapter-provenance.json')
else:
names.append('terminal-queues.npz')
names += [name for name in [*compare.TRACE_FILES.values(), 'tick-state-hashes.jsonl', 'queue.jsonl']
if (directory / name).is_file()]
for name in names:
self.bind(directory / name, registered=legacy)
return compare.load_run(directory, allow_legacy_am=legacy)
def finish(self):
for row in self.reads.values():
path = Path(row['path'])
require(path.is_file() and path.stat().st_size == row['bytes'] and _sha_file(path) == row['sha256'],
'Read evidence changed during analysis: ' + str(path))
return [self.reads[key] for key in sorted(self.reads)]
def close(self):
for value in self.arrays:
mmap = getattr(value, '_mmap', None)
if mmap is not None:
mmap.close()
def _registration(reg, reg_sha, verified):
require(type(reg) is dict and reg.get('schema') == 'phase4an-engineering-registration-v1' and
reg.get('training') is False and compare._sha(reg_sha), 'Invalid engineering registration')
plan = plan_module.validate_plan(reg['plan'])
ids = [row['row_id'] for row in plan['rows']]
require(type(verified) is dict and set(verified) == set(ids) and set(reg['cases']) == set(ids),
'Exactly all 32 registered runtime receipts/cases are required')
exp = None
for row in plan['rows']:
identity = row['row_id']; path = Path(reg['cases'][identity]['case_dir'])
require(path.is_absolute() and path.name == identity and path.parent.name == 'cases', 'Invalid registered case layout')
if exp is None:
exp = path.parent.parent
require(path == exp / 'cases' / identity, 'Cases span inconsistent experiment roots')
receipt = verified[identity]
require(receipt.get('schema') == 'phase4an-engineering-runtime-record-v1' and receipt.get('complete') is True and
receipt.get('training') is False and receipt.get('registration_sha256') == reg_sha and
_canonical(receipt.get('row')) == _canonical(row) and compare._sha(receipt.get('checkpoint_manifest_sha256')),
'Runtime receipt is not the exact verified registered row: ' + identity)
require(exp.name == 'phase4an-engineering-validation' and exp.parent.name == 'experiments', 'Unexpected fixed engineering output root')
require(set(reg['overlays']) == set(reg['selected_case']['AM_outputs']) == {'zero', 'static25'}, 'Incomplete overlay/AM authority')
return plan, exp
def _materialize(reader, reg, graph, stage):
"""Use the exact pinned pure AM helper; no original globals are patched."""
declaration = reg['overlays'][stage]
meta = reader.json(declaration['overlay_manifest'])
require(meta.get('schema') == 'phase4an-overlay-v1' and meta.get('overlay_name') == stage and
meta.get('am_overlay_sha256') == AM_OVERLAY_SHA and meta.get('mask_sha256') == MASK_SHA,
'Original fixed overlay authority changed')
helper_path = reader.bind(meta['am_overlay_path'], expected=AM_OVERLAY_SHA)
module = types.ModuleType('phase4an_analysis_pinned_overlay')
module.__file__ = str(helper_path)
exec(compile(helper_path.read_bytes(), str(helper_path), 'exec'), module.__dict__)
mask = reader.npy(meta['mask_path']); base_candidate = reader.npy(meta['base_candidate_path'])
require(reader.reads[str(Path(meta['mask_path']).resolve())]['sha256'] == meta['mask_sha256'] and
reader.reads[str(Path(meta['base_candidate_path']).resolve())]['sha256'] == meta['base_candidate_sha256'],
'Overlay external mask/base hash mismatch')
overlay_file = reader.bind(declaration['overlay_file'], expected=meta['overlay_sha256'])
arrays = reader.npz(overlay_file, {'edge_id', 'plastic_delta_mv'})
require(mask.dtype == np.int64 and mask.shape == (CANDIDATES,), 'Complete frozen plastic mask required')
values = module.validate_arrays(arrays['edge_id'], arrays['plastic_delta_mv'], mask, base_candidate)
if stage == 'zero':
require(not np.any(values['plastic_delta_mv'] != 0), 'Zero overlay is nonzero')
else:
require(module.bit_equal(values['plastic_delta_mv'], np.multiply(np.float32(-.25), base_candidate)),
'static25 differs from the original float32-mV floor')
effective, mapping = module.materialize_numpy(graph['weights'], graph['edge_id'], graph['apl_out_mask'], values)
require(mapping['mapped_candidate_count'] == CANDIDATES and mapping['missing_candidate_count'] == 0 and
mapping['apl_overlap_count'] == 0, 'Incomplete/overlapping candidate mapping')
base_sha, effective_sha = module.expected_hashes(graph['weights'], graph['apl_out_mask'], mapping)
require(queue_audit._raw_sha(effective) == effective_sha, 'CPU materialization/hash disagreement')
return effective, dict(effective_weights_raw_sha256=effective_sha, base_weights_raw_sha256=base_sha,
candidate_count=CANDIDATES, installed_nonzero_count=mapping['installed_nonzero_count'],
apl_fast_edges_zeroed=FAST_APL_EDGES, bridge=module.BRIDGE,
helper_source_sha256=AM_OVERLAY_SHA, weights_updated_during_run=False)
def _load_context(reg, reg_sha, plan, exp):
reader = _Reader(reg['input_sha256'])
records, legacy = {}, {}
try:
actual_reg = reader.json(exp / 'protocol/preregistered.json', registered=False, expected=reg_sha)
require(_canonical(actual_reg) == _canonical(reg), 'Supplied registration differs from actual registered bytes')
for path, expected in sorted(reg['source_sha256'].items()):
reader.bind(path, registered=False, expected=expected)
original = Path(reg['selected_case']['original_dir'])
spec = reader.json(original / 'spec.json')
require(spec.get('kind') == 'fullgraph' and spec.get('steps') == 1000 and spec.get('dt_ms') == .1 and
spec.get('apl_delay_steps') == 18, 'Original fixed fullgraph D18 spec changed')
requests = reader.npz(original / 'requests.npz', {'tick', 'neuron_idx'})
graph_dir, bundle = Path(spec['graph_dir']), Path(spec['apl_bundle_dir'])
graph = {key: reader.npy(graph_dir / (key + '.npy')) for key in ('nodes', 'ptr', 'dst', 'edge_id', 'weights')}
require(graph['nodes'].dtype == np.int32 and np.array_equal(graph['nodes'], np.arange(N, dtype=np.int32)),
'Graph is not the complete canonical neuron set')
n, e = queue_audit._graph(graph['ptr'], graph['dst'], graph['edge_id'], graph['weights'])
require((n, e) == (N, E), 'Original graph dimensions changed')
apl = reader.npy(bundle / 'apl_globals.npy')
require(apl.dtype == np.int32 and apl.shape == (2,) and np.all((apl >= 0) & (apl < N)) and
apl[0] < apl[1], 'Invalid canonical APL source identity')
mask = np.zeros(E, np.bool_)
for source in apl:
mask[graph['ptr'][source]:graph['ptr'][source + 1]] = True
require(int(mask.sum()) == FAST_APL_EDGES, 'Frozen fast-APL source-edge mask changed')
graph['apl_out_mask'] = mask
storage = {}
for row in plan['rows']:
identity = row['row_id']; folder = exp / 'runs' / identity
require(folder.is_dir() and folder.resolve() == folder and not any(p.is_symlink() for p in folder.rglob('*')),
'Missing/linked runtime evidence folder')
records[identity] = reader.run(folder, legacy=False)
storage[identity] = sum(p.stat().st_size for p in folder.rglob('*') if p.is_file())
for stage in ('zero', 'static25'):
legacy[stage] = reader.run(reg['selected_case']['AM_outputs'][stage], legacy=True)
return dict(reader=reader, records=records, legacy=legacy, graph=graph, requests=requests,
storage=storage, input_scope='read-only authenticated selected original inputs and AM outputs')
except BaseException:
for record in [*records.values(), *legacy.values()]:
compare.close_run(record)
reader.close()
raise
def _pair(candidate, reference):
checks, observed = compare._pair_checks(candidate, reference)
return dict(numerical_exact=all(checks.values()), checks=checks,
failed_checks=[key for key, value in checks.items() if not value],
reference_comparisons_available=observed, learning_pass_claimed=False)
def _dispatch(descriptor, records, legacy, overlay):
candidates = [records[name] for name in descriptor['candidate_row_ids']]
reference = records[descriptor['reference_row_id']] if descriptor['reference_row_id'] is not None else None
kind = descriptor['check']
if kind == 'am_exact':
return compare.compare_fresh(candidates[0], legacy[overlay])
if kind == 'fresh_exact':
return compare.compare_fresh(candidates[0], reference)
if kind in ('prefix_of_reference', 'segment_of_reference'):
return scopes.compare_slice(candidates[0], reference)
if kind == 'segment_exact':
return _pair(candidates[0], reference)
if kind == 'checkpoint_exact':
return scopes.compare_checkpoint(candidates[0], reference)
if kind == 'parent_carry':
return scopes.compare_parent_carry(candidates[0], reference)
if kind == 'chain_exact':
return compare.compare_segments(reference, candidates)
raise ValueError('Unimplemented registered comparison: ' + str(kind))
def _number(value, name, integer=False):
if value is None:
return None
require((type(value) is int if integer else type(value) in (int, float)) and math.isfinite(value) and value >= 0,
'Malformed observed metric: ' + name)
return value
def _run_checks(row, record, expected_hash, reference):
compare.validate_run(record)
summary = record['summary']; identity = row['row_id']
require(len(record['state']['v']) == N and record['terminal']['apl_pending'].shape == (18, K), 'Incomplete full brain state: ' + identity)
checks = {key: summary.get(key) is True for key in ('complete', 'pass_all', 'all_finite', 'counts_match_sparse_events')}
fixed = dict(nodes=N, edges=E, steps=row['steps'], completed_ticks=row['steps'], apl_delay_steps=18,
requested=len(record['requests']), total_spikes=len(record['state']['step']),
accepted_requests=int(record['requests'][:, 3].sum(dtype=np.int64)),
rejected_requests=int(record['requests'][:, 4].sum(dtype=np.int64)),
apl_fast_edges_zeroed=FAST_APL_EDGES, kc_targets_present=K)
for key, value in fixed.items():
checks[key] = type(summary.get(key)) is int and summary[key] == value
checks['registered_clock'] = record['steps'] == row['steps'] and record['global_tick_offset'] == row['start']
checks['recording_mode'] = summary.get('diagnostic_mode') == row['mode']
checks['effective_weight_bytes'] = summary.get('effective_weights_raw_sha256') == expected_hash
checks['within_trial_weights_fixed'] = summary.get('within_trial_weight_updates') is False
checks['learning_not_claimed'] = summary.get('learning_pass_claimed') is False
expected_queue = reference['queues'][row['start']:row['stop']]
for field, queue_key in [('arriving_edges', 'count'), ('accepted_arriving_edges', 'accepted_count')]:
expected = sum(q[queue_key] for q in expected_queue)
checks[field] = type(summary.get(field)) is int and summary[field] == expected
return {name: bool(value) for name, value in checks.items()}
def analyze(reg, reg_sha, verified_records):
"""No writes/runs. The caller rechecks registration and receipts afterward."""
plan, exp = _registration(reg, reg_sha, verified_records)
context = _load_context(reg, reg_sha, plan, exp)
records, legacy = context['records'], context['legacy']
try:
require(set(records) == {r['row_id'] for r in plan['rows']} and set(legacy) == {'zero', 'static25'},
'Loaded evidence inventory is incomplete')
comparisons = []
for row in plan['rows']:
for number, descriptor in enumerate(row['expected_comparisons']):
try:
result = _dispatch(descriptor, records, legacy, row['overlay'])
require(type(result.get('numerical_exact')) is bool, 'Comparator lacks an explicit Boolean numerical gate')
except (ValueError, KeyError, TypeError, IndexError, OverflowError) as error:
result = dict(numerical_exact=False, error_type=type(error).__name__, error=str(error),
incomplete_observation_not_fabricated=True)
comparisons.append(dict(descriptor_id=row['row_id'] + '/' + str(number),
descriptor=descriptor, result=result))
expected_descriptors = sum(len(r['expected_comparisons']) for r in plan['rows'])
require(len(comparisons) == expected_descriptors, 'Not every registered descriptor was evaluated')
overlay_reports, metric_rows, all_row_checks = {}, [], {}
for stage in ('zero', 'static25'):
effective, installation = _materialize(context['reader'], reg, context['graph'], stage)
stage_rows = [r for r in plan['rows'] if r['overlay'] == stage]
ref_row = next(r for r in stage_rows if r['kind'] == 'fresh' and r['mode'] == 'reference')
reference = records[ref_row['row_id']]
reference_chain = [records[r['row_id']] for r in stage_rows if r['kind'] == 'chain' and r['mode'] == 'reference']
cuts = [r['start'] for r in stage_rows if r['kind'] == 'chain' and r['mode'] == 'reference' and r['start'] > 0]
independent = queue_audit.audit_queues(reference, reference_chain, ptr=context['graph']['ptr'], dst=context['graph']['dst'],
edge_id=context['graph']['edge_id'], effective_weights=effective, requests=context['requests'], cuts=cuts)
del effective
reference_result = next(item['result'] for item in comparisons if item['descriptor']['check'] == 'chain_exact' and
item['descriptor']['reference_row_id'] == ref_row['row_id'])
light_result = next(item['result'] for item in comparisons if item['descriptor']['check'] == 'chain_exact' and
item['descriptor']['reference_row_id'] == plan_module._fresh_id(stage, 'light'))
coverage = reference_result.get('coverage', {})
apl_values = coverage.get('carried_APL_nonzero_applied_values')
simultaneous_carry = any(type(row.get('pending_source_spike_events')) is int and row['pending_source_spike_events'] > 0 and
type(row.get('carried_APL_nonzero_pending_values')) is int and row['carried_APL_nonzero_pending_values'] > 0
for row in coverage.get('boundaries', []))
coverage_pass = (reference_result.get('nonvacuous_temporal_coverage') is True and
type(apl_values) is int and apl_values > 0 and simultaneous_carry and
independent.get('accepted_nonzero_crossing_witness_verified') is True)
legacy_checks = dict(full_dimensions=type(legacy[stage]['summary'].get('nodes')) is int and legacy[stage]['summary']['nodes'] == N and
type(legacy[stage]['summary'].get('edges')) is int and legacy[stage]['summary']['edges'] == E,
effective_weight_bytes=legacy[stage]['summary'].get('effective_weights_raw_sha256') == installation['effective_weights_raw_sha256'])
overlay_reports[stage] = dict(installation=installation, independent_queue_audit=independent,
reference_numerical_exact=reference_result.get('numerical_exact') is True,
light_numerical_exact=light_result.get('numerical_exact') is True,
live_coverage_pass=coverage_pass, legacy_checks=legacy_checks,
same_internal_boundary_has_spike_and_APL_carry=simultaneous_carry,
carried_APL_nonzero_applied_values=apl_values,
light_standalone_coverage=light_result.get('coverage'),
light_per_tick_float_state_directly_observed=False,
reference_coverage_does_not_fabricate_light_observations=True)
for row in stage_rows:
identity = row['row_id']; record = records[identity]; summary = record['summary']; receipt = verified_records[identity]
checks = _run_checks(row, record, installation['effective_weights_raw_sha256'], reference)
all_row_checks[identity] = checks
metric_rows.append(dict(row_id=identity, overlay=stage, mode=row['mode'], start=row['start'], stop=row['stop'],
biological_time_ms=row['steps'] * .1, runtime_and_immediate_seal_wall_seconds=_number(receipt.get('runtime_and_immediate_seal_wall_seconds'), 'runtime wall'),
runtime_reported_wall_seconds=_number(summary.get('wall_seconds'), 'runtime reported wall'),
diagnostic_simulation_seconds=_number(summary.get('diagnostic_simulation_seconds'), 'simulation'),
build_or_cache_seconds=_number(summary.get('build_or_cache_seconds'), 'build/cache'),
peak_allocated_vram_bytes=_number(summary.get('peak_allocated_vram_bytes'), 'allocated VRAM', True),
peak_reserved_vram_bytes=_number(summary.get('peak_reserved_vram_bytes'), 'reserved VRAM', True),
sealed_output_file_bytes=context['storage'][identity], compressed_storage=receipt.get('storage'),
gpu_utilization=None, cpu_utilization=None, unobserved_metrics=['gpu_utilization', 'cpu_utilization'],
checkpoint_manifest_sha256=receipt['checkpoint_manifest_sha256'],
verified_runtime_receipt_canonical_sha256=hashlib.sha256(_canonical(receipt).encode('utf-8')).hexdigest()))
numerical = (all(item['result']['numerical_exact'] for item in comparisons) and
all(all(checks.values()) for checks in all_row_checks.values()) and
all(all(r['legacy_checks'].values()) and r['independent_queue_audit']['queue_ref_request_exact'] for r in overlay_reports.values()))
covered = all(row['live_coverage_pass'] for row in overlay_reports.values())
read_manifest = context['reader'].finish()
report = dict(schema='phase4an-engineering-analysis-v1', registration_sha256=reg_sha,
status='FAIL-NUMERICAL' if not numerical else (SUCCESS if covered else 'INCOMPLETE-COVERAGE'),
pass_all=bool(numerical and covered), numerical_exact=bool(numerical), nonvacuous_temporal_coverage=bool(covered),
expected_calls=32, analyzed_calls=len(records), executed_ticks=plan['total_ticks'],
expected_comparisons=expected_descriptors, evaluated_comparisons=len(comparisons), comparisons=comparisons,
run_checks=all_row_checks, failed_run_checks={identity: [k for k, ok in checks.items() if not ok] for identity, checks in all_row_checks.items() if not all(checks.values())},
overlay_reports=overlay_reports, metrics=metric_rows, read_manifest=read_manifest,
source_sha256=reg['source_sha256'], process_receipts_and_checkpoint_chains_verified_by_caller=True,
external_AM_full_seals_reverified_here=False, no_new_brain_runs_during_analysis=True,
GPU_used=False, training=False, learning_pass_claimed=False, candidate_generated=False,
physiology_or_memory_success_claimed=False,
limitations=['Fixed zero/static25 engineering trajectories only; not conditioning or learned memory.',
'Light intermediate V/g/ref, APL histories and per-tick queue arrays were not directly recorded.',
'Independent queue/ref reconstruction is conditional on recorded spikes; V/g dynamics are not independently re-simulated.',
'Missing performance metrics are null, not zero. No universal performance claim is made.',
'Caller verifies all process receipts, parent chains, registration authority and full AM/frozen seals.'])
_canonical(report) # Fail rather than return non-JSON/nonfinite output.
return report
finally:
for record in [*records.values(), *legacy.values()]:
compare.close_run(record)
context['reader'].close()
@@ -0,0 +1,413 @@
"""Read-only CPU comparisons for the UNREGISTERED AN engineering gate.
load_run(path, allow_legacy_am=False) reads actual AJ/AM/AN artifact schemas.
Call close_run(record) after using explicitly loaded memory-mapped traces.
compare_fresh(left, right, legacy_am=None) and compare_segments(monolithic,
segments) accept loaded dictionaries or paths and return JSON-safe reports.
No report is written, no GPU is imported, no training/biological PASS is claimed.
The caller owns external source/input/root SHA pins and case selection. These
comparisons do not replace seal verification or prove unseen inputs unchanged.
AM did not save its raw spike ring: its fresh terminal ring is explicitly
reconstructed from sparse events, never described as independently recorded.
Reference queue joint hashes include LOCAL tick headers; cross-segment checks
compare the separate ordered-edge/accepted-mask hashes and counts instead.
"""
from __future__ import annotations
import csv
import hashlib
import json
from pathlib import Path
import re
import numpy as np
STATE_KEYS = ('step', 'neuron_idx', 'counts', 'v', 'g', 'ref')
REQUEST_FIELDS = ('tick', 'neuron_idx', 'attempted', 'accepted', 'rejected')
TRACE_FILES = {'drive': 'apl-drive.float32.npy', 'state': 'apl-state.float32.npy',
'generated': 'apl-inhibition.float32.npy', 'applied': 'apl-applied.float32.npy'}
HASH_FIELDS = ('v_sha256', 'g_sha256', 'ref_sha256')
QUEUE_FIELDS = ('count', 'accepted_count', 'ordered_edge_sha256', 'accepted_mask_sha256')
def require(ok, message):
if not ok:
raise ValueError(message)
def bit_equal(a, b):
return (isinstance(a, np.ndarray) and isinstance(b, np.ndarray) and a.dtype == b.dtype and a.shape == b.shape and
np.array_equal(np.ascontiguousarray(a).view(np.uint8), np.ascontiguousarray(b).view(np.uint8)))
def _sha(value):
return isinstance(value, str) and re.fullmatch('[0-9a-f]{64}', value) is not None
def _json(path):
def pairs(values):
result = {}
for key, value in values:
require(key not in result, 'Duplicate JSON key: ' + key)
result[key] = value
return result
return json.loads(Path(path).read_text(encoding='utf-8'), object_pairs_hook=pairs,
parse_constant=lambda value: (_ for _ in ()).throw(ValueError('Nonfinite JSON: ' + value)))
def _jsonl(path):
with Path(path).open(encoding='utf-8') as stream:
return [json.loads(line) for line in stream if line.strip()]
def _npz(path, keys):
with np.load(path, allow_pickle=False) as archive:
require(len(archive.files) == len(keys) and set(archive.files) == set(keys), 'Unexpected array keys: ' + str(path))
return {key: archive[key].copy() for key in keys}
def reconstructed_fresh_ring(state, steps, neurons):
ring = np.zeros((19, neurons), np.uint8)
for tick in range(max(0, steps - 19), steps):
first, last = np.searchsorted(state['step'], (tick, tick + 1))
ring[tick % 19, state['neuron_idx'][first:last]] = 1
return ring
def normalized_ring(terminal):
ring, phase = terminal['spike_ring'], int(terminal['next_tick'])
return ring[(np.arange(19) + phase % 19) % 19]
def _requests(path):
with Path(path).open(encoding='utf-8', newline='') as stream:
reader = csv.DictReader(stream)
require(tuple(reader.fieldnames or ()) == REQUEST_FIELDS, 'requests.csv must use the exact five AJ fields in order')
rows = []
for row in reader:
require(set(row) == set(REQUEST_FIELDS), 'Extra/missing request CSV field')
rows.append([int(row[key]) for key in REQUEST_FIELDS])
return np.asarray(rows, dtype=np.int64).reshape(-1, 5)
def load_run(directory, *, allow_legacy_am=False):
directory = Path(directory).resolve(strict=True)
require(directory.is_dir(), 'Output is not a directory')
require(not any('error' in p.name.lower() or 'failure' in p.name.lower() for p in directory.iterdir()),
'Run contains error/failure evidence')
summary = _json(directory / 'summary.json')
require(summary.get('complete') is True and summary.get('pass_all') is True and summary.get('all_finite') is True and
summary.get('steps') == summary.get('completed_ticks'), 'Run is incomplete or failed')
state = _npz(directory / 'state.npz', STATE_KEYS)
steps, neurons = summary['steps'], len(state['v'])
require(summary.get('nodes') == neurons and summary.get('precision') == 'float64' and
summary.get('apl_precision') == 'float32' and summary.get('apl_delay_steps') == 18, 'Unexpected AN/AM numerical schema')
timing = _json(directory / 'apl-timing-audit.json')
pending = np.load(directory / 'apl-pending-tail.float32.npy', mmap_mode='r', allow_pickle=False)
terminal_path = directory / 'terminal-queues.npz'
if terminal_path.is_file():
terminal = _npz(terminal_path, ('spike_ring', 'apl_state', 'apl_pending', 'next_tick'))
origin = 'recorded_terminal_queues'
else:
require(allow_legacy_am and (directory / 'adapter-provenance.json').is_file(), 'Recorded terminal queues are missing')
bits = timing.get('final_apl_state_u32_bits')
require(isinstance(bits, list) and len(bits) == 2 and all(type(v) is int and 0 <= v < 2**32 for v in bits),
'AM has no exact terminal APL bits')
terminal = {'spike_ring': reconstructed_fresh_ring(state, steps, neurons),
'apl_state': np.asarray(bits, dtype=np.uint32).view(np.float32), 'apl_pending': np.asarray(pending),
'next_tick': np.asarray(steps, np.int64)}
origin = 'AM_spike_ring_derived_from_fresh_events_APL_bits_from_timing'
require(bit_equal(terminal['apl_pending'], pending), 'Terminal/pending file mismatch')
require(timing.get('final_apl_state_u32_bits') == terminal['apl_state'].view(np.uint32).tolist(), 'Terminal APL bit metadata mismatch')
continued = summary.get('checkpoint_continuation_used', False)
require(type(continued) is bool, 'Invalid continuation scope flag')
offset = summary.get('global_tick_offset', 0)
require(not continued or (type(offset) is int and offset > 0 and summary.get('global_next_tick') == offset + steps),
'Continued output lacks an explicit global clock')
require(continued or offset == 0, 'Fresh output has a nonzero global offset')
traces = {name: np.load(directory / file, mmap_mode='r', allow_pickle=False)
for name, file in TRACE_FILES.items() if (directory / file).is_file()}
reference = summary.get('diagnostic_mode') == 'reference' or origin.startswith('AM_')
if reference or summary.get('full_APL_history_recorded') is True:
require(len(traces) == 4, 'Declared reference APL history is incomplete')
hashes = _jsonl(directory / 'tick-state-hashes.jsonl') if (directory / 'tick-state-hashes.jsonl').is_file() else None
queues = _jsonl(directory / 'queue.jsonl') if (directory / 'queue.jsonl').is_file() else None
if reference or summary.get('full_tick_state_hashes_recorded') is True:
require(hashes is not None, 'Declared state hashes are missing')
if reference or summary.get('full_queue_hashes_recorded') is True:
require(queues is not None, 'Declared queue hashes are missing')
result = {'source': str(directory), 'state': state, 'terminal': terminal, 'requests': _requests(directory / 'requests.csv'),
'steps': steps, 'global_tick_offset': offset, 'summary': summary, 'traces': traces,
'state_hashes': hashes, 'queues': queues, 'terminal_origin': origin}
validate_run(result)
return result
def close_run(run):
"""Release read-only mappings opened by load_run; no files are modified."""
arrays = [*run.get('traces', {}).values(), *run.get('terminal', {}).values()]
closed = set()
for array in arrays:
current = array
while current is not None:
mapping = getattr(current, '_mmap', None)
if mapping is not None and id(mapping) not in closed:
mapping.close(); closed.add(id(mapping))
current = getattr(current, 'base', None)
def validate_run(run):
"""Array-level contract, also usable by synthetic tests without files."""
require(isinstance(run, dict), 'Run must be a dictionary')
steps, offset = run['steps'], run['global_tick_offset']
require(type(steps) is int and steps > 0 and type(offset) is int and offset >= 0, 'Invalid segment clock')
state, terminal = run['state'], run['terminal']
require(set(state) == set(STATE_KEYS), 'Expected exactly the six state keys')
n = len(state['v'])
require(n > 0, 'Empty neuron population')
for name, dtype in (('v', np.float64), ('g', np.float64), ('ref', np.int32), ('counts', np.int32)):
value = state[name]
require(value.dtype == dtype and value.shape == (n,) and np.isfinite(value).all(), 'Invalid state ' + name)
require(np.all((state['ref'] >= 0) & (state['ref'] <= 22)) and np.all(state['counts'] >= 0), 'Invalid ref/count values')
tick, node = state['step'], state['neuron_idx']
require(tick.dtype == node.dtype == np.int32 and tick.ndim == node.ndim == 1 and tick.shape == node.shape and
np.all((tick >= 0) & (tick < steps)) and np.all((node >= 0) & (node < n)), 'Invalid events')
keys = tick.astype(np.int64) * n + node
require(len(keys) < 2 or np.all(keys[1:] > keys[:-1]), 'Events must be sorted unique')
require(np.array_equal(np.bincount(node, minlength=n), state['counts']), 'Counts disagree with segment events')
request = run['requests']
require(isinstance(request, np.ndarray) and request.dtype == np.int64 and request.ndim == 2 and request.shape[1] == 5,
'Requests must be int64 rows in the five AJ fields')
require(np.all((request[:, 0] >= 0) & (request[:, 0] < steps)) and np.all((request[:, 1] >= 0) & (request[:, 1] < n)) and
np.all(request[:, 2] == 1) and np.all((request[:, 3] == 0) | (request[:, 3] == 1)) and
np.all(request[:, 4] == 1 - request[:, 3]), 'Invalid request values')
request_keys = request[:, 0] * n + request[:, 1]
require(len(request_keys) < 2 or np.all(request_keys[1:] > request_keys[:-1]), 'Requests must be sorted unique')
require(set(terminal) == {'spike_ring', 'apl_state', 'apl_pending', 'next_tick'}, 'Invalid terminal keys')
ring = terminal['spike_ring']
require(ring.dtype == np.uint8 and ring.shape == (19, n) and np.all(ring <= 1), 'Invalid spike ring')
phase = terminal['next_tick']
require(isinstance(phase, np.ndarray) and phase.dtype == np.int64 and phase.shape == () and int(phase) == steps,
'Terminal phase must be local segment length')
pending, apl_state = terminal['apl_pending'], terminal['apl_state']
require(pending.dtype == np.float32 and pending.ndim == 2 and pending.shape[0] == 18 and pending.shape[1] > 0 and
apl_state.dtype == np.float32 and apl_state.shape == (2,) and np.isfinite(pending).all() and np.isfinite(apl_state).all(),
'Invalid APL terminal state')
for t in range(max(0, steps - 19), steps):
expected = np.zeros(n, np.uint8)
first, last = np.searchsorted(tick, (t, t + 1)); expected[node[first:last]] = 1
require(bit_equal(ring[t % 19], expected), 'Written ring slot differs from sparse events')
if offset == 0 and steps < 19:
require(not np.any(ring[steps:]), 'Fresh uninitialized ring slots are not zero')
traces = run.get('traces', {})
require(not traces or set(traces) == set(TRACE_FILES), 'Partial APL trace set')
for key, trace in traces.items():
width = 2 if key in ('drive', 'state') else pending.shape[1]
require(trace.dtype == np.float32 and trace.shape == (steps, width) and np.isfinite(trace).all(), 'Invalid APL history: ' + key)
if traces:
require(bit_equal(traces['state'][-1], apl_state), 'Final APL history/state mismatch')
kept = min(steps, 18)
require(bit_equal(pending[-kept:], traces['generated'][-kept:]), 'Terminal APL queue does not retain generated suffix')
if steps > 18:
require(bit_equal(traces['applied'][18:], traces['generated'][:-18]), 'Within-segment APL delay differs from 18')
if offset == 0:
require(np.all(traces['applied'][:min(18, steps)].view(np.uint32) == 0), 'Fresh APL application padding is not positive zero')
hashes, queues = run.get('state_hashes'), run.get('queues')
if hashes is not None:
require(len(hashes) == steps, 'State hash coverage incomplete')
for t, row in enumerate(hashes):
require(set(row) == {'tick', *HASH_FIELDS} and type(row['tick']) is int and row['tick'] == t and
all(_sha(row[key]) for key in HASH_FIELDS), 'Invalid state hash row')
for name in ('v', 'g', 'ref'):
raw = np.ascontiguousarray(state[name], dtype=state[name].dtype.newbyteorder('<')).tobytes()
require(hashes[-1][name + '_sha256'] == hashlib.sha256(raw).hexdigest(), 'Last state hash differs from terminal ' + name)
if queues is not None:
require(len(queues) == steps, 'Queue coverage incomplete')
for t, row in enumerate(queues):
require(set(row) == {'tick', *QUEUE_FIELDS, 'joint_sha256'} and type(row['tick']) is int and row['tick'] == t and
type(row['count']) is int and type(row['accepted_count']) is int and 0 <= row['accepted_count'] <= row['count'] and
all(_sha(row[key]) for key in ('ordered_edge_sha256', 'accepted_mask_sha256', 'joint_sha256')), 'Invalid queue row')
return run
def _run(value, legacy=False):
return load_run(value, allow_legacy_am=legacy) if isinstance(value, (str, Path)) else validate_run(value)
def _activity(run):
terminal = run['terminal']
result = {'spikes': len(run['state']['step']), 'requested': len(run['requests']),
'accepted_requests': int(run['requests'][:, 3].sum(dtype=np.int64)),
'terminal_apl_state_nonzero_values': int(np.count_nonzero(terminal['apl_state'])),
'terminal_pending_nonzero_values': int(np.count_nonzero(terminal['apl_pending']))}
traces = run.get('traces', {})
result['generated_h_nonzero_values'] = int(np.count_nonzero(traces['generated'])) if traces else None
result['applied_h_nonzero_values'] = int(np.count_nonzero(traces['applied'])) if traces else None
result['apl_nonzero_observed'] = bool(result['terminal_apl_state_nonzero_values'] or result['terminal_pending_nonzero_values'] or
(result['generated_h_nonzero_values'] or 0) or (result['applied_h_nonzero_values'] or 0))
return result
def _pair_checks(left, right, prefix='pair'):
checks = {prefix + '.steps': left['steps'] == right['steps'], prefix + '.offset': left['global_tick_offset'] == right['global_tick_offset']}
for name in STATE_KEYS:
checks[prefix + '.state.' + name] = bit_equal(left['state'][name], right['state'][name])
checks[prefix + '.requests_five_fields'] = bit_equal(left['requests'], right['requests'])
for name in ('spike_ring', 'apl_state', 'apl_pending', 'next_tick'):
checks[prefix + '.terminal.' + name] = bit_equal(left['terminal'][name], right['terminal'][name])
for field in ('effective_weights_raw_sha256',):
a, b = left.get('summary', {}).get(field), right.get('summary', {}).get(field)
if a is not None or b is not None:
checks[prefix + '.' + field] = _sha(a) and a == b
availability = {'APL_traces': bool(left.get('traces')) and bool(right.get('traces')),
'state_hashes': left.get('state_hashes') is not None and right.get('state_hashes') is not None,
'queue_hashes': left.get('queues') is not None and right.get('queues') is not None}
if availability['APL_traces']:
for name in TRACE_FILES:
checks[prefix + '.trace.' + name] = bit_equal(left['traces'][name], right['traces'][name])
if availability['state_hashes']:
checks[prefix + '.state_hashes'] = left['state_hashes'] == right['state_hashes']
if availability['queue_hashes']:
checks[prefix + '.queues'] = left['queues'] == right['queues']
return checks, availability
def compare_fresh(left, right, legacy_am=None):
left, right = _run(left), _run(right)
require(left['global_tick_offset'] == right['global_tick_offset'] == 0, 'Fresh comparison requires fresh runs')
checks, available = _pair_checks(left, right)
origins = [left.get('terminal_origin', 'caller_arrays'), right.get('terminal_origin', 'caller_arrays')]
if legacy_am is not None:
legacy = _run(legacy_am, legacy=True)
require(legacy['global_tick_offset'] == 0, 'AM control must be fresh')
extra, legacy_available = _pair_checks(left, legacy, 'AM')
checks.update(extra); available['AM'] = legacy_available
# The left run may be light. Do not thereby skip the independent full
# diagnostic comparison between the right reference and sealed AM.
extra, reference_available = _pair_checks(right, legacy, 'reference_vs_AM')
checks.update(extra); available['reference_vs_AM'] = reference_available
origins.append(legacy.get('terminal_origin', 'caller_arrays'))
activity = _activity(left)
exact = bool(all(checks.values()))
nonvacuous = activity['spikes'] > 0 and activity['apl_nonzero_observed']
return {'schema': 'phase4an-fresh-engineering-comparison-v1', 'numerical_exact': exact,
'nonvacuous_activity': nonvacuous, 'pass_all': exact and nonvacuous,
'status': 'FAIL-NUMERICAL' if not exact else ('PASS-FRESH-SCOPE' if nonvacuous else 'INCOMPLETE-COVERAGE'),
'checks': checks, 'failed_checks': [key for key, value in checks.items() if not value], 'activity': activity,
'reference_comparisons_available': available, 'terminal_origins': origins,
'cross_segment_validation_performed': False, 'temporal_validation_complete': False,
'scope': 'Fresh event/state/request/terminal identity only; no continuation, learning, physiology or performance PASS',
'independent_CSR_queue_reconstruction_performed': False,
'accepted_nonzero_weight_edge_witness_verified': False,
'external_seals_and_input_authority_verified_here': False}
def compare_segments(monolithic, segments):
mono = _run(monolithic)
parts = [_run(value) for value in segments]
require(len(parts) >= 2, 'Need a monolithic run and at least two segments')
origin = mono['global_tick_offset']
if origin:
parent = mono.get('summary', {}).get('parent_checkpoint_manifest_sha256')
require(_sha(parent) and parts[0].get('summary', {}).get('parent_checkpoint_manifest_sha256') == parent,
'Nonzero-origin comparisons must bind the same starting parent checkpoint')
cursor = origin
for part in parts:
require(part['global_tick_offset'] == cursor, 'Segment clock gap/overlap/order error')
require(len(part['state']['v']) == len(mono['state']['v']), 'Segment population changed')
cursor += part['steps']
require(cursor == origin + mono['steps'], 'Segments do not cover the monolithic duration')
checks = {}
step = np.concatenate([part['state']['step'].astype(np.int64) + part['global_tick_offset'] for part in parts])
neuron = np.concatenate([part['state']['neuron_idx'] for part in parts])
checks['concatenated_global_events'] = np.array_equal(step, mono['state']['step'].astype(np.int64) + origin) and bit_equal(neuron, mono['state']['neuron_idx'])
counts = np.zeros_like(mono['state']['counts'], dtype=np.int64)
for part in parts:
counts += part['state']['counts'].astype(np.int64)
checks['sum_segment_counts'] = np.array_equal(counts, mono['state']['counts'].astype(np.int64))
requests = []
for part in parts:
rows = part['requests'].copy(); rows[:, 0] += part['global_tick_offset']; requests.append(rows)
mono_requests = mono['requests'].copy(); mono_requests[:, 0] += origin
checks['concatenated_requests_five_fields'] = bit_equal(np.concatenate(requests), mono_requests)
last = parts[-1]
for name in ('v', 'g', 'ref'):
checks['terminal.' + name] = bit_equal(last['state'][name], mono['state'][name])
checks['terminal.normalized_spike_ring'] = bit_equal(normalized_ring(last['terminal']), normalized_ring(mono['terminal']))
for name in ('apl_state', 'apl_pending'):
checks['terminal.' + name] = bit_equal(last['terminal'][name], mono['terminal'][name])
expected_weight = mono.get('summary', {}).get('effective_weights_raw_sha256')
if expected_weight is not None:
checks['unchanged_effective_weights'] = _sha(expected_weight) and all(part.get('summary', {}).get('effective_weights_raw_sha256') == expected_weight for part in parts)
trace_available = bool(mono.get('traces')) and all(part.get('traces') for part in parts)
hash_available = mono.get('state_hashes') is not None and all(part.get('state_hashes') is not None for part in parts)
queue_available = mono.get('queues') is not None and all(part.get('queues') is not None for part in parts)
if trace_available:
for name in TRACE_FILES:
checks['concatenated_APL.' + name] = all(bit_equal(part['traces'][name], mono['traces'][name][part['global_tick_offset'] - origin:part['global_tick_offset'] - origin + part['steps']]) for part in parts)
if hash_available:
checks['concatenated_per_tick_state_hashes'] = all(all(row[name] == mono['state_hashes'][part['global_tick_offset'] - origin + tick][name] for name in HASH_FIELDS)
for part in parts for tick, row in enumerate(part['state_hashes']))
if queue_available:
checks['concatenated_queue_order_and_acceptance'] = all(all(row[name] == mono['queues'][part['global_tick_offset'] - origin + tick][name] for name in QUEUE_FIELDS)
for part in parts for tick, row in enumerate(part['queues']))
boundary_rows = []
events = mono['state']['step'].astype(np.int64) + origin
for before, part in zip(parts, parts[1:]):
cut, stop = part['global_tick_offset'], part['global_tick_offset'] + part['steps']
emitted_before = (events < cut) & (events + 18 >= cut)
arriving_this_segment = emitted_before & (events + 18 < stop)
pending = before['terminal']['apl_pending']
consume = min(part['steps'], 18)
spike_carry = normalized_ring(before['terminal'])
# Future local arrival tick j consumes normalized slot j+1. This also
# covers spikes inherited from before a nonzero monolithic origin,
# which are legitimately absent from that run's local event log.
pending_spikes = int(spike_carry[1:].sum(dtype=np.int64))
arriving_spikes = int(spike_carry[1:consume + 1].sum(dtype=np.int64))
if cut - origin >= 18:
checks['boundary.' + str(cut) + '.ring_pending_event_count'] = pending_spikes == int(emitted_before.sum())
# Slots not overwritten by this short segment must remain bit-identical.
if part['steps'] < 19:
checks['boundary.' + str(cut) + '.unwritten_spike_slots'] = bit_equal(part['terminal']['spike_ring'][part['steps']:], spike_carry[part['steps']:])
if part['steps'] < 18:
checks['boundary.' + str(cut) + '.unconsumed_APL'] = bit_equal(part['terminal']['apl_pending'][:18 - part['steps']], pending[part['steps']:])
if part.get('traces'):
checks['boundary.' + str(cut) + '.applied_parent_APL'] = bit_equal(part['traces']['applied'][:consume], pending[:consume])
queue_rows = (part['queues'][:consume] if part.get('queues') is not None else
mono['queues'][cut - origin:cut - origin + consume] if mono.get('queues') is not None else None)
boundary_rows.append({'global_tick': cut, 'next_segment_steps': part['steps'],
'pending_source_spike_events': pending_spikes, 'source_spikes_arriving_in_next_segment': arriving_spikes,
'visible_local_log_source_spikes_arriving_in_next_segment': int(arriving_this_segment.sum()),
'pending_source_spike_origin': 'recorded normalized ring slots; includes pre-origin carry',
'observed_carried_arriving_edges': sum(row['count'] for row in queue_rows) if queue_rows is not None else None,
'observed_carried_accepted_edges': sum(row['accepted_count'] for row in queue_rows) if queue_rows is not None else None,
'carried_APL_nonzero_pending_values': int(np.count_nonzero(pending)),
'carried_APL_nonzero_values_applied_in_next_segment': int(np.count_nonzero(pending[:consume])),
'carried_APL_nonzero_rows_applied_in_next_segment': int(np.count_nonzero(np.any(pending[:consume] != 0, axis=1)))})
source_arrivals = sum(row['source_spikes_arriving_in_next_segment'] for row in boundary_rows)
edge_counts = [row['observed_carried_arriving_edges'] for row in boundary_rows]
observed_edges = sum(value for value in edge_counts if value is not None) if any(value is not None for value in edge_counts) else None
applied_values = sum(row['carried_APL_nonzero_values_applied_in_next_segment'] for row in boundary_rows)
activity = _activity(mono)
coverage = {'boundaries': boundary_rows, 'source_spikes_crossing_boundaries_and_due_in_receiver': source_arrivals,
'observed_carried_arriving_edges': observed_edges, 'carried_APL_nonzero_applied_values': applied_values,
'boundary_pending_event_counts_may_overlap_across_short_segments': True,
'actual_edge_arrival_coverage_known': observed_edges is not None}
nonvacuous = bool(activity['spikes'] > 0 and activity['apl_nonzero_observed'] and source_arrivals > 0 and
observed_edges is not None and observed_edges > 0 and applied_values > 0)
exact = bool(all(checks.values()))
return {'schema': 'phase4an-segment-engineering-comparison-v1', 'numerical_exact': exact,
'nonvacuous_temporal_coverage': nonvacuous, 'temporal_validation_complete': exact and nonvacuous,
'pass_all': exact and nonvacuous,
'status': 'FAIL-NUMERICAL' if not exact else ('PASS-SEGMENT-SCOPE' if nonvacuous else 'INCOMPLETE-COVERAGE'),
'checks': checks, 'failed_checks': [key for key, value in checks.items() if not value],
'activity': activity, 'coverage': coverage, 'segments': len(parts), 'steps': mono['steps'], 'global_tick_offset': origin,
'reference_comparisons_available': {'APL_traces': trace_available, 'state_hashes': hash_available, 'queue_hashes': queue_available},
'queue_joint_hash_compared': False, 'queue_joint_hash_exclusion_reason': 'local tick header differs; ordered-edge/mask hashes and counts compared when available',
'scope': 'Explicit finite engineering trajectory/terminal/carry checks only; no learning, physiology, or performance PASS',
'independent_CSR_queue_reconstruction_performed': False,
'accepted_nonzero_weight_edge_witness_verified': False,
'external_seals_and_input_authority_verified_here': False}
@@ -0,0 +1,152 @@
"""Pure exact input derivations for a future preregistered engineering gate.
No I/O, random draws, scientific execution or registration. The coordinator
must bind the original case/overlay and every static dependency to the sealed
AM manifest before using these helpers. A self-consistent dictionary is not
independent input authority. Derivation outputs are copies; validate_requests
only checks and returns its original, unmodified input.
"""
from __future__ import annotations
import copy
import re
from pathlib import PurePosixPath
import numpy as np
N = 166700
AM_HELPER_SHA = '7f891d0ba79b393c96e75fd2dee2eddc51e57d8213eaac2779703bc2c18712b4'
MASK_SHA = '8b0fb6eb2e6c31da4d17dc20d890aa3f367e514f05642de5bad6b42a1611c05b'
def require(ok, message):
if not ok:
raise ValueError(message)
def _positive_int(value, name):
require(type(value) is int and 0 < value <= np.iinfo(np.int32).max, 'Invalid ' + name)
def _sha(value):
return type(value) is str and re.fullmatch('[0-9a-f]{64}', value) is not None
def _absolute(value):
return (type(value) is str and value.startswith('/') and not value.startswith('//') and
PurePosixPath(value).is_absolute() and
'..' not in PurePosixPath(value).parts)
def validate_requests(requests, steps):
_positive_int(steps, 'steps')
require(type(requests) is dict and set(requests) == {'tick', 'neuron_idx'}, 'Exact sparse request schema required')
for key, value in requests.items():
require(isinstance(value, np.ndarray) and value.dtype == np.int32 and value.ndim == 1,
'Request requires int32 vector: ' + key)
tick, neuron = requests['tick'], requests['neuron_idx']
require(tick.shape == neuron.shape, 'Request shape mismatch')
require(np.all((tick >= 0) & (tick < steps)) and np.all((neuron >= 0) & (neuron < N)), 'Request outside fixed trial/canonical set')
key = tick.astype(np.int64) * N + neuron
require(len(key) < 2 or np.all(key[1:] > key[:-1]), 'Requests must be sorted and unique; no implicit deduplication')
return requests
def slice_requests(requests, steps, start, stop):
"""Half-open exact slice; translate ticks only, never redraw or reorder."""
validate_requests(requests, steps)
require(type(start) is int and type(stop) is int and 0 <= start < stop <= steps, 'Invalid half-open input interval')
lo, hi = np.searchsorted(requests['tick'], (start, stop))
result = {'tick': (requests['tick'][lo:hi] - np.int32(start)).copy(),
'neuron_idx': requests['neuron_idx'][lo:hi].copy()}
validate_requests(result, stop - start)
return result
def verify_partition(original, steps, intervals, parts):
"""Verify complete coverage plus exact request reconstruction, even blanks."""
validate_requests(original, steps)
require(type(intervals) in (list, tuple) and type(parts) in (list, tuple) and
len(intervals) == len(parts) > 0, 'Partition needs all intervals and parts')
end, reconstructed = 0, {'tick': [], 'neuron_idx': []}
for interval, part in zip(intervals, parts):
require(type(interval) in (list, tuple) and len(interval) == 2, 'Invalid interval schema')
start, stop = interval
require(type(start) is int and type(stop) is int and start == end and start < stop <= steps,
'Partition gap/overlap/nonpositive interval')
validate_requests(part, stop - start)
expected = slice_requests(original, steps, start, stop)
require(all(part[k].tobytes() == expected[k].tobytes() for k in part), 'Partition changes actual requests')
reconstructed['tick'].append(part['tick'].astype(np.int64) + start)
reconstructed['neuron_idx'].append(part['neuron_idx'])
end = stop
require(end == steps, 'Partition omits the terminal interval')
require(np.array_equal(np.concatenate(reconstructed['tick']), original['tick']) and
np.array_equal(np.concatenate(reconstructed['neuron_idx']), original['neuron_idx']), 'Reconstructed input mismatch')
return {'complete': True, 'intervals': [list(x) for x in intervals], 'steps': steps,
'requests': len(original['tick']), 'random_draws': 0, 'input_reselected': False}
def derive_spec(original, *, start, stop, budget_root, output_budget_bytes, parent_sha=None):
"""Explicit new case admission/budget labels; frozen model fields unchanged.
New budget_root must be outside the OLD budget tree. Full source/data/
ancestor containment checks remain the runtime/coordinator's responsibility.
A nonzero interval requires the actual parent manifest's external SHA;
future parent hashes must not be fabricated during preregistration.
"""
require(type(original) is dict and original.get('kind') == 'fullgraph', 'Original fullgraph spec required')
steps = original.get('steps')
_positive_int(steps, 'original steps')
require(original.get('dt_ms') == .1 and original.get('duration_ms') == steps * .1 and
type(original.get('apl_delay_steps')) is int and original['apl_delay_steps'] == 18 and
original.get('record_history') is False and original.get('training') is False and
original.get('within_trial_weight_updates', False) is False, 'Original fixed D18 trial contract required')
require(not any('checkpoint' in k.lower() or k in ('global_tick_offset', 'resume', 'continuation', 'initial_state')
for k in original), 'Original must be a fresh case')
require(type(start) is int and type(stop) is int and 0 <= start < stop <= steps, 'Invalid case interval')
require(_absolute(budget_root) and _absolute(original.get('budget_root')), 'Explicit absolute budget roots required')
new, old = PurePosixPath(budget_root), PurePosixPath(original['budget_root'])
require(not new.is_relative_to(old) and not old.is_relative_to(new), 'New output budget must not overlap old experiment tree')
require(type(output_budget_bytes) is int and output_budget_bytes > 0, 'Explicit positive byte budget required')
require((start == 0 and parent_sha is None) or (start > 0 and _sha(parent_sha)), 'Only a continued interval needs a real parent SHA')
result = copy.deepcopy(original)
result.update(steps=stop - start, duration_ms=(stop - start) * .1, max_steps=steps,
budget_root=budget_root, output_budget_bytes=output_budget_bytes,
within_trial_weight_updates=False,
outer_protocol='phase4an-engineering-validation-no-training')
if start:
result.update(global_tick_offset=start, parent_checkpoint_manifest_sha256=parent_sha)
changes = {key: {'present_before': key in original, 'before': copy.deepcopy(original.get(key)),
'after': copy.deepcopy(value)} for key, value in result.items()
if key not in original or value != original[key]}
allowed = {'steps', 'duration_ms', 'max_steps', 'budget_root', 'output_budget_bytes',
'within_trial_weight_updates', 'outer_protocol', 'global_tick_offset',
'parent_checkpoint_manifest_sha256'}
require(set(changes) <= allowed and all(result[k] == v for k, v in original.items() if k not in allowed),
'Unexpected scientific spec edit')
return result, {'input_interval': [start, stop], 'explicit_field_changes': changes,
'all_other_fields_unchanged': True, 'random_draws': 0, 'training': False}
def adapt_overlay_manifest(original, am_helper_path):
"""Translate observed AM manifest fields; preserve original overlay bytes.
This function validates declarations only. Callers independently rehash
every referenced file against external AM/source authority before use.
"""
require(type(original) is dict and original.get('schema') == 'phase4am-overlay-v1' and
original.get('overlay_name') in ('zero', 'static25') and
original.get('training') is False and original.get('fixed_for_whole_trial') is True,
'Only the fixed AM zero/static25 overlays are in this engineering gate')
require(original.get('mask_sha256') == MASK_SHA and _absolute(am_helper_path), 'Original mask/helper identity required')
names = ('overlay_name', 'overlay_sha256', 'mask_path', 'mask_sha256',
'base_candidate_path', 'base_candidate_sha256', 'aj_model_dir')
require(all(k in original for k in names), 'Missing observed AM manifest field')
require(all(_sha(original[k]) for k in ('overlay_sha256', 'mask_sha256', 'base_candidate_sha256')),
'Invalid external overlay/base SHA')
require(all(_absolute(original[k]) for k in ('mask_path', 'base_candidate_path', 'aj_model_dir')),
'Invalid external overlay/base path')
return dict(schema='phase4an-overlay-v1', **{k: copy.deepcopy(original[k]) for k in names},
am_overlay_path=am_helper_path, am_overlay_sha256=AM_HELPER_SHA)
@@ -0,0 +1,153 @@
"""Pure fixed 32-call engineering inventory; no I/O, GPU or input preparation.
Rows describe required executions and comparisons, not evidence they occurred.
The external AM reference is a symbolic role: a coordinator must bind its actual
case, overlay, sealed outputs and hashes before execution. A validated plan does
not prove input authority, comparison success, conditioning or learning.
All public results are independently constructed mutable copies.
"""
from __future__ import annotations
SCHEMA = 'phase4an-engineering-plan-draft-v1'
OVERLAYS = ('zero', 'static25')
MODES = ('reference', 'light')
CHAIN_LENGTHS = (5, 13, 1, 481, 500)
PREFIX_ENDS = (18, 19, 500)
def _fresh_id(overlay, mode, repeat=0):
return f'{overlay}--fresh--{mode}-r{repeat}'
def _prefix_id(overlay, end):
return f'{overlay}--prefix--reference-t{end:04d}'
def _chain_id(overlay, mode, index):
return f'{overlay}--chain--{mode}-s{index:02d}'
def _comparison(check, row, *, reference=None, external=None, scope, candidates=None):
"""Exact descriptors; start/stop are always absolute, half-open ticks."""
return dict(check=check, reference_row_id=reference, external_reference=external,
candidate_row_ids=[row['row_id']] if candidates is None else list(candidates),
start=row['start'], stop=row['stop'], scope=scope)
def _row(identity, overlay, mode, kind, start, stop, parent=None):
return dict(row_id=identity, overlay=overlay, mode=mode, kind=kind,
start=start, stop=stop, steps=stop - start, parent_row_id=parent,
expected_comparisons=[])
def _terminal_reference(overlay, stop):
if stop == 5:
return _chain_id(overlay, 'reference', 0)
if stop == 1000:
return _fresh_id(overlay, 'reference')
return _prefix_id(overlay, stop)
def build_plan():
"""Return the sole current proposal: 32 calls / 11,074 full-graph ticks.
kind='chain' with parent_row_id=None is its fresh first segment, NOT a
continuation from an assumed zero checkpoint. Later segments explicitly
reference their predecessor. A chain's local request/event coordinates
are translated using start; terminal.next_tick remains its local phase.
All fresh/prefix rows globally precede chains. Reference chains precede
light chains, allowing the reference first segment to supply prefix 5.
No extra prefix-5 execution, synthetic 52-call matrix, long idle run,
input selection, training, parameter update or executor is implemented.
Comparison scopes:
reference_full/reference_prefix/reference_slice: include reference
tick-state hashes, APL traces, queues, requests, events and states;
light_recorded: requests.csv, events/counts, final state, APL state,
normalized pending queues and finite/weight/arrival accounting;
terminal_state_and_normalized_queues: no comparison of segment counts
against cumulative prefix counts; counts are handled by chain_exact;
parent_state_and_pending_queues: bound parent restoration/carry checks.
A comparator must implement these scopes from artifacts, not this plan's
existence. Shift local queue headers before comparing absolute timelines.
"""
rows = []
for overlay in OVERLAYS:
for mode, repeat in (('reference', 0), ('light', 0), ('light', 1)):
row = _row(_fresh_id(overlay, mode, repeat), overlay, mode, 'fresh', 0, 1000)
scope = 'reference_full' if mode == 'reference' else 'light_recorded'
row['expected_comparisons'].append(_comparison(
'am_exact', row, external='sealed_AM_selected_case_same_overlay_r0', scope=scope))
if mode == 'light':
reference = _fresh_id(overlay, 'reference' if repeat == 0 else 'light')
row['expected_comparisons'].append(_comparison(
'fresh_exact', row, reference=reference, scope='light_recorded'))
rows.append(row)
for end in PREFIX_ENDS:
row = _row(_prefix_id(overlay, end), overlay, 'reference', 'prefix', 0, end)
row['expected_comparisons'].append(_comparison(
'prefix_of_reference', row, reference=_fresh_id(overlay, 'reference'), scope='reference_prefix'))
rows.append(row)
for overlay in OVERLAYS:
for mode in MODES:
start, parent, members = 0, None, []
for index, length in enumerate(CHAIN_LENGTHS):
row = _row(_chain_id(overlay, mode, index), overlay, mode, 'chain', start, start + length, parent)
members.append(row['row_id'])
if mode == 'reference':
row['expected_comparisons'].append(_comparison(
'segment_of_reference', row, reference=_fresh_id(overlay, 'reference'), scope='reference_slice'))
else:
row['expected_comparisons'].append(_comparison(
'segment_exact', row, reference=_chain_id(overlay, 'reference', index), scope='light_recorded'))
terminal = _terminal_reference(overlay, row['stop'])
if terminal != row['row_id']:
row['expected_comparisons'].append(_comparison(
'checkpoint_exact', row, reference=terminal, scope='terminal_state_and_normalized_queues'))
if parent is not None:
row['expected_comparisons'].append(_comparison(
'parent_carry', row, reference=parent, scope='parent_state_and_pending_queues'))
if index == len(CHAIN_LENGTHS) - 1:
comparison = _comparison('chain_exact', row, reference=_fresh_id(overlay, mode),
scope='reference_full' if mode == 'reference' else 'light_recorded', candidates=members)
comparison['start'] = 0
row['expected_comparisons'].append(comparison)
rows.append(row)
start, parent = row['stop'], row['row_id']
return dict(schema=SCHEMA, overlays=list(OVERLAYS), chain_lengths=list(CHAIN_LENGTHS),
rows=rows, total_calls=32, total_ticks=11074,
training=False, candidate_generated=False)
def _exact(actual, expected, path):
"""JSON-shaped values with exact Python types; True is not integer 1."""
if type(actual) is not type(expected):
raise ValueError(f'Plan type mismatch at {path}')
if type(expected) is dict:
if any(type(key) is not str for key in actual) or set(actual) != set(expected):
raise ValueError(f'Plan fields mismatch at {path}')
for key in expected:
_exact(actual[key], expected[key], path + '.' + key)
elif type(expected) is list:
if len(actual) != len(expected):
raise ValueError(f'Plan inventory length mismatch at {path}')
for index, (left, right) in enumerate(zip(actual, expected)):
_exact(left, right, f'{path}[{index}]')
elif actual != expected:
raise ValueError(f'Plan value mismatch at {path}')
def validate_plan(plan):
"""Reject any noncanonical plan; return a new copy, never a PASS receipt.
Exact canonical reconstruction rejects unknown fields, order changes,
omitted/extra overlays or calls, interval gaps, false parent links and
weakening/removal of required comparisons. No input files are read.
"""
canonical = build_plan()
_exact(plan, canonical, 'plan')
return canonical
@@ -0,0 +1,532 @@
"""Read-only AM authority verification and exclusive AN engineering preparation.
Public prepare(model_manifest_sha, experiment_manifest_sha, final_sha) requires
three externally recorded, actual AM evidence hashes. It verifies both complete
AM seals once, checks their declared gates, then fixes the first planned r0 case.
It creates only templates/requests/overlay descriptors and a preregistration;
never a GPU trajectory or fictitious future checkpoint hash. The caller records
the returned preregistration SHA outside this experiment before execution.
read_registration(expected_sha) rehashes the exact used source/input inventory
and control anchors without rerunning either scientific gate or full AM seal.
Future spec.json/checkpoint receipts are runtime artifacts, not preregistered
bytes. No frozen model module globals are modified; no Torch is imported.
"""
from __future__ import annotations
import hashlib
import importlib.util
import json
from pathlib import Path, PurePosixPath
import re
import shutil
import sys
import numpy as np
ROOT = Path('/data/digital-drosophila')
MODEL = Path(__file__).resolve().parent
EXP = ROOT / 'experiments/phase4an-engineering-validation'
AM_MODEL_REL = Path('models/male-cns-lif/wholebrain-plastic-interface-v1')
AM_EXP_REL = Path('experiments/phase4am-wholebrain-plastic-interface')
AK_REL = Path('experiments/phase4ak-apl-timing-validation')
AI_HELPER_REL = Path('models/male-cns-lif/pn-recruitment-audit-v1/publication.py')
AI_HELPER_SHA = '7a79c499098b74163e7bb16adabcbc54e7069dafa204d0b99c7f4c03d79f8016'
AM_PREREG_SHA = '465886b78ada08092c2c83de278844707499718977fb6dc6a7608391da0e5304'
AM_OVERLAY_SHA = '7f891d0ba79b393c96e75fd2dee2eddc51e57d8213eaac2779703bc2c18712b4'
FINAL_STATUS = 'PASS-EXACT-WHOLEBRAIN-PLASTIC-INTERFACE-NOT-LEARNING'
REG_SCHEMA = 'phase4an-engineering-registration-v1'
BUDGET = {'output_budget_bytes': 8 * 1024**3, 'minimum_free_reserve_bytes': 10 * 1024**3}
MODULES = ('runtime_light', 'runtime_continuation', 'engineering_inputs', 'engineering_plan',
'engineering_compare', 'engineering_queue_audit', 'engineering_prepare', 'engineering_workflow',
'engineering_scopes', 'engineering_analysis')
SOURCE_FILES = tuple(sorted([name + '.py' for name in MODULES] + ['test_' + name + '.py' for name in MODULES] + ['ENGINEERING_GATE_DRAFT.md']))
REFERENCES = ('auditory-10hz', 'auditory-100hz', 'olfactory-50hz', 'mechanosensory-50hz', 'gustatory-50hz')
GRAPH_FILES = ('nodes.npy', 'edge_id.npy', 'src.npy', 'dst.npy', 'signed_raw.npy', 'ptr.npy', 'weights.npy')
APL_FILES = ('input_matrix.npy', 'output_matrix.npy', 'kc_globals.npy', 'apl_globals.npy')
_MODULE_CACHE = {}
def require(ok, message):
if not ok:
raise ValueError(message)
def digest(path):
value = hashlib.sha256()
with Path(path).open('rb') as stream:
for block in iter(lambda: stream.read(8 << 20), b''):
value.update(block)
return value.hexdigest()
def sha(value):
return type(value) is str and re.fullmatch('[0-9a-f]{64}', value) is not None
def read(path):
def unique(pairs):
result = {}
for key, value in pairs:
require(key not in result, 'Duplicate JSON key: ' + key)
result[key] = value
return result
return json.loads(Path(path).read_text(encoding='utf-8'), object_pairs_hook=unique,
parse_constant=lambda value: (_ for _ in ()).throw(ValueError('Nonfinite JSON: ' + value)))
def save(path, value):
path = Path(path)
path.parent.mkdir(parents=True, exist_ok=True)
with path.open('x', encoding='utf-8') as stream:
json.dump(value, stream, sort_keys=True, indent=2, ensure_ascii=False, allow_nan=False)
stream.write('\n')
def load_module(name, path, expected):
path = Path(path).resolve(strict=True)
raw = path.read_bytes()
require(hashlib.sha256(raw).hexdigest() == expected, 'Source pin mismatch: ' + str(path))
identity = str(path), expected
if name in sys.modules:
require(_MODULE_CACHE.get(name) == (identity, sys.modules[name]), 'Unverified module namespace collision')
return sys.modules[name]
spec = importlib.util.spec_from_file_location(name, path)
module = importlib.util.module_from_spec(spec)
sys.modules[name] = module
try:
exec(compile(raw, str(path), 'exec'), module.__dict__)
except BaseException:
sys.modules.pop(name, None)
raise
_MODULE_CACHE[name] = identity, module
return module
def source_inventory(model):
model = Path(model).resolve(strict=True)
result = {}
for name in SOURCE_FILES:
path = model / name
require(path.is_file() and not path.is_symlink() and path.resolve().parent == model, 'Required engineering source missing/linked: ' + name)
result[str(path)] = digest(path)
return result
def _relative(value):
require(type(value) is str and value and '\\' not in value, 'Invalid relative inventory path')
path = PurePosixPath(value)
require(not path.is_absolute() and str(path) == value and all(part not in ('', '.', '..') and ':' not in part for part in path.parts),
'Noncanonical relative inventory path')
return path
def _safe_file(root, relative):
relative = _relative(relative)
root = Path(root).resolve(strict=True)
path = root
for part in relative.parts:
path = path / part
require(not path.is_symlink(), 'Linked authority/input file')
require(path.is_file() and path.resolve().is_relative_to(root), 'Missing or escaping input file')
return path
def _index(rows):
require(type(rows) is list and rows, 'Missing sealed inventory rows')
result = {}
for row in rows:
require(type(row) is dict and set(row) == {'path', 'bytes', 'sha256'} and
type(row['bytes']) is int and row['bytes'] >= 0 and sha(row['sha256']), 'Invalid sealed inventory row')
_relative(row['path'])
require(row['path'] not in result, 'Duplicate sealed inventory row')
result[row['path']] = row
return result
def _bind(inputs, path, expected=None):
path = Path(path).resolve(strict=True)
actual = digest(path)
require(expected is None or actual == expected, 'Authority/input SHA mismatch: ' + str(path))
require(str(path) not in inputs or inputs[str(path)] == actual, 'Conflicting authority for input')
inputs[str(path)] = actual
return path
def _sealed_json(experiment, index, relative):
require(relative in index, 'Required authority absent from AM seal: ' + relative)
path = _safe_file(experiment, relative)
require(digest(path) == index[relative]['sha256'], 'Sealed JSON changed: ' + relative)
return read(path)
def validate_final_status(final, scientific, before, after, refs):
"""Check actual AM final/summary/report schemas, never infer scientific PASS."""
require(final.get('status') == scientific.get('status') == FINAL_STATUS, 'AM final status is not the required completed interface gate')
for key in ('zero_overlay_exact', 'static25_repeat_exact', 'old_integrity_pass', 'frozen_five_reference_pass',
'frozen_eligibility_and_mask_unchanged', 'base_weights_unchanged'):
require(final.get(key) is True, 'AM final gate failed/missing: ' + key)
require(final.get('primary_trials_per_stage') == 120 and final.get('repeats_per_stage') == 24 and
scientific.get('primary_trials_per_stage') == 120 and scientific.get('technical_repeats_per_stage') == 24,
'AM complete coverage counts changed')
for value in (final, scientific):
require(value.get('training') is False and value.get('learned_candidate') is False, 'AM interface must not claim training/learning')
require(scientific.get('zero_overlay_exact') is True and scientific.get('static25_repeat_exact') is True, 'Scientific summary gate mismatch')
require(type(before) is dict and type(after) is dict and len(before) == len(after) == 28 and set(before) == set(after),
'AM must retain the same complete 28-root old-integrity inventory')
for name in before:
for report in (before[name], after[name]):
require(type(report) is dict and report.get('pass') is True, 'Old-integrity gate failed: ' + name)
if 'mismatches' in report:
require(report['mismatches'] == [], 'Old-integrity mismatch list is nonempty')
for key in ('manifest_sha256', 'checksums_sha256'):
if key in before[name] or key in after[name]:
require(sha(before[name].get(key)) and before[name][key] == after[name].get(key), 'Old-root control changed: ' + name)
require(refs.get('pass_all') is True and type(refs.get('trials')) is list and
[row.get('trial') for row in refs['trials']] == list(REFERENCES), 'Five old deterministic references are incomplete or reordered')
for row in refs['trials']:
require(type(row.get('exact')) is dict and set(row['exact']) == {'step', 'neuron_idx', 'counts', 'v', 'g'} and
all(value is True for value in row['exact'].values()), 'Old reference arrays are not exact')
def validate_planned(planned):
require(type(planned) is list and len(planned) == 144, 'AM planned coverage must be 144')
keys = []
for row in planned:
require(type(row) is dict and type(row.get('name')) is str and re.fullmatch('[A-Za-z0-9_-]+', row['name']) and
row.get('cohort') in ('seed_validation', 'novel_to_aj') and type(row.get('repeat')) is int and row['repeat'] in (0, 1),
'Invalid AM planned row identity')
keys.append((row['name'], row['repeat']))
require(len(set(keys)) == 144 and sum(row['repeat'] == 0 for row in planned) == 120 and sum(row['repeat'] == 1 for row in planned) == 24,
'AM primary/repeat set incomplete')
require(all((name, 0) in set(keys) for name, repeat in keys if repeat == 1), 'AM repeat lacks primary')
return next(dict(row) for row in planned if row['repeat'] == 0)
def validate_stage(experiment, index, stage, planned, root):
"""Validate all 144 declared results against the already verified AM seal."""
receipt = _sealed_json(experiment, index, 'audit/' + stage + '-complete.json')
require(receipt.get('stage') == stage and receipt.get('pass_all') is True and receipt.get('trials') == 144 and
receipt.get('primary_trials') == 120 and receipt.get('technical_repeats') == 24 and receipt.get('training') is False,
'AM stage is incomplete: ' + stage)
completed = receipt.get('completed')
require(type(completed) is list and len(completed) == 144, 'AM stage completion inventory truncated')
for row, done in zip(planned, completed):
require(done.get('name') == row['name'] and done.get('repeat') == row['repeat'], 'AM stage completion order/identity mismatch')
prefix = 'gpu/' + stage + '/' + row['name'] + '-r' + str(row['repeat']) + '/'
expected = [dict(path=key[len(prefix):], bytes=value['bytes'], sha256=value['sha256']) for key, value in sorted(index.items())
if key.startswith(prefix) and key != prefix + 'output-hashes.json']
inventory = _sealed_json(experiment, index, prefix + 'output-hashes.json')
require(inventory == expected and inventory, 'AM completed output inventory differs from sealed payload')
require(done.get('output_hashes_sha256') == index[prefix + 'output-hashes.json']['sha256'] and
done.get('comparison_sha256') == index[prefix + 'comparison.json']['sha256'], 'AM stage receipt SHA mismatch')
summary = _sealed_json(experiment, index, prefix + 'summary.json')
compare = _sealed_json(experiment, index, prefix + 'comparison.json')
folder, case = experiment / prefix.rstrip('/'), root / AK_REL / 'cases' / row['cohort'] / row['name']
require(summary.get('case') == row['name'] and summary.get('complete') is True and summary.get('pass_all') is True and
summary.get('all_finite') is True and summary.get('steps') == summary.get('completed_ticks') == 1000,
'AM selected stage contains a failed/truncated trial')
require(compare.get('pass_all') is True and compare.get('overlay_name') == stage and compare.get('case_dir') == str(case) and
compare.get('right') == str(folder), 'AM comparison identity/gate mismatch')
require(type(compare.get('checks')) is dict and compare['checks'] and all(value is True for value in compare['checks'].values()) and
compare.get('failed_checks') == [] and 'error' not in compare and 'read_manifest_error' not in compare and
compare.get('full_queue_reconstructed') is True, 'AM comparison has a hidden negative/missing numerical gate')
mode = 'zero' if stage == 'zero' else ('repeat' if row['repeat'] else 'self')
left = root / AK_REL / 'gpu' / row['cohort'] / (row['name'] + '-r' + str(row['repeat'])) if stage == 'zero' else (
experiment / 'gpu' / stage / (row['name'] + '-r0') if row['repeat'] else folder)
require(compare.get('mode') == mode and compare.get('left') == str(left), 'AM comparison reference/mode mismatch')
return receipt
def _layout(root, model, experiment):
root, model = Path(root).resolve(strict=True), Path(model).resolve(strict=True)
experiment = Path(experiment).resolve(strict=False)
require(experiment == root / 'experiments/phase4an-engineering-validation' and model.is_relative_to(root / 'models/male-cns-lif'),
'Unexpected project/output layout')
require(not experiment.exists() or (experiment.is_dir() and not any(experiment.iterdir())), 'Refusing populated experiment; preserve all partials')
return root, model, experiment
def _budget(experiment):
experiment = Path(experiment)
used = sum(path.stat().st_size for path in experiment.rglob('*') if path.is_file()) if experiment.exists() else 0
require(used < BUDGET['output_budget_bytes'], '8 GiB engineering budget exhausted; preserve partials')
free = shutil.disk_usage(experiment.parent).free
require(free >= BUDGET['minimum_free_reserve_bytes'] + BUDGET['output_budget_bytes'] - used,
'Insufficient capacity for remaining 8 GiB budget plus 10 GiB free reserve')
return dict(BUDGET, used_bytes=used, free_bytes=free)
def budget():
return _budget(EXP)
def verify_authority(anchors):
"""Full read-only AM seal recheck for the finalizer; no gate rerun."""
require(sys.dont_write_bytecode, 'Use python -B for immutable authority verification')
root = ROOT.resolve(strict=True)
values = [(root / AM_MODEL_REL, 'am_model_manifest_sha256'), (root / AM_EXP_REL, 'am_experiment_manifest_sha256')]
for directory, key in values:
require(sha(anchors.get(key)) and digest(directory / 'manifest.json') == anchors[key], 'External AM manifest anchor mismatch')
require(sha(anchors.get('am_final_sha256')) and digest(root / AM_EXP_REL / 'audit/final-status.json') == anchors['am_final_sha256'],
'External AM final anchor mismatch')
require(anchors.get('am_preregistration_sha256') == AM_PREREG_SHA and
digest(root / AM_EXP_REL / 'protocol/preregistered.json') == AM_PREREG_SHA, 'AM preregistration anchor mismatch')
require(anchors.get('AI_publication_helper_sha256') == AI_HELPER_SHA, 'Publication helper authority changed')
publication = load_module('phase4an_pinned_AI_publication', root / AI_HELPER_REL, AI_HELPER_SHA)
result = {'model': publication.verify_seal(values[0][0]), 'experiment': publication.verify_seal(values[1][0])}
require(result['model']['manifest_sha256'] == anchors['am_model_manifest_sha256'] and
result['experiment']['manifest_sha256'] == anchors['am_experiment_manifest_sha256'], 'AM seal changed during verification')
return result
def _old_integrity(root, context_sha, expected):
"""Reuse the pinned AM read-only integrity chain, not a brain runtime."""
context_path = root / AM_MODEL_REL / 'context.py'
previous_path = list(sys.path)
torch_before = 'torch' in sys.modules
try:
context = load_module('phase4an_sealed_AM_integrity', context_path, context_sha)
require(context.ROOT.resolve() == root.resolve(), 'AM integrity helper belongs to another root')
result = context.integrity()
finally:
sys.path[:] = previous_path
require(torch_before or 'torch' not in sys.modules, 'Integrity unexpectedly imported Torch')
require(type(result) is dict and len(result) == 28 and set(result) == set(expected) and
all(type(row) is dict and row.get('pass') is True for row in result.values()), 'Current 28-root old-integrity gate failed')
for name, row in result.items():
if 'mismatches' in row:
require(row['mismatches'] == [], 'Current old-integrity mismatch list is nonempty')
for key in ('manifest_sha256', 'checksums_sha256', 'reference_sha256'):
if key in expected[name] or key in row:
require(sha(row.get(key)) and row[key] == expected[name].get(key), 'Old integrity authority changed: ' + name)
return result
def verify_old_integrity(registration):
"""Rehash the pinned 28 old roots/snapshot via original CPU-only code."""
require(sys.dont_write_bytecode, 'Use python -B for old-integrity verification')
path = ROOT / AM_MODEL_REL / 'context.py'
expected_sha = registration['input_sha256'].get(str(path.resolve(strict=True)))
require(sha(expected_sha) and digest(path) == expected_sha, 'AM integrity source is not a registered input')
before_path = EXP / 'regression/old-integrity-before.json'
require(digest(before_path) == registration['input_sha256'].get(str(before_path.resolve(strict=True))), 'Before-integrity evidence changed')
return _old_integrity(ROOT, expected_sha, read(before_path))
def _prepare(root, model, experiment, am_model_manifest_sha, am_experiment_manifest_sha, am_final_sha):
require(sys.dont_write_bytecode, 'Preparation requires python -B; old sources must remain byte-for-byte read-only')
require(all(sha(value) for value in (am_model_manifest_sha, am_experiment_manifest_sha, am_final_sha)), 'Three actual external AM SHA-256 anchors are mandatory')
root, model, experiment = _layout(root, model, experiment)
_budget(experiment)
sources = source_inventory(model)
inputs_module = load_module('phase4an_prepare_inputs', model / 'engineering_inputs.py', sources[str(model / 'engineering_inputs.py')])
plan_module = load_module('phase4an_prepare_plan', model / 'engineering_plan.py', sources[str(model / 'engineering_plan.py')])
runtime = load_module('phase4an_prepare_light', model / 'runtime_light.py', sources[str(model / 'runtime_light.py')])
publication = load_module('phase4an_pinned_AI_publication', root / AI_HELPER_REL, AI_HELPER_SHA)
am_model, am = root / AM_MODEL_REL, root / AM_EXP_REL
require(digest(am_model / 'manifest.json') == am_model_manifest_sha and digest(am / 'manifest.json') == am_experiment_manifest_sha and
digest(am / 'audit/final-status.json') == am_final_sha, 'External AM authority mismatch')
verified = {'model': publication.verify_seal(am_model), 'experiment': publication.verify_seal(am)}
require(verified['model']['manifest_sha256'] == am_model_manifest_sha and verified['experiment']['manifest_sha256'] == am_experiment_manifest_sha,
'AM controls changed during full seal verification')
idx = _index(verified['experiment']['files']); midx = _index(verified['model']['files'])
reg = _sealed_json(am, idx, 'protocol/preregistered.json')
require(idx['protocol/preregistered.json']['sha256'] == AM_PREREG_SHA, 'AM preregistration authority differs from the fixed anchor')
planned = reg.get('planned'); selected = validate_planned(planned)
final = _sealed_json(am, idx, 'audit/final-status.json')
scientific = _sealed_json(am, idx, 'audit/scientific-summary.json')
require(final.get('scientific_summary_sha256') == idx['audit/scientific-summary.json']['sha256'], 'AM final does not bind scientific summary')
before = _sealed_json(am, idx, 'regression/old-integrity-before.json')
after = _sealed_json(am, idx, 'regression/old-integrity-after.json')
refs = _sealed_json(am, idx, 'regression/frozen-five.json')
validate_final_status(final, scientific, before, after, refs)
for stage in ('zero', 'static25'):
validate_stage(am, idx, stage, planned, root)
current_old = _old_integrity(root, midx['context.py']['sha256'], after)
inputs = {}
anchors = {'am_model_manifest_sha256': am_model_manifest_sha, 'am_experiment_manifest_sha256': am_experiment_manifest_sha,
'am_final_sha256': am_final_sha, 'am_preregistration_sha256': AM_PREREG_SHA, 'AI_publication_helper_sha256': AI_HELPER_SHA}
for path, expected in ((am_model / 'manifest.json', am_model_manifest_sha), (am / 'manifest.json', am_experiment_manifest_sha),
(am / 'audit/final-status.json', am_final_sha), (root / AI_HELPER_REL, AI_HELPER_SHA),
(am_model / 'overlay.py', AM_OVERLAY_SHA)):
_bind(inputs, path, expected)
require(midx.get('overlay.py', {}).get('sha256') == AM_OVERLAY_SHA, 'AM model seal does not contain the pinned overlay helper')
for name in ('context.py', 'config.json'):
_bind(inputs, am_model / name, midx[name]['sha256'])
for path in (am_model / 'checksums.sha256', am / 'checksums.sha256'):
_bind(inputs, path)
for relative in ('protocol/preregistered.json', 'protocol/prepared-hashes.json', 'audit/scientific-summary.json', 'audit/zero-complete.json', 'audit/static25-complete.json',
'regression/old-integrity-before.json', 'regression/old-integrity-after.json', 'regression/frozen-five.json'):
_bind(inputs, am / relative, idx[relative]['sha256'])
am_inputs = {}
for row in reg['inputs']:
require(set(row) == {'path', 'sha256'} and sha(row['sha256']), 'Invalid AM registered input row')
_relative(row['path']); require(row['path'] not in am_inputs, 'Duplicate AM registered input')
am_inputs[row['path']] = row['sha256']
def authority(path):
path = Path(path).resolve(strict=True)
require(path.is_relative_to(root), 'AM input leaves project root')
relative = path.relative_to(root).as_posix()
if path.is_relative_to(am):
expected = idx[path.relative_to(am).as_posix()]['sha256']
else:
require(relative in am_inputs, 'Dependency absent from AM preregistration: ' + relative)
expected = am_inputs[relative]
return _bind(inputs, path, expected)
original_dir = root / AK_REL / 'cases' / selected['cohort'] / selected['name']
authority(original_dir / 'spec.json'); authority(original_dir / 'requests.npz')
original = read(original_dir / 'spec.json')
require(original['steps'] == 1000 and original['kind'] == 'fullgraph' and original['name'] == selected['name'], 'Fixed selected AM case changed')
with np.load(original_dir / 'requests.npz', allow_pickle=False) as archive:
require(set(archive.files) == {'tick', 'neuron_idx'} and len(archive.files) == 2, 'Unexpected original request arrays')
requests = {key: archive[key].copy() for key in archive.files}
inputs_module.validate_requests(requests, 1000)
for name in GRAPH_FILES:
authority(Path(original['graph_dir']) / name)
for name in APL_FILES:
authority(Path(original['apl_bundle_dir']) / name)
authority(original['coefficients_path'])
selected_outputs = {}
for stage in ('zero', 'static25'):
prefix = 'gpu/' + stage + '/' + selected['name'] + '-r0/'
selected_outputs[stage] = str(am / prefix.rstrip('/'))
for relative, row in idx.items():
if relative.startswith(prefix):
_bind(inputs, am / relative, row['sha256'])
adapted = {}
for stage in ('zero', 'static25'):
old_manifest = authority(am / 'overlays' / (stage + '.json'))
overlay_file = authority(am / 'overlays' / (stage + '.npz'))
declaration = read(old_manifest)
require(declaration['overlay_name'] == stage and declaration['overlay_sha256'] == digest(overlay_file), 'AM overlay declaration mismatch')
for label in ('mask', 'base_candidate'):
require(digest(authority(declaration[label + '_path'])) == declaration[label + '_sha256'], 'Overlay external array authority mismatch')
adapted[stage] = inputs_module.adapt_overlay_manifest(declaration, str(am_model / 'overlay.py'))
require(adapted[stage]['am_overlay_sha256'] == AM_OVERLAY_SHA, 'Adapted helper pin differs')
for relative, expected in runtime.PINS.items():
_bind(inputs, model.parent / relative, expected)
plan = plan_module.validate_plan(plan_module.build_plan())
require(plan['total_calls'] == 32 and plan['total_ticks'] == 11074, 'Engineering plan matrix changed')
# No writes have occurred before all authority/source/space checks above.
experiment.mkdir(parents=True, exist_ok=True)
try:
for folder in ('protocol', 'cases', 'overlays', 'audit', 'runs', 'receipts', 'logs', 'regression', 'extension-build'):
(experiment / folder).mkdir(exist_ok=False)
save(experiment / 'regression/old-integrity-before.json', current_old)
_bind(inputs, experiment / 'regression/old-integrity-before.json')
for source, expected in sources.items():
target = experiment / 'protocol/source-snapshot' / Path(source).name
publication._copy_verified(Path(source), target, Path(source).stat().st_size, expected)
_bind(inputs, target, expected)
save(experiment / 'protocol/AM-seal-verification.json', verified)
save(experiment / 'protocol/AM-gate-authority.json', dict(anchors=anchors, all_144_per_stage_declared_gates_pass=True,
old_roots=28, old_references=list(REFERENCES), verification_scope='Full AM seals and finalized declared gate evidence; no new scientific execution'))
for name in ('AM-seal-verification.json', 'AM-gate-authority.json'):
_bind(inputs, experiment / 'protocol' / name)
overlays, cases = {}, {}
for stage, value in adapted.items():
manifest_path = experiment / 'overlays' / (stage + '.json')
save(manifest_path, value)
_bind(inputs, manifest_path)
overlays[stage] = {'overlay_file': str(am / 'overlays' / (stage + '.npz')), 'overlay_manifest': str(manifest_path)}
for row in plan['rows']:
case = experiment / 'cases' / row['row_id']; case.mkdir(exist_ok=False)
template = {'schema': 'phase4an-engineering-case-template-v1', 'row_id': row['row_id'], 'original_spec': original,
'start': row['start'], 'stop': row['stop'], 'budget_root': str(experiment), 'output_budget_bytes': BUDGET['output_budget_bytes']}
save(case / 'template-spec.json', template)
part = inputs_module.slice_requests(requests, 1000, row['start'], row['stop'])
with (case / 'requests.npz').open('xb') as stream:
np.savez_compressed(stream, **part)
_bind(inputs, case / 'template-spec.json'); _bind(inputs, case / 'requests.npz')
cases[row['row_id']] = {'case_dir': str(case), 'template_spec_sha256': digest(case / 'template-spec.json'),
'requests_sha256': digest(case / 'requests.npz')}
require(source_inventory(model) == sources and all(digest(path) == expected for path, expected in inputs.items()), 'Source/input changed during preparation')
# Repeat the full AM verification after producing only small new inputs.
after_verified = {'model': publication.verify_seal(am_model), 'experiment': publication.verify_seal(am)}
require(after_verified == verified, 'AM sealed payload changed during preparation')
_budget(experiment)
registration = {'schema': REG_SCHEMA, 'plan': plan, 'selected_case': {'row': selected, 'original_dir': str(original_dir), 'AM_outputs': selected_outputs},
'source_sha256': sources, 'input_sha256': inputs, 'cases': cases, 'overlays': overlays, 'budget': dict(BUDGET), 'anchors': anchors,
'training': False, 'new_brain_runs_during_preparation': 0, 'future_checkpoint_hashes_fabricated': False,
'runtime_spec_derivation': 'engineering_inputs.derive_spec(template.original_spec, interval, budget, actual verified parent SHA or None)',
'scope': 'Fixed 32-call engineering gate only; not conditioning or learning registration'}
save(experiment / 'protocol/preregistered.json', registration)
return {'prepared': True, 'registration_path': str(experiment / 'protocol/preregistered.json'),
'registration_sha256': digest(experiment / 'protocol/preregistered.json'), 'calls': 32, 'brain_runs': 0}
except Exception as error:
failure = experiment / 'audit/preparation-failure.json'
if not failure.exists():
save(failure, {'prepared': False, 'type': type(error).__name__, 'message': str(error), 'partial_outputs_preserved': True})
raise
def prepare(am_model_manifest_sha, am_experiment_manifest_sha, am_final_sha):
return _prepare(ROOT, MODEL, EXP, am_model_manifest_sha, am_experiment_manifest_sha, am_final_sha)
def row_path(registration, row_id):
require(type(row_id) is str and row_id in {row['row_id'] for row in registration['plan']['rows']}, 'Unknown engineering row ID')
expected = EXP.resolve(strict=True) / 'cases' / row_id
actual = Path(registration['cases'][row_id]['case_dir'])
require(actual == expected and actual.resolve(strict=True) == expected and actual.is_dir(), 'Row case path is not its fixed safe mapping')
return actual
def read_registration(expected_sha):
require(sys.dont_write_bytecode and sha(expected_sha), 'Require python -B and an external preregistration SHA')
path = EXP / 'protocol/preregistered.json'
require(digest(path) == expected_sha, 'Preregistration differs from external authority')
require(not (EXP / 'audit/preparation-failure.json').exists(), 'Preparation failed; preserve partials')
registration = read(path)
require(registration.get('schema') == REG_SCHEMA and registration.get('training') is False and registration.get('budget') == BUDGET,
'Unexpected registration schema/scope/budget')
require(registration['source_sha256'] == source_inventory(MODEL), 'Registered engineering sources changed or inventory incomplete')
for name, expected in registration['input_sha256'].items():
require(Path(name).is_absolute() and sha(expected) and digest(name) == expected, 'Registered input changed: ' + name)
plan_module = load_module('phase4an_read_plan', MODEL / 'engineering_plan.py', registration['source_sha256'][str(MODEL / 'engineering_plan.py')])
inputs_module = load_module('phase4an_read_inputs', MODEL / 'engineering_inputs.py', registration['source_sha256'][str(MODEL / 'engineering_inputs.py')])
plan_module.validate_plan(registration['plan'])
selected = validate_planned(read(ROOT / AM_EXP_REL / 'protocol/preregistered.json')['planned'])
original_dir = ROOT / AK_REL / 'cases' / selected['cohort'] / selected['name']
require(registration['selected_case']['row'] == selected and registration['selected_case']['original_dir'] == str(original_dir),
'Registered case is not the fixed first AM planned primary')
original = read(original_dir / 'spec.json')
with np.load(original_dir / 'requests.npz', allow_pickle=False) as archive:
require(set(archive.files) == {'tick', 'neuron_idx'} and len(archive.files) == 2, 'Unexpected original sparse request schema')
requests = {key: archive[key].copy() for key in archive.files}
inputs_module.validate_requests(requests, 1000)
require(set(registration['cases']) == {row['row_id'] for row in registration['plan']['rows']}, 'Registered case inventory differs from plan')
for row in registration['plan']['rows']:
case = row_path(registration, row['row_id']); entry = registration['cases'][row['row_id']]
require(entry['template_spec_sha256'] == digest(case / 'template-spec.json') and entry['requests_sha256'] == digest(case / 'requests.npz'),
'Template/request metadata does not bind registered bytes')
template = read(case / 'template-spec.json')
require(set(template) == {'schema', 'row_id', 'original_spec', 'start', 'stop', 'budget_root', 'output_budget_bytes'} and
template['schema'] == 'phase4an-engineering-case-template-v1' and template['original_spec'] == original and
template['row_id'] == row['row_id'] and template['start'] == row['start'] and template['stop'] == row['stop'] and
template['budget_root'] == str(EXP) and template['output_budget_bytes'] == BUDGET['output_budget_bytes'], 'Template rule differs from canonical row')
expected = inputs_module.slice_requests(requests, 1000, row['start'], row['stop'])
with np.load(case / 'requests.npz', allow_pickle=False) as archive:
require(len(archive.files) == 2 and set(archive.files) == set(expected) and
all(archive[key].dtype == expected[key].dtype and archive[key].shape == expected[key].shape and
archive[key].tobytes() == expected[key].tobytes() for key in expected), 'Registered request slice is not exact')
require(set(registration['overlays']) == {'zero', 'static25'}, 'Registered overlay inventory changed')
for stage in ('zero', 'static25'):
entry = registration['overlays'][stage]
expected_file = ROOT / AM_EXP_REL / 'overlays' / (stage + '.npz')
expected_manifest = EXP / 'overlays' / (stage + '.json')
require(entry == {'overlay_file': str(expected_file), 'overlay_manifest': str(expected_manifest)}, 'Overlay path mapping changed')
original_overlay = read(ROOT / AM_EXP_REL / 'overlays' / (stage + '.json'))
require(read(expected_manifest) == inputs_module.adapt_overlay_manifest(original_overlay, str(ROOT / AM_MODEL_REL / 'overlay.py')),
'Adapted overlay manifest is not the exact declared transformation')
anchors = registration['anchors']
for name, relative in (('am_model_manifest_sha256', AM_MODEL_REL / 'manifest.json'),
('am_experiment_manifest_sha256', AM_EXP_REL / 'manifest.json'),
('am_final_sha256', AM_EXP_REL / 'audit/final-status.json')):
require(sha(anchors[name]) and digest(ROOT / relative) == anchors[name], 'AM external anchor changed')
require(anchors['am_preregistration_sha256'] == AM_PREREG_SHA and digest(ROOT / AM_EXP_REL / 'protocol/preregistered.json') == AM_PREREG_SHA and
anchors['AI_publication_helper_sha256'] == AI_HELPER_SHA and digest(ROOT / AI_HELPER_REL) == AI_HELPER_SHA, 'Fixed AM/helper authority changed')
require(digest(path) == expected_sha, 'Registration changed during verification')
return registration
@@ -0,0 +1,243 @@
"""Independent, read-only CPU queue/ref audit for the draft AN 32-call gate.
The caller supplies externally authenticated records and CSR arrays. No file
I/O, runtime import, GPU, membrane integration, weight update or learning is
performed. Spikes are observed facts, not independently predicted here.
queue digest bytes independently implement frozen AF oracle_full.queue_summary
(source SHA below): header, little-endian signed int64 local tick/count, all
global edge IDs in source/CSR order, then uint8 post-threshold acceptance bits.
The function never target-sorts or drops zero-weight edges from the queue.
"""
from __future__ import annotations
import hashlib
import struct
import numpy as np
AF_QUEUE_SOURCE_SHA256 = 'd72003d98f0f1624b12126eff50bd361c43d888f4a64d90ba9247a58976f1961'
QUEUE_HEADER = b'phase4af-queue-v1\0'
DELAY, REFRACTORY = 18, 22
CHUNK = 1 << 18
QUEUE_KEYS = {'tick', 'count', 'accepted_count', 'ordered_edge_sha256', 'accepted_mask_sha256', 'joint_sha256'}
def require(value, message):
if not value:
raise ValueError(message)
def _array(value, dtype, shape, name):
require(isinstance(value, np.ndarray) and value.dtype == np.dtype(dtype) and value.shape == shape,
'Invalid dtype/shape: ' + name)
def _raw_sha(value):
result = hashlib.sha256()
for start in range(0, len(value), CHUNK):
result.update(np.ascontiguousarray(value[start:start + CHUNK], dtype=value.dtype.newbyteorder('<')).tobytes())
return result.hexdigest()
def _event_schema(tick, node, steps, n, label):
require(isinstance(tick, np.ndarray) and tick.ndim == 1, 'Invalid event vector: ' + label)
_array(tick, np.int32, tick.shape, label + '.tick')
_array(node, np.int32, tick.shape, label + '.neuron_idx')
require(np.all((tick >= 0) & (tick < steps)) and np.all((node >= 0) & (node < n)),
'Out-of-range event: ' + label)
key = tick.astype(np.int64) * n + node
require(len(key) < 2 or np.all(key[1:] > key[:-1]), 'Events must be sorted/unique: ' + label)
def _record_schema(record, n, label):
require(isinstance(record, dict), 'Record must be a dictionary: ' + label)
steps, offset = record['steps'], record['global_tick_offset']
require(type(steps) is int and 0 < steps <= np.iinfo(np.int32).max and
type(offset) is int and offset >= 0, 'Invalid record clock: ' + label)
state = record['state']
require(set(state) == {'step', 'neuron_idx', 'counts', 'v', 'g', 'ref'}, 'Invalid state keys: ' + label)
_event_schema(state['step'], state['neuron_idx'], steps, n, label)
for name, dtype in [('counts', np.int32), ('ref', np.int32), ('v', np.float64), ('g', np.float64)]:
_array(state[name], dtype, (n,), label + '.' + name)
require(np.isfinite(state[name]).all(), 'Nonfinite state: ' + label + '.' + name)
require(np.all(state['counts'] >= 0) and np.all((state['ref'] >= 0) & (state['ref'] <= REFRACTORY)),
'Invalid state count/ref: ' + label)
rows = record['requests']
require(isinstance(rows, np.ndarray) and rows.ndim == 2 and rows.shape[1] == 5,
'Invalid request table: ' + label)
_array(rows, np.int64, rows.shape, label + '.requests')
queues, hashes = record.get('queues'), record.get('state_hashes')
require(isinstance(queues, list) and isinstance(hashes, list) and len(queues) == len(hashes) == steps,
'Complete reference queue and state hashes are required: ' + label)
for t, (queue, state_hash) in enumerate(zip(queues, hashes)):
require(isinstance(queue, dict) and set(queue) == QUEUE_KEYS and type(queue['tick']) is int and queue['tick'] == t,
'Invalid local queue row: ' + label)
require(isinstance(state_hash, dict) and set(state_hash) == {'tick', 'v_sha256', 'g_sha256', 'ref_sha256'} and
type(state_hash['tick']) is int and state_hash['tick'] == t,
'Invalid local state-hash row: ' + label)
return record
def _graph(ptr, dst, edge_id, effective_weights):
require(isinstance(ptr, np.ndarray) and ptr.ndim == 1 and len(ptr) > 1, 'Invalid CSR ptr')
n, e = len(ptr) - 1, len(dst)
_array(ptr, np.int64, (n + 1,), 'ptr')
_array(dst, np.int32, (e,), 'dst')
_array(edge_id, np.int64, (e,), 'edge_id')
_array(effective_weights, np.float64, (e,), 'effective_weights_SI')
require(ptr[0] == 0 and ptr[-1] == e and np.all(ptr[1:] >= ptr[:-1]), 'Invalid CSR pointers')
for start in range(0, e, CHUNK):
stop = min(start + CHUNK, e)
require(np.all((dst[start:stop] >= 0) & (dst[start:stop] < n)), 'CSR destination out of range')
require(np.all(edge_id[start:stop] >= 0) and
np.all(np.diff(edge_id[max(0, start - 1):stop]) > 0), 'Global edge IDs must preserve canonical increasing CSR order')
require(np.isfinite(effective_weights[start:stop]).all(), 'Nonfinite effective SI weight')
return n, e
def _queue(tick, local_tick, sources, ref, ptr, dst, edge_id, weights, boundaries):
"""Two headers, shared source-ordered payload; memory bounded by one chunk."""
count = sum(int(ptr[int(s) + 1] - ptr[int(s)]) for s in sources)
joints = [hashlib.sha256(QUEUE_HEADER + struct.pack('<qq', t, count)) for t in (tick, local_tick)]
edges_hash, accepted_hash = hashlib.sha256(), hashlib.sha256()
for source in sources:
for start in range(int(ptr[source]), int(ptr[source + 1]), CHUNK):
stop = min(start + CHUNK, int(ptr[source + 1]))
data = np.ascontiguousarray(edge_id[start:stop], dtype='<i8').tobytes()
edges_hash.update(data)
for joint in joints:
joint.update(data)
accepted_count, nonzero_count, first = 0, 0, None
for source in sources:
for start in range(int(ptr[source]), int(ptr[source + 1]), CHUNK):
stop = min(start + CHUNK, int(ptr[source + 1]))
mask = np.ascontiguousarray(ref[dst[start:stop]] == 0, dtype=np.uint8)
accepted_count += int(mask.sum(dtype=np.int64))
data = mask.tobytes(); accepted_hash.update(data)
for joint in joints:
joint.update(data)
if boundaries:
live = np.flatnonzero((mask != 0) & (weights[start:stop] != 0))
nonzero_count += len(live)
if first is None and len(live):
index = start + int(live[0])
first = {'source_neuron_idx': int(source), 'target_neuron_idx': int(dst[index]),
'global_edge_id': int(edge_id[index]), 'csr_edge_index': index,
'source_spike_tick': tick - DELAY, 'arrival_tick': tick,
'effective_weight_SI_volt': float(weights[index]),
'effective_weight_float64_le_hex': np.asarray(weights[index], dtype='<f8').tobytes().hex(),
'accepted': True, 'post_threshold_target_ref': 0,
'weight_is_nonzero': True, 'evidence_is_unique_cause_or_necessity': False}
rows = [{'tick': t, 'count': count, 'accepted_count': accepted_count,
'ordered_edge_sha256': edges_hash.hexdigest(), 'accepted_mask_sha256': accepted_hash.hexdigest(),
'joint_sha256': joint.hexdigest()} for t, joint in zip((tick, local_tick), joints)]
return rows, first, nonzero_count
def audit_queues(monolithic, segments, *, ptr, dst, edge_id, effective_weights, requests, cuts):
"""Audit a fresh reference plus its complete reference segmentation.
All inputs are read-only. Return PASS/FAIL/INCOMPLETE JSON-safe evidence.
Structural input damage raises ValueError; valid but differing artifacts
return FAIL. ``cuts`` must be the exact internal segment boundaries.
No initial state is inferred for arbitrary nonzero-origin experiments.
"""
n, e = _graph(ptr, dst, edge_id, effective_weights)
mono = _record_schema(monolithic, n, 'monolithic')
require(mono['global_tick_offset'] == 0, 'This audit requires the registered fresh monolithic origin')
require(isinstance(segments, (list, tuple)) and len(segments) >= 2, 'At least two reference segments required')
parts = [_record_schema(p, n, 'segment-' + str(i)) for i, p in enumerate(segments)]
cursor, actual_cuts = 0, []
for i, part in enumerate(parts):
require(part['global_tick_offset'] == cursor, 'Segment clock gap/overlap')
if i:
actual_cuts.append(cursor)
cursor += part['steps']
require(cursor == mono['steps'] and type(cuts) in (list, tuple) and all(type(t) is int for t in cuts) and
list(cuts) == actual_cuts, 'Cuts/segments must cover the exact complete monolithic interval')
require(isinstance(requests, dict) and set(requests) == {'tick', 'neuron_idx'}, 'Exact original request arrays required')
_event_schema(requests['tick'], requests['neuron_idx'], mono['steps'], n, 'original_requests')
records = [('monolithic', mono), *[('segment-' + str(i), p) for i, p in enumerate(parts)]]
differences, difference_count = [], 0
def check(value, field, **location):
nonlocal difference_count
if not value:
difference_count += 1
if len(differences) < 100:
differences.append({'field': field, **location})
tick, node = mono['state']['step'], mono['state']['neuron_idx']
for label, record in records:
start, stop = record['global_tick_offset'], record['global_tick_offset'] + record['steps']
lo, hi = np.searchsorted(tick, (start, stop))
check(np.array_equal(record['state']['step'].astype(np.int64) + start, tick[lo:hi]), 'events.tick', record=label)
check(np.array_equal(record['state']['neuron_idx'], node[lo:hi]), 'events.neuron_idx', record=label)
check(np.array_equal(record['state']['counts'], np.bincount(record['state']['neuron_idx'], minlength=n)), 'counts', record=label)
expected_requests = np.empty((len(requests['tick']), 5), np.int64)
if len(expected_requests):
expected_requests[:, 0] = requests['tick']; expected_requests[:, 1] = requests['neuron_idx']; expected_requests[:, 2] = 1
ref = np.zeros(n, np.int32)
boundary_report = {t: {'boundary_tick': t, 'accepted_nonzero_crossing_edge_events': 0, 'first_witness': None} for t in cuts}
due_total, accepted_total, segment_index = 0, 0, 0
for t in range(mono['steps']):
while t >= parts[segment_index]['global_tick_offset'] + parts[segment_index]['steps']:
segment_index += 1
part = parts[segment_index]; local = t - part['global_tick_offset']
rlo, rhi = np.searchsorted(requests['tick'], (t, t + 1))
requested = requests['neuron_idx'][rlo:rhi]
accepted_requests = ref[requested] <= 1
expected_requests[rlo:rhi, 3] = accepted_requests
expected_requests[rlo:rhi, 4] = ~accepted_requests
np.maximum(ref - np.int32(1), np.int32(0), out=ref)
slo, shi = np.searchsorted(tick, (t, t + 1)); fired = node[slo:shi]
check(np.all(ref[fired] == 0), 'spike_during_refractory', global_tick=t)
check(np.isin(requested[accepted_requests], fired, assume_unique=True).all(), 'accepted_request_without_spike', global_tick=t)
ref[fired] = REFRACTORY
sources = node[0:0]
if t >= DELAY:
dlo, dhi = np.searchsorted(tick, (t - DELAY, t - DELAY + 1)); sources = node[dlo:dhi]
crossing = [cut for cut in cuts if t - DELAY < cut <= t]
rows, witness, live_count = _queue(t, local, sources, ref, ptr, dst, edge_id, effective_weights, crossing)
due_total += rows[0]['count']; accepted_total += rows[0]['accepted_count']
ref_sha = _raw_sha(ref)
for label, record, at, expected in [('monolithic', mono, t, rows[0]), ('segment-' + str(segment_index), part, local, rows[1])]:
for key in sorted(QUEUE_KEYS):
check(type(record['queues'][at][key]) is type(expected[key]) and record['queues'][at][key] == expected[key],
'queue.' + key, record=label, local_tick=at, global_tick=t)
check(record['state_hashes'][at]['ref_sha256'] == ref_sha, 'state_hash.ref_sha256', record=label, local_tick=at, global_tick=t)
if at == record['steps'] - 1:
check(np.array_equal(record['state']['ref'], ref), 'terminal.ref', record=label)
for cut in crossing:
boundary_report[cut]['accepted_nonzero_crossing_edge_events'] += live_count
if witness is not None and boundary_report[cut]['first_witness'] is None:
boundary_report[cut]['first_witness'] = {'boundary_tick': cut, **witness}
for label, record in records:
start, stop = record['global_tick_offset'], record['global_tick_offset'] + record['steps']
lo, hi = np.searchsorted(requests['tick'], (start, stop))
expected = expected_requests[lo:hi].copy(); expected[:, 0] -= start
check(np.array_equal(record['requests'], expected), 'requests.five_fields', record=label)
first_witness = min((row['first_witness'] for row in boundary_report.values() if row['first_witness'] is not None),
key=lambda row: (row['arrival_tick'], row['source_neuron_idx'], row['csr_edge_index'], row['boundary_tick']), default=None)
exact, live = difference_count == 0, first_witness is not None
return {'schema': 'phase4an-independent-queue-audit-v1', 'pass_all': exact and live,
'status': 'FAIL-QUEUE-OR-REF' if not exact else ('PASS-QUEUE-WITNESS-SCOPE' if live else 'INCOMPLETE-NO-ACCEPTED-NONZERO-CROSSING'),
'queue_ref_request_exact': exact, 'accepted_nonzero_crossing_witness_verified': exact and live,
'difference_count': difference_count, 'differences': differences, 'differences_truncated': difference_count > len(differences),
'neurons': n, 'edges': e, 'steps': mono['steps'], 'segments': len(parts), 'cuts': list(cuts),
'arriving_edges': due_total, 'accepted_arriving_edges': accepted_total, 'rejected_arriving_edges': due_total - accepted_total,
'requests': len(expected_requests), 'accepted_requests': int(expected_requests[:, 3].sum(dtype=np.int64)),
'rejected_requests': int(expected_requests[:, 4].sum(dtype=np.int64)),
'first_witness': first_witness, 'boundaries': list(boundary_report.values()),
'same_edge_event_may_cross_multiple_boundaries': True,
'input_array_sha256': {name: _raw_sha(value) for name, value in [('ptr', ptr), ('dst', dst), ('edge_id', edge_id),
('effective_weights_float64_SI', effective_weights), ('request_tick', requests['tick']), ('request_neuron_idx', requests['neuron_idx'])]},
'digest_definition': {'AF_source_sha256': AF_QUEUE_SOURCE_SHA256, 'header_hex': QUEUE_HEADER.hex(),
'payload_order': 'local tick/count <qq; all canonical source/CSR edge IDs <i8; all post-threshold accepted u1'},
'delay_steps': DELAY, 'refractory_steps': REFRACTORY,
'scope': 'Conditional on recorded spikes: exact ref, original requests, source-ordered real-edge arrival queues and finite-cut witness only',
'membrane_dynamics_independently_replayed': False, 'v_g_hashes_independently_verified': False,
'unique_causal_or_necessity_claim': False, 'external_seals_and_input_authority_verified_here': False,
'training': False, 'GPU_used': False}
@@ -0,0 +1,219 @@
"""Pure CPU comparisons of already-loaded engineering_compare run records.
No paths are accepted, no artifacts are opened, no GPU is imported. Structural
or missing-required-observation errors raise ValueError. Valid but different
records return numerical_exact=False. External seals, parent SHA authority and
independent CSR/local-header queue reconstruction remain coordinator duties.
"""
from __future__ import annotations
import hashlib
import numpy as np
import engineering_compare as ec
SCHEMA = 'phase4an-engineering-scopes-v1'
def _require(condition, message):
if not condition:
raise ValueError(message)
def _raw_sha(value):
return hashlib.sha256(np.ascontiguousarray(value, dtype=value.dtype.newbyteorder('<')).tobytes()).hexdigest()
def _record(value, label):
_require(type(value) is dict, label + ' must be an already-loaded run dictionary, not a path')
try:
ec.validate_run(value)
summary = value['summary']
_require(type(summary) is dict, label + ' requires summary metadata')
steps, offset = value['steps'], value['global_tick_offset']
_require(steps <= np.iinfo(np.int32).max, label + ' steps exceed int32 event capacity')
for key in ('complete', 'pass_all', 'all_finite'):
_require(summary.get(key) is True, label + ' is incomplete or failed: ' + key)
for key in ('steps', 'completed_ticks'):
_require(type(summary.get(key)) is int and summary[key] == steps, label + ' summary clock mismatch')
_require(type(summary.get('nodes')) is int and summary['nodes'] == len(value['state']['v']),
label + ' summary node count mismatch')
_require(summary.get('precision') == 'float64' and summary.get('apl_precision') == 'float32' and
type(summary.get('apl_delay_steps')) is int and summary['apl_delay_steps'] == 18,
label + ' numerical metadata mismatch')
continued = summary.get('checkpoint_continuation_used', False)
_require(type(continued) is bool and continued == (offset > 0), label + ' continuation scope mismatch')
declared_offset = summary.get('global_tick_offset', 0)
_require(type(declared_offset) is int and declared_offset == offset, label + ' summary offset mismatch')
if continued or 'global_next_tick' in summary:
_require(type(summary.get('global_next_tick')) is int and summary['global_next_tick'] == offset + steps,
label + ' summary global end mismatch')
_require(ec._sha(summary.get('effective_weights_raw_sha256')), label + ' effective weight SHA is missing')
mode = summary.get('diagnostic_mode')
legacy = str(value.get('terminal_origin', '')).startswith('AM_')
_require(mode in ('light', 'reference') or (mode is None and legacy), label + ' recording mode is not declared')
observations = {'apl_traces': bool(value.get('traces')), 'state_hashes': value.get('state_hashes') is not None,
'queues': value.get('queues') is not None}
for name, field in (('apl_traces', 'full_APL_history_recorded'),
('state_hashes', 'full_tick_state_hashes_recorded'), ('queues', 'full_queue_hashes_recorded')):
if field in summary:
_require(type(summary[field]) is bool, label + ' observation flag must be Boolean')
if mode == 'reference' or legacy or summary.get(field) is True:
_require(observations[name], label + ' declared reference observation missing: ' + name)
return value, observations
except (KeyError, TypeError, AttributeError, IndexError, OverflowError) as error:
raise ValueError(label + ' has an invalid run-record schema') from error
def _common(left, right):
return {'same_neuron_count': len(left['state']['v']) == len(right['state']['v']),
'same_APL_width': left['terminal']['apl_pending'].shape[1] == right['terminal']['apl_pending'].shape[1],
'same_effective_weights': left['summary']['effective_weights_raw_sha256'] ==
right['summary']['effective_weights_raw_sha256']}
def _report(comparison, checks, observed, limitations):
# All predicates are JSON booleans, not array scalars or success claims.
checks = {key: bool(value) for key, value in checks.items()}
return dict(schema=SCHEMA, comparison=comparison, numerical_exact=all(checks.values()),
checks=checks, failed_checks=[key for key, value in checks.items() if not value],
observed_scope=observed, limitations=list(limitations), learning_pass_claimed=False,
complete_dynamical_equivalence_proven=False, external_authority_verified_here=False,
independent_CSR_queue_reconstruction_performed=False)
def compare_slice(candidate, monolithic):
"""Compare candidate's absolute [start,stop) with an origin-zero full reference.
The final candidate V/g/ref bytes must match the reference's tick stop-1
hashes even for light candidates. Missing light intermediate observations
are disclosed, never replaced by fabricated zero arrays or inferred hashes.
"""
candidate, observed = _record(candidate, 'candidate')
monolithic, full = _record(monolithic, 'monolithic')
_require(monolithic['global_tick_offset'] == 0, 'Monolithic slice reference must have origin zero')
_require(monolithic['summary'].get('diagnostic_mode') == 'reference' or
str(monolithic.get('terminal_origin', '')).startswith('AM_'),
'Monolithic slice source must be declared reference or loaded legacy AM')
_require(all(full.values()), 'Monolithic slice reference needs all four APL traces, state hashes and queues')
start, steps = candidate['global_tick_offset'], candidate['steps']
stop = start + steps
_require(stop <= monolithic['steps'], 'Candidate slice extends beyond monolithic reference')
checks = _common(candidate, monolithic)
state, whole = candidate['state'], monolithic['state']
lo, hi = np.searchsorted(whole['step'], (start, stop))
expected_ticks = (whole['step'][lo:hi].astype(np.int64) - start).astype(np.int32)
expected_nodes = whole['neuron_idx'][lo:hi]
checks['slice.event_steps'] = ec.bit_equal(state['step'], expected_ticks)
checks['slice.event_neurons'] = ec.bit_equal(state['neuron_idx'], expected_nodes)
counts = np.bincount(expected_nodes, minlength=len(whole['v'])).astype(np.int32)
checks['slice.segment_counts'] = ec.bit_equal(state['counts'], counts)
requests = monolithic['requests']
lo, hi = np.searchsorted(requests[:, 0], (start, stop))
expected_requests = requests[lo:hi].copy()
expected_requests[:, 0] -= start
checks['slice.requests_five_fields'] = ec.bit_equal(candidate['requests'], expected_requests)
for name in ('v', 'g', 'ref'):
checks['slice.final_' + name + '_raw_hash'] = _raw_sha(state[name]) == monolithic['state_hashes'][stop - 1][name + '_sha256']
raw_ring = ec.reconstructed_fresh_ring(whole, stop, len(whole['v']))
expected_ring = raw_ring[(np.arange(19) + stop % 19) % 19]
checks['slice.normalized_spike_ring'] = ec.bit_equal(ec.normalized_ring(candidate['terminal']), expected_ring)
traces = monolithic['traces']
checks['slice.APL_state'] = ec.bit_equal(candidate['terminal']['apl_state'], traces['state'][stop - 1])
pending = np.zeros((18, traces['generated'].shape[1]), dtype=np.float32)
kept = min(stop, 18)
pending[-kept:] = traces['generated'][stop - kept:stop]
checks['slice.APL_pending'] = ec.bit_equal(candidate['terminal']['apl_pending'], pending)
if observed['apl_traces']:
for name in ec.TRACE_FILES:
checks['slice.APL_trace.' + name] = ec.bit_equal(candidate['traces'][name], traces[name][start:stop])
if observed['state_hashes']:
checks['slice.per_tick_state_hashes'] = all(
all(left[key] == right[key] for key in ec.HASH_FIELDS)
for left, right in zip(candidate['state_hashes'], monolithic['state_hashes'][start:stop]))
if observed['queues']:
checks['slice.queue_order_and_acceptance'] = all(
all(left[key] == right[key] for key in ec.QUEUE_FIELDS)
for left, right in zip(candidate['queues'], monolithic['queues'][start:stop]))
limitations = ['External input/seal/case/parent authority is not established by these records.',
'Queue joint hashes contain local tick headers; their independent CSR reconstruction is not performed here.',
'Numerical equality of this observed slice is not a learning, live-coverage or complete dynamics proof.']
for key, label in (('apl_traces', 'APL histories'), ('state_hashes', 'intermediate V/g/ref state hashes'),
('queues', 'per-tick queue order/acceptance')):
if not observed[key]:
limitations.append('Candidate ' + label + ' are unobserved; no direct comparison is claimed.')
return _report('slice', checks, dict(start=start, stop=stop, segment_steps=steps,
events=True, segment_counts=True, requests_five_fields=True, final_state_against_reference_tick_hash=True,
normalized_spike_ring_from_reference_prefix_events=True, terminal_APL_state_and_pending=True,
candidate_APL_histories=observed['apl_traces'], candidate_per_tick_state_hashes=observed['state_hashes'],
candidate_per_tick_queue_fields=observed['queues'], queue_joint_hashes_compared=False), limitations)
def compare_checkpoint(candidate, prefix):
"""Compare equal absolute endpoints, not local phases or segment counts."""
candidate, _ = _record(candidate, 'candidate')
prefix, _ = _record(prefix, 'prefix')
checks = _common(candidate, prefix)
end = candidate['global_tick_offset'] + candidate['steps']
reference_end = prefix['global_tick_offset'] + prefix['steps']
checks['checkpoint.same_absolute_end'] = end == reference_end
for name in ('v', 'g', 'ref'):
checks['checkpoint.state.' + name] = ec.bit_equal(candidate['state'][name], prefix['state'][name])
checks['checkpoint.normalized_spike_ring'] = ec.bit_equal(
ec.normalized_ring(candidate['terminal']), ec.normalized_ring(prefix['terminal']))
for name in ('apl_state', 'apl_pending'):
checks['checkpoint.' + name] = ec.bit_equal(candidate['terminal'][name], prefix['terminal'][name])
return _report('checkpoint', checks, dict(candidate_absolute_end=end, reference_absolute_end=reference_end,
candidate_local_steps=candidate['steps'], reference_local_steps=prefix['steps'], terminal_v_g_ref=True,
normalized_spike_ring=True, terminal_APL_state_and_pending=True, counts_compared=False,
event_histories_compared=False, per_tick_state_histories_compared=False), [
'Segment counts and cumulative prefix counts are deliberately not compared in this endpoint scope.',
'Matching terminal records do not prove matching intermediate states or complete event histories.',
'External input/seal/parent authority and live-carry coverage remain separate gates.'])
def compare_parent_carry(candidate, parent):
"""Check recorded short-segment carry invariants, not unobserved restoration.
The parent ends where the candidate starts. Slots already overwritten by
this segment cannot be checked as initial carry from terminal data alone.
When present, the child's applied trace directly tests consumed APL rows.
"""
candidate, observed = _record(candidate, 'candidate')
parent, _ = _record(parent, 'parent')
_require(candidate['global_tick_offset'] > 0, 'Parent-carry candidate must be a continued segment')
checks = _common(candidate, parent)
start, steps = candidate['global_tick_offset'], candidate['steps']
parent_end = parent['global_tick_offset'] + parent['steps']
checks['parent.contiguous_absolute_clock'] = parent_end == start
restored_ring = ec.normalized_ring(parent['terminal'])
unconsumed_spikes, unconsumed_apl = steps < 19, steps < 18
if unconsumed_spikes:
checks['parent.unwritten_spike_slots'] = ec.bit_equal(
candidate['terminal']['spike_ring'][steps:], restored_ring[steps:])
pending = parent['terminal']['apl_pending']
if unconsumed_apl:
checks['parent.unconsumed_APL_prefix'] = ec.bit_equal(
candidate['terminal']['apl_pending'][:18 - steps], pending[steps:])
consume = min(18, steps)
if observed['apl_traces']:
checks['parent.applied_APL_prefix'] = ec.bit_equal(candidate['traces']['applied'][:consume], pending[:consume])
limitations = ['Initial V/g/ref/APL-state restoration is not directly observed by this terminal-only carry scope.',
'Overwritten spike slots and consumed pending slots require the separate full-reference slice/endpoint gates.',
'Parent manifest hashes and ancestor provenance must be independently bound by the coordinator.',
'No queue arrival/acceptance reconstruction, live-carry or learning success is claimed here.']
if not observed['apl_traces']:
limitations.append('Candidate applied APL history is unobserved; consumed parent APL values are not directly compared.')
carry_observed = unconsumed_spikes or unconsumed_apl or observed['apl_traces']
if not carry_observed:
limitations.append('No carry array remains directly observable in this long light segment; this scope checks clock/shape/weight identity only.')
return _report('parent_carry', checks, dict(parent_absolute_start=parent['global_tick_offset'],
parent_absolute_end=parent_end, candidate_absolute_start=start, candidate_absolute_end=start + steps,
candidate_local_steps=steps, unwritten_spike_slots_checked=unconsumed_spikes,
unwritten_spike_slot_count=max(19 - steps, 0), unconsumed_APL_prefix_checked=unconsumed_apl,
unconsumed_APL_row_count=max(18 - steps, 0), applied_parent_APL_prefix_checked=observed['apl_traces'],
applied_parent_APL_rows_compared=consume if observed['apl_traces'] else None,
any_carry_values_directly_compared=carry_observed,
initial_neural_state_restoration_directly_observed=False), limitations)
@@ -0,0 +1,421 @@
"""Finite, no-training execution of the separately registered 32-call gate.
All files are exclusive creations. A partial/failed row is preserved and is
not retried. Runtime outputs are sealed before external reports are written.
The preparation module owns source/input authority; no summary Boolean alone
is accepted as proof that a subprocess or a complete engineering gate passed.
"""
from __future__ import annotations
import argparse
from datetime import datetime, timezone
import importlib
import json
import os
from pathlib import Path
import shutil
import subprocess
import sys
import time
import traceback
import urllib.request
import zipfile
import numpy as np
import engineering_inputs as I
import engineering_plan as PLAN
import runtime_light as R
import runtime_continuation as C
GPU = 'GPU-79f3a85d-adce-7833-7003-6488140ca596'
ROOT = Path('/data/digital-drosophila')
MODEL = Path(__file__).resolve().parent
EXP = ROOT / 'experiments/phase4an-engineering-validation'
require = R.require
def canonical(value):
return json.dumps(value, sort_keys=True, separators=(',', ':'), allow_nan=False)
def same_json(left, right):
return canonical(left) == canonical(right)
def preparation():
return importlib.import_module('engineering_prepare')
def row_from(registration, row_id):
plan = PLAN.validate_plan(registration['plan'])
rows = [row for row in plan['rows'] if row['row_id'] == row_id]
require(len(rows) == 1, 'Only an exact registered row may execute')
return rows[0]
def locations(row_id):
require(type(row_id) is str and row_id and all(ch.isalnum() or ch in '-_' for ch in row_id),
'Unsafe row identity')
return dict(output=EXP / 'runs' / row_id,
runtime=EXP / 'receipts' / (row_id + '.runtime.json'),
execution=EXP / 'receipts' / (row_id + '.json'),
derivation=EXP / 'receipts' / (row_id + '.derivation.json'),
stdout=EXP / 'logs' / (row_id + '.stdout.log'),
stderr=EXP / 'logs' / (row_id + '.stderr.log'))
def disk_budget(registration):
limit = registration['budget']['output_budget_bytes']
reserve = registration['budget']['minimum_free_reserve_bytes']
require(type(limit) is type(reserve) is int and limit > 0 and reserve > 0, 'Invalid registered budget')
files = [p for p in EXP.rglob('*') if p.is_file()]
require(not any(p.is_symlink() for p in EXP.rglob('*')), 'Experiment contains a linked output')
used = sum(p.stat().st_size for p in files)
free = shutil.disk_usage(EXP).free
require(used <= limit and free >= reserve + max(0, limit - used), 'Engineering budget/reserve exceeded; preserve partials')
return dict(used_bytes=used, free_bytes=free, hard_cap_bytes=limit, reserve_bytes=reserve)
def preflight():
"""Never evict an existing service; configure only this child process."""
os.environ['CUDA_VISIBLE_DEVICES'] = GPU
os.environ['CUDA_HOME'] = '/usr/local/cuda'
os.environ['PATH'] = str(ROOT / 'models/male-cns-lif/.venv/bin') + ':/usr/local/cuda/bin:' + os.environ['PATH']
os.environ['MAX_JOBS'] = '2'
os.environ['TORCH_CUDA_ARCH_LIST'] = '8.9'
for port in (8188, 8189, 8190):
with urllib.request.urlopen(f'http://192.168.10.51:{port}/queue', timeout=5) as response:
queue = json.load(response)
require(queue['queue_running'] == [] and queue['queue_pending'] == [],
'ComfyUI busy: defer rather than evict')
def derive_case(template, row, parent_sha):
require(type(template) is dict and set(template) == {'schema', 'row_id', 'original_spec', 'start', 'stop',
'budget_root', 'output_budget_bytes'}, 'Unexpected registered case template fields')
require(template['schema'] == 'phase4an-engineering-case-template-v1' and template['row_id'] == row['row_id'] and
type(template['start']) is int and type(template['stop']) is int and
(template['start'], template['stop']) == (row['start'], row['stop']), 'Template differs from exact plan')
require((row['parent_row_id'] is None) == (parent_sha is None), 'Parent linkage differs from registered plan')
return I.derive_spec(template['original_spec'], start=row['start'], stop=row['stop'],
budget_root=template['budget_root'], output_budget_bytes=template['output_budget_bytes'], parent_sha=parent_sha)
def npz_storage(path):
with zipfile.ZipFile(path) as archive:
rows = archive.infolist()
require(rows and all(r.compress_type == zipfile.ZIP_DEFLATED and r.filename.endswith('.npy') for r in rows),
'Declared lossless compressed state format differs from actual archive')
return dict(file_bytes=Path(path).stat().st_size,
uncompressed_entry_bytes=sum(r.file_size for r in rows),
compressed_entry_bytes=sum(r.compress_size for r in rows),
entries=[dict(name=r.filename, bytes=r.file_size, compressed_bytes=r.compress_size,
compression=r.compress_type) for r in rows])
def runtime_environment():
"""Read settings after the runtime imported Torch; change none of them."""
torch = sys.modules.get('torch')
require(torch is not None and os.environ.get('CUDA_VISIBLE_DEVICES') == GPU and
torch.cuda.device_count() == 1 and '4090' in torch.cuda.get_device_name(0),
'Actual runtime is not the registered single RTX4090')
properties = torch.cuda.get_device_properties(0)
inventory = subprocess.run(['nvidia-smi', '-i', GPU,
'--query-gpu=uuid,name,memory.total,driver_version', '--format=csv,noheader'],
check=True, capture_output=True, text=True, timeout=20).stdout.strip()
require(inventory.startswith(GPU + ','), 'GPU hardware inventory does not match the registered UUID')
return dict(python_executable=sys.executable, python_version=sys.version, numpy_version=np.__version__,
torch_version=str(torch.__version__), torch_cuda_version=torch.version.cuda,
device_name=properties.name, total_memory_bytes=int(properties.total_memory),
gpu_uuid=GPU, nvidia_smi_inventory=inventory,
matmul_allow_tf32=bool(torch.backends.cuda.matmul.allow_tf32),
cudnn_allow_tf32=bool(torch.backends.cudnn.allow_tf32),
deterministic_algorithms=bool(torch.are_deterministic_algorithms_enabled()),
cudnn_deterministic=bool(torch.backends.cudnn.deterministic),
cudnn_benchmark=bool(torch.backends.cudnn.benchmark),
float32_matmul_precision=torch.get_float32_matmul_precision(),
CUBLAS_WORKSPACE_CONFIG=os.environ.get('CUBLAS_WORKSPACE_CONFIG'),
TORCH_CUDA_ARCH_LIST=os.environ.get('TORCH_CUDA_ARCH_LIST'),
settings_changed_by_environment_recorder=False,
environment_recorder_adds_no_timing=True,
build_timing_source='Use inherited summary.build_or_cache_seconds; build/cache is also included in runtime wall time')
def verify_environment(recorded, component):
require(type(recorded) is type(component) is dict and
recorded.get('gpu_uuid') == GPU and recorded.get('settings_changed_by_environment_recorder') is False,
'Missing or invalid actual environment receipt')
require(component.get('cuda_matmul_allow_tf32') is False and
component.get('float32_matmul_precision') == 'highest' and
component.get('precision_configuration_changed_by_component') is False,
'Frozen AM precision configuration changed; never correct it automatically')
pairs = {'matmul_allow_tf32': 'cuda_matmul_allow_tf32', 'cudnn_allow_tf32': 'cudnn_allow_tf32',
'deterministic_algorithms': 'deterministic_algorithms', 'cudnn_deterministic': 'cudnn_deterministic',
'cudnn_benchmark': 'cudnn_benchmark', 'float32_matmul_precision': 'float32_matmul_precision',
'CUBLAS_WORKSPACE_CONFIG': 'CUBLAS_WORKSPACE_CONFIG', 'device_name': 'device_name',
'torch_version': 'torch', 'torch_cuda_version': 'cuda', 'numpy_version': 'numpy'}
for left, right in pairs.items():
require(left in recorded and right in component and same_json(recorded[left], component[right]),
'Actual runtime/APL environment disagreement: ' + left)
require(type(recorded['device_name']) is str and '4090' in recorded['device_name'], 'Wrong physical device class')
def check_runtime_record(reg, reg_sha, row, record):
"""Recheck actual bytes, runtime completion and parent chain, not intent."""
paths = locations(row['row_id'])
require(record.get('complete') is True and record.get('registration_sha256') == reg_sha and
same_json(record.get('row'), row) and record.get('training') is False,
'Runtime record is not bound to this registered row')
case = Path(reg['cases'][row['row_id']]['case_dir'])
checkpoint = C.load_checkpoint(paths['output'], record['checkpoint_manifest_sha256'])
summary = checkpoint['summary']
verify_environment(record['environment'], summary['apl_component']['environment'])
require(checkpoint['steps'] == row['steps'] and checkpoint['global_tick_offset'] == row['start'] and
checkpoint['global_next_tick'] == row['stop'] and summary['nodes'] == 166700 and summary['edges'] == 25582938 and
summary['diagnostic_mode'] == row['mode'], 'Runtime clock/population/mode differs from plan')
require(record['case_sha256'] == R.digest(case / 'spec.json') and
record['requests_sha256'] == R.digest(case / 'requests.npz') and
record['derivation_sha256'] == R.digest(paths['derivation']), 'Derived input/receipt bytes changed')
template = R._read_json(case / 'template-spec.json')
parent_pin = None
if row['parent_row_id'] is not None:
parent = verify_execution(reg, reg_sha, row_from(reg, row['parent_row_id']))
parent_pin = parent['checkpoint_manifest_sha256']
expected, derivation = derive_case(template, row, parent_pin)
require(same_json(R._read_json(case / 'spec.json'), expected) and
same_json(R._read_json(paths['derivation']), derivation), 'Case was not generated by the registered exact rule')
require(record['parent_checkpoint_manifest_sha256'] == parent_pin and
summary.get('parent_checkpoint_manifest_sha256') == parent_pin, 'Execution parent SHA mismatch')
require(record['checkpoint_inventory_sha256'] == R.digest(paths['output'] / C.MANIFEST), 'Checkpoint manifest changed')
for name in ('state.npz', 'terminal-queues.npz'):
require(same_json(record['storage'][name], npz_storage(paths['output'] / name)), 'Storage measurement changed')
return record
def verify_execution(reg, reg_sha, row):
paths = locations(row['row_id'])
receipt = R._read_json(paths['execution'])
require(receipt.get('complete') is True and type(receipt.get('process_exit_code')) is int and receipt['process_exit_code'] == 0 and
receipt.get('registration_sha256') == reg_sha and receipt.get('row_id') == row['row_id'],
'Missing completed process receipt')
for field, key in (('runtime_record_sha256', 'runtime'), ('stdout_sha256', 'stdout'), ('stderr_sha256', 'stderr')):
require(receipt[field] == R.digest(paths[key]), 'Execution log/runtime receipt changed')
require(not (EXP / 'audit/stopped.json').exists(), 'A failed/deferred run requires explicit review, not automatic retry')
record = check_runtime_record(reg, reg_sha, row, R._read_json(paths['runtime']))
require(receipt['checkpoint_manifest_sha256'] == record['checkpoint_manifest_sha256'], 'Execution/checkpoint linkage mismatch')
return record
def worker(reg_sha, row_id):
require(sys.dont_write_bytecode, 'Use the scientific venv with -B')
reg = preparation().read_registration(reg_sha)
row = row_from(reg, row_id)
paths = locations(row_id)
require(not paths['output'].exists() and not paths['runtime'].exists() and not paths['derivation'].exists(),
'Never overwrite/retry a partial row')
budget_before = disk_budget(reg)
preflight()
parent_sha = None
parent_dir = None
if row['parent_row_id'] is not None:
parent_record = verify_execution(reg, reg_sha, row_from(reg, row['parent_row_id']))
parent_sha = parent_record['checkpoint_manifest_sha256']
parent_dir = locations(row['parent_row_id'])['output']
case = Path(reg['cases'][row_id]['case_dir'])
template = R._read_json(case / 'template-spec.json')
spec, derivation = derive_case(template, row, parent_sha)
require(not (case / 'spec.json').exists(), 'Derived spec already exists; preserve partial instead of rerun')
R._save(case / 'spec.json', spec)
R._save(paths['derivation'], derivation)
overlay = reg['overlays'][row['overlay']]
start_utc = datetime.now(timezone.utc).isoformat()
start = time.perf_counter()
if parent_dir is None:
function = R.run_reference if row['mode'] == 'reference' else R.run_light
function(case, paths['output'], EXP / 'extension-build', overlay['overlay_file'], overlay['overlay_manifest'])
pin = C.seal_checkpoint(paths['output'])['sha256']
else:
result = C.run_continued(case, paths['output'], EXP / 'extension-build',
overlay['overlay_file'], overlay['overlay_manifest'], parent_dir, mode=row['mode'])
pin = result['checkpoint_manifest_sha256']
elapsed = time.perf_counter() - start
budget_after = disk_budget(reg)
record = dict(schema='phase4an-engineering-runtime-record-v1', complete=True, training=False,
registration_sha256=reg_sha, row=row, case_sha256=R.digest(case / 'spec.json'),
requests_sha256=R.digest(case / 'requests.npz'), derivation_sha256=R.digest(paths['derivation']),
checkpoint_manifest_sha256=pin, checkpoint_inventory_sha256=R.digest(paths['output'] / C.MANIFEST),
parent_checkpoint_manifest_sha256=parent_sha, runtime_and_immediate_seal_wall_seconds=elapsed,
started_utc=start_utc, completed_utc=datetime.now(timezone.utc).isoformat(),
environment=runtime_environment(),
budget_before=budget_before, budget_after=budget_after,
storage={name: npz_storage(paths['output'] / name) for name in ('state.npz', 'terminal-queues.npz')},
no_result_selected_retry=True, learning_pass_claimed=False)
# Do not write this or any later report INSIDE the sealed checkpoint.
check_runtime_record(reg, reg_sha, row, record)
preparation().read_registration(reg_sha)
R._save(paths['runtime'], record)
print('COMPLETED ENGINEERING ROW', row_id, flush=True)
def run_registered(reg_sha):
require(sys.dont_write_bytecode, 'Use python -B')
reg = preparation().read_registration(reg_sha)
require(not (EXP / 'audit/stopped.json').exists() and not (EXP / 'audit/execution-complete.json').exists(),
'Stopped/completed gate must not be restarted')
started = time.perf_counter()
completed = []
try:
for row in reg['plan']['rows']:
paths = locations(row['row_id'])
if paths['execution'].exists():
verify_execution(reg, reg_sha, row)
completed.append(dict(row_id=row['row_id'], execution_sha256=R.digest(paths['execution']),
reverified_existing=True))
continue
require(not any(paths[k].exists() for k in ('output', 'runtime', 'stdout', 'stderr', 'derivation')),
'Partial row present; no automatic retry')
disk_budget(reg)
preflight()
began = time.perf_counter()
print('START', row['row_id'], flush=True)
with paths['stdout'].open('xb') as out, paths['stderr'].open('xb') as err:
result = subprocess.run([sys.executable, '-B', str(Path(__file__)), 'worker',
'--registration-sha256', reg_sha, '--row-id', row['row_id']], cwd=MODEL, stdout=out, stderr=err)
require(result.returncode == 0, 'Worker failed; preserve its stdout/stderr and partial output')
record = check_runtime_record(reg, reg_sha, row, R._read_json(paths['runtime']))
receipt = dict(schema='phase4an-engineering-execution-receipt-v1', complete=True,
process_exit_code=result.returncode, row_id=row['row_id'], registration_sha256=reg_sha,
runtime_record_sha256=R.digest(paths['runtime']), stdout_sha256=R.digest(paths['stdout']),
stderr_sha256=R.digest(paths['stderr']), process_wall_seconds=time.perf_counter() - began,
checkpoint_manifest_sha256=record['checkpoint_manifest_sha256'], scientific_pass_claimed=False)
R._save(paths['execution'], receipt)
verify_execution(reg, reg_sha, row)
completed.append(dict(row_id=row['row_id'], execution_sha256=R.digest(paths['execution']),
reverified_existing=False))
print('COMPLETE', row['row_id'], flush=True)
require(len(completed) == 32, 'Incomplete registered inventory')
preparation().read_registration(reg_sha)
R._save(EXP / 'audit/execution-complete.json', dict(complete=True, registration_sha256=reg_sha,
completed=completed, expected_calls=32, executed_ticks=11074, wall_seconds=time.perf_counter() - started,
numerical_gate_evaluated=False, training=False, learning_pass_claimed=False))
except Exception as error:
R._save(EXP / 'audit/stopped.json', dict(complete=False, type=type(error).__name__, message=str(error),
traceback=traceback.format_exc(), completed_before_error=completed, partial_outputs_preserved=True,
no_automatic_retry=True, scientific_failure_not_inferred_from_engineering_failure=True))
raise
def verified_inventory(reg, reg_sha):
"""A completed child and every planned descriptor must remain auditable."""
plan = PLAN.validate_plan(reg['plan'])
receipt = R._read_json(EXP / 'audit/execution-complete.json')
require(receipt.get('complete') is True and receipt.get('registration_sha256') == reg_sha and
type(receipt.get('expected_calls')) is int and receipt['expected_calls'] == plan['total_calls'] and
type(receipt.get('executed_ticks')) is int and receipt['executed_ticks'] == plan['total_ticks'] and
receipt.get('training') is False and receipt.get('learning_pass_claimed') is False,
'Execution inventory is not a complete no-training registered gate')
require(type(receipt.get('completed')) is list and len(receipt['completed']) == plan['total_calls'],
'Execution inventory truncated')
records = {}
for row, entry in zip(plan['rows'], receipt['completed']):
require(entry.get('row_id') == row['row_id'] and
entry.get('execution_sha256') == R.digest(locations(row['row_id'])['execution']),
'Execution completion inventory reordered or receipt changed')
records[row['row_id']] = verify_execution(reg, reg_sha, row)
return records
def evaluate(reg_sha):
reg = preparation().read_registration(reg_sha)
records = verified_inventory(reg, reg_sha)
report = importlib.import_module('engineering_analysis').analyze(reg, reg_sha, records)
require(type(report) is dict and report.get('training') is False and report.get('learning_pass_claimed') is False,
'Engineering analysis must not claim training or learning')
preparation().read_registration(reg_sha)
return reg, report
def analyze(reg_sha):
require(sys.dont_write_bytecode, 'Use python -B')
target = EXP / 'audit/engineering-analysis.json'
require(not target.exists() and not (EXP / 'manifest.json').exists(), 'Analysis/seal already exists; do not overwrite')
began = time.perf_counter()
_, report = evaluate(reg_sha)
R._save(target, report)
R._save(EXP / 'audit/analysis-execution.json', dict(registration_sha256=reg_sha,
analysis_sha256=R.digest(target), wall_seconds=time.perf_counter() - began, GPU_used=False,
training=False, learning_pass_claimed=False))
print(report['status'], flush=True)
def seal_experiment():
"""Seal only the new engineering experiment, not the unfinished lab model."""
controls = ('manifest.json', 'checksums.sha256')
require(not any((EXP / name).exists() for name in controls), 'Do not replace an existing or partial seal')
paths = sorted(EXP.rglob('*'))
require(not any(path.is_symlink() for path in paths), 'Cannot seal linked artifacts')
require(not any('__pycache__' in path.relative_to(EXP).parts for path in paths), 'Unexpected bytecode cache in new experiment')
rows = [dict(path=path.relative_to(EXP).as_posix(), bytes=path.stat().st_size, sha256=R.digest(path))
for path in paths if path.is_file()]
require(rows, 'Cannot seal an empty experiment')
# Only ROOT controls are excluded from their own hash. Nested checkpoint
# manifests and source snapshots are ordinary payload and remain included.
R._save(EXP / controls[0], dict(root=str(EXP), files=rows))
with (EXP / controls[1]).open('x', encoding='utf-8', newline='\n') as stream:
stream.writelines(row['sha256'] + ' ' + row['path'] + '\n' for row in rows)
stream.write(R.digest(EXP / controls[0]) + ' manifest.json\n')
return dict(manifest_sha256=R.digest(EXP / controls[0]), checksums_sha256=R.digest(EXP / controls[1]))
def finalize(reg_sha):
require(sys.dont_write_bytecode, 'Use python -B')
require(not (EXP / 'manifest.json').exists() and not (EXP / 'audit/final-status.json').exists(),
'Final/partial seal already exists; preserve it')
reg, actual = evaluate(reg_sha)
target = EXP / 'audit/engineering-analysis.json'
require(same_json(actual, R._read_json(target)), 'Recomputed analysis differs from saved evidence')
execution = R._read_json(EXP / 'audit/analysis-execution.json')
require(execution.get('registration_sha256') == reg_sha and execution.get('analysis_sha256') == R.digest(target),
'Saved analysis is not bound to its execution receipt')
authority = preparation().verify_authority(reg['anchors'])
old = preparation().verify_old_integrity(reg)
R._save(EXP / 'regression/AM-seals-after.json', authority)
R._save(EXP / 'regression/old-integrity-after.json', old)
preparation().read_registration(reg_sha)
final = dict(schema='phase4an-engineering-final-v1', status=actual['status'],
registration_sha256=reg_sha, analysis_sha256=R.digest(target),
analysis_recomputed_exactly=True, completed_calls=32, executed_ticks=11074,
old_integrity_pass=True, old_integrity_roots=len(old), AM_seals_preserved=True,
old_five_reference_evidence='Sealed AM five-trial evidence retained; not rerun by this engineering gate',
training=False, learning_pass_claimed=False, learned_candidate=False,
model_directory_sealed=False, used_engineering_source_snapshots_sealed=True,
budget=disk_budget(reg),
scope='Finite registered 100 ms input and fixed continuation cuts only; no universal determinism, learning or physiology claim')
R._save(EXP / 'audit/final-status.json', final)
pins = seal_experiment()
helper = preparation().load_module('phase4an_pinned_AI_publication',
ROOT / preparation().AI_HELPER_REL, preparation().AI_HELPER_SHA)
verified = helper.verify_seal(EXP)
require(verified['manifest_sha256'] == pins['manifest_sha256'] and
verified['checksums_sha256'] == pins['checksums_sha256'], 'Final experiment seal verification failed')
print(canonical(dict(status=final['status'], **pins)), flush=True)
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('action', choices=('run', 'worker', 'analyze', 'finalize'))
parser.add_argument('--registration-sha256', required=True)
parser.add_argument('--row-id')
args = parser.parse_args()
if args.action == 'worker':
require(args.row_id is not None, 'Worker requires a row ID')
worker(args.registration_sha256, args.row_id)
else:
require(args.row_id is None, 'Full inventory run has no selected row')
{'run': run_registered, 'analyze': analyze, 'finalize': finalize}[args.action](args.registration_sha256)
if __name__ == '__main__':
main()
@@ -0,0 +1,260 @@
"""Pure, UNREGISTERED evaluation inventory; no files, RNG draws or execution.
build_evaluation_plan(trial_payload, config) requires the complete existing
trial_plan payload, including its caller-chosen master and seed manifest. It
reuses that payload's 80 evaluation inputs; repeated probes are paired
observations, never extra independent training replicates. No defaults select
CS offset, reversal curve checkpoints, control snapshot boundaries, or baseline
execution policy. See required_choices(). A complete inventory is not approval,
an executed snapshot, an externally verified source pin, or a scientific gate.
"""
from __future__ import annotations
from collections import Counter
import hashlib
import json
import seed_registry
import trial_plan
SCHEMA = "phase4an-evaluation-plan-draft-v1"
CONFIG_SCHEMA = "phase4an-evaluation-config-draft-v1"
CONFIG_FIELDS = frozenset(("schema", "probe_steps", "cs_start_tick",
"cs_stop_tick_exclusive", "reversal_checkpoints", "baseline_execution",
"snapshot_binding_policy", "snapshot_bindings"))
BINDING_FIELDS = frozenset(("block_id", "assay", "arm", "checkpoint", "completed_epochs"))
def _require(condition, message):
if not condition:
raise ValueError(message)
def _canonical(value):
return json.dumps(value, sort_keys=True, separators=(",", ":"),
ensure_ascii=False, allow_nan=False)
def _sha(value):
return hashlib.sha256(_canonical(value).encode("utf-8")).hexdigest()
def required_choices():
"""Describe mandatory choices, without filling or approving any of them."""
return {
"probe_steps": "Explicit integer 1200: the draft's fixed 120 ms metric window.",
"cs_start_tick": "Explicit integer 0: stimulus onset defines probe latency.",
"cs_stop_tick_exclusive": "Explicit integer in [1,1200]; draft does not settle evaluation CS offset.",
"reversal_checkpoints": "Explicit sorted subset of [0,1,2,5,10,20,50], including 0 and 50.",
"baseline_execution": "Only execute_each_snapshot is implemented; declare it explicitly. No output deduplication.",
"snapshot_binding_policy": "Explicit completed_epochs; nominal control labels are not actual update counts.",
"snapshot_bindings": "Every block/assay/arm/checkpoint, in trial-plan order, with explicit completed_epochs. No control alignment is guessed.",
"authority": "Provide the complete existing trial payload; no master seed is chosen here. Source/input pins remain coordinator responsibilities.",
}
def _configuration(config):
_require(type(config) is dict, "Explicit evaluation config is required; see required_choices()")
_require(set(config) == CONFIG_FIELDS,
"Evaluation config fields: missing=" + str(sorted(CONFIG_FIELDS - set(config))) +
"; unknown=" + str(sorted(set(config) - CONFIG_FIELDS)))
_require(config["schema"] == CONFIG_SCHEMA, "Wrong evaluation config schema")
_require(type(config["probe_steps"]) is int and config["probe_steps"] == 1200,
"Explicit probe_steps must be integer 1200")
_require(type(config["cs_start_tick"]) is int and config["cs_start_tick"] == 0,
"Explicit CS onset must be integer tick 0")
stop = config["cs_stop_tick_exclusive"]
_require(type(stop) is int and 0 < stop <= 1200,
"Evaluation CS stop tick is required; no training-CS default is inferred")
checks = config["reversal_checkpoints"]
_require(type(checks) is list and len(checks) >= 2 and
all(type(v) is int and v in trial_plan.PROBE_CHECKPOINTS for v in checks) and
checks == sorted(set(checks)) and checks[0] == 0 and checks[-1] == 50,
"Explicit reversal checkpoints must include 0/50, sorted, unique, and within the draft curve")
_require(config["baseline_execution"] == "execute_each_snapshot",
"Declare execute_each_snapshot explicitly; baseline reuse is not implemented or inferred")
_require(config["snapshot_binding_policy"] == "explicit_completed_epochs",
"Explicit control snapshot epoch bindings are mandatory")
_require(type(config["snapshot_bindings"]) is list, "snapshot_bindings must be an explicit ordered list")
return json.loads(_canonical(config))
def _binding_inventory(plan, config):
expected, specs = [], []
for block in plan["blocks"]:
for suite in block["suites"]:
assay = suite["assay"]
points = config["reversal_checkpoints"] if assay == "reversal" else block["evaluation_checkpoints"]
for arm in suite["arms"]:
for point in points:
expected.append((block["block_id"], assay, arm["arm"], point))
specs.append((block, suite, arm))
bindings = config["snapshot_bindings"]
_require(len(bindings) == len(expected), "Missing/extra snapshot bindings: require every block, arm and checkpoint")
previous = {}
result = []
for binding, identity, spec in zip(bindings, expected, specs):
_require(type(binding) is dict and set(binding) == BINDING_FIELDS, "Invalid snapshot binding fields")
actual = tuple(binding[key] for key in ("block_id", "assay", "arm", "checkpoint"))
_require(all(type(v) is str for v in actual[:3]) and type(actual[3]) is int and actual == identity,
"Snapshot bindings must match exact canonical coverage/order/types")
block, suite, arm = spec
count = binding["completed_epochs"]
_require(type(count) is int and 0 <= count <= len(arm["epochs"]), "Invalid completed_epochs")
point = identity[-1]
_require((count == 0) == (point == 0), "Checkpoint zero is before the arm's first epoch")
_require(point != 50 or count == len(arm["epochs"]), "Terminal checkpoint must include the entire arm, including neutral/blank epochs")
key = identity[:3]
_require(key not in previous or count > previous[key], "Snapshot epochs must advance strictly")
previous[key] = count
us = sum(e["us"]["requested"] for e in arm["epochs"][:count])
updates = sum(e["update_enabled"] and e["D_command"] != 0 for e in arm["epochs"][:count])
if arm["arm"] in ("paired", "reversal", "no_reversal"):
_require(us == point and updates == point,
"Paired/reversal checkpoint must contain exactly its declared reinforcement opportunities")
result.append((binding, block, suite, arm, us, updates))
return result
def _streams(trial):
"""Reconstruct evaluation semantic keys, not just plausible seed strings."""
manifest = trial["seed_manifest"]
master = trial["plan"]["master"]
independent = {(r["split"], tuple(r["key"])): r for r in manifest["independent_streams"]}
non_eval = {r["seed"] for r in manifest["independent_streams"] if r["split"] != "evaluation"}
used, result = set(), {}
for block in trial["plan"]["blocks"]:
_require([v["cue"] for v in block["evaluation_streams"]] == ["A", "B"], "Both original evaluation cues are required")
for row in block["evaluation_streams"]:
cue = row["cue"]
key = (block["block_id"], "evaluation", "CS", cue, "00", "base-probe")
seed, derivation = seed_registry.seed_for(master, "evaluation", key)
registered = independent.get(("evaluation", key))
_require(registered == {"split": "evaluation", "key": list(key), "seed": seed,
"derivation_sha256": derivation}, "Missing exact evaluation registry key")
stream_id = "evaluation:" + _sha([trial_plan.SCHEMA, "stream-id", master, "evaluation", list(key)])
_require(type(row["seed"]) is int and row["seed"] == seed and row["seed_bits"] == 63 and
row["stream_id"] == stream_id and row["odor"] == trial_plan.ODORS[cue] and row["role"] == "CS",
"Evaluation seed/key/odor lineage mismatch")
_require(seed not in used and seed not in non_eval, "Evaluation seed overlaps another independent stream")
used.add(seed)
result[(block["block_id"], cue)] = dict(stream_id=stream_id, split="evaluation",
semantic_key=list(key), seed=seed, seed_bits=63, derivation_sha256=derivation,
input_id=block["block_id"] + "/evaluation/input-" + cue,
reuse="paired repeated probes within this training replicate; not additional independent seeds")
_require(len(result) == manifest["split_counts"]["evaluation"] == 80, "Exactly the existing 80 evaluation streams are required")
return result
def _snapshot_id(block_id, assay, arm, checkpoint):
return f"{block_id}/{assay}/{arm}/checkpoint-{checkpoint:02d}"
def _assemble(trial, config):
"""Internal assembly; public entry points first validate full trial authority."""
config = _configuration(config)
records = _binding_inventory(trial["plan"], config)
streams = _streams(trial)
snapshots, probes = [], []
def emit(block, snapshot):
common = dict(block_id=block["block_id"], mode=block["mode"], dan_type=block["dan_type"],
compartment_verbatim=block["compartment_verbatim"], training_replicate=block["replicate"],
original_reinforced_cue=block["initial_reinforced_cue"], original_neutral_cue=block["initial_neutral_cue"],
original_S_orientation="original reinforced minus original neutral, never flipped for reversal")
snapshot = dict(common, **snapshot)
snapshots.append(snapshot)
for cue in trial_plan.CUES:
probes.append(dict(common, probe_id=snapshot["snapshot_id"] + "/probe-" + cue,
snapshot_id=snapshot["snapshot_id"], baseline_snapshot_id=snapshot["baseline_snapshot_id"],
assay=snapshot["assay"], arm=snapshot["arm"], checkpoint=snapshot["checkpoint"],
elapsed_idle_ticks=snapshot["elapsed_idle_ticks"], cue=cue, odor=trial_plan.ODORS[cue],
steps=config["probe_steps"], dt_ms=0.1, cs_start_tick=config["cs_start_tick"],
cs_stop_tick_exclusive=config["cs_stop_tick_exclusive"],
input=streams[(block["block_id"], cue)], execution="independent fork; discard probe terminal state",
US_requested=False, D_command=0.0, weight_updates_enabled=False,
feedback_to_training_or_idle=False, adds_training_replicate=False))
for binding, block, suite, arm, us, updates in records:
block_id, assay, name = block["block_id"], suite["assay"], arm["arm"]
point, count = binding["checkpoint"], binding["completed_epochs"]
last = None if count == 0 else arm["epochs"][count - 1]["epoch_id"]
delta_source = (dict(arm["delta_start"]) if count == 0 else
{"kind": "epoch_delta_after", "source": last + "/delta-after"})
baseline = _snapshot_id(block_id, "normal" if assay == "reversal" else assay,
"paired" if assay == "reversal" else name, 0)
emit(block, dict(snapshot_id=_snapshot_id(block_id, assay, name, point), assay=assay, arm=name,
checkpoint=point, checkpoint_semantics="nominal reinforcement progress; control epoch boundary explicitly supplied",
completed_epochs=count, last_completed_epoch_id=last,
actual_US_requests_completed=us, enabled_nonzero_D_update_opportunities_completed=updates,
arm_reinforced_cue=arm["reinforced_cue"], delta_source=delta_source,
neural_state_source={"kind": "common_frozen_reset", "from_training_terminal_state": False},
baseline_snapshot_id=baseline, elapsed_idle_ticks=None,
probe_count=2, snapshot_mutated_by_planning=False))
for block in trial["plan"]["blocks"]:
block_id = block["block_id"]
normal = {a["arm"]: a for a in block["suites"][0]["arms"]}
for branch in block["retention"]["branches"]:
name = branch.split("/")[1]
terminal_epoch = normal[name]["epochs"][-1]["epoch_id"]
lineage = block_id + "/retention/" + name + "/idle-10s"
for seconds in block["retention"]["checkpoint_seconds"]:
ticks = seconds * 10000
emit(block, dict(snapshot_id=f"{block_id}/retention/{name}/idle-{seconds:02d}s",
assay="retention", arm=name, checkpoint=None, completed_epochs=len(normal[name]["epochs"]),
last_completed_epoch_id=terminal_epoch, delta_source={"kind": "epoch_delta_after", "source": terminal_epoch + "/delta-after"},
neural_state_source={"kind": "actual_terminal_checkpoint" if seconds == 0 else "continuous_idle_checkpoint",
"terminal_epoch_id": terminal_epoch, "idle_lineage_id": lineage,
"elapsed_ticks": ticks, "total_idle_ticks": 100000,
"one_second_is_prefix_of_same_ten_second_idle": True,
"carry": ["v", "g", "ref", "spike_ring", "apl_state", "apl_pending", "absolute_tick", "delta", "relevant_rng_state"],
"reset_or_analytic_time_advance": False},
baseline_snapshot_id=_snapshot_id(block_id, "normal", name, 0),
elapsed_idle_ticks=ticks, probe_count=2, snapshot_mutated_by_planning=False))
ids = {row["snapshot_id"] for row in snapshots}
_require(len(ids) == len(snapshots) and all(p["baseline_snapshot_id"] in ids for p in probes),
"Duplicate snapshot or missing original pre baseline")
by_assay = Counter(p["assay"] for p in probes)
steps = len(probes) * config["probe_steps"]
result = dict(schema=SCHEMA, status="DRAFT-NOT-REGISTERED", config=config,
trial_payload_sha256=trial["payload_sha256"], seed_manifest_sha256=trial["seed_manifest"]["payload_sha256"],
config_sha256=_sha(config), snapshots=snapshots, probes=probes,
budget=dict(snapshot_count=len(snapshots), evaluation_probes=len(probes),
probes_by_assay=dict(sorted(by_assay.items())), evaluation_steps=steps,
evaluation_biological_seconds=steps / 10000,
preexisting_training_control_reversal_epochs=trial["budget"]["training_control_reversal_epochs"],
preexisting_minimum_steps_excluding_evaluation=trial["budget"]["minimum_steps_excluding_evaluation"],
planned_scientific_steps_including_evaluation=trial["budget"]["minimum_steps_excluding_evaluation"] + steps,
planned_scientific_biological_seconds=(trial["budget"]["minimum_steps_excluding_evaluation"] + steps) / 10000,
independent_evaluation_inputs=80, independent_training_blocks=40,
evaluation_probes_are_training_replicates=False, baseline_output_deduplication=False,
scope="conditional full draft inventory, not an executed count or wall-time/storage estimate",
excludes=["implementation validation", "frozen regressions", "encoding", "I/O", "resource reserve"]),
evaluation_inventory_structurally_complete=True, scientific_gate_added=False,
protocol_approval_claimed=False, execution_authorized=False, training_executed=False,
inputs_generated=False, input_generator_implementation_verified=False, candidate_generated=False,
learning_pass_claimed=False, external_source_and_snapshot_authority_verified=False,
subsequent_assays_condition="not authorized here; full planned inventory remains subject to the draft's acquisition gate")
result["payload_sha256"] = _sha(result)
# No returned nested object aliases caller-owned trial/config structures.
return json.loads(_canonical(result))
def build_evaluation_plan(trial_payload, config):
"""Build a complete draft inventory, with mandatory explicit choices."""
_configuration(config) # Report missing choices before expensive reconstruction.
trial_plan.validate_trial_plan(trial_payload)
return _assemble(trial_payload, config)
def validate_evaluation_plan(payload, trial_payload, config):
"""Reconstruct every probe/binding; an updated self-hash cannot excuse drift."""
_require(type(payload) is dict and "payload_sha256" in payload, "Invalid evaluation payload")
_require(payload["payload_sha256"] == _sha({k: v for k, v in payload.items() if k != "payload_sha256"}),
"Evaluation payload SHA mismatch")
expected = build_evaluation_plan(trial_payload, config)
_require(_canonical(payload) == _canonical(expected), "Evaluation coverage, binding, type, seed, or order differs from exact reconstruction")
return True
@@ -0,0 +1,294 @@
"""Pure pre-outcome DRAFT snapshot policy; no I/O, inputs, execution or RNG.
The caller must supply the complete existing trial_plan payload. This module
does not select a master seed. Its fixed policy choices are newly proposed
planning choices, not inherited biological measurements or scientific gates.
build_evaluation_policy(trial_payload) returns a config directly consumable by
evaluation_plan.build_evaluation_plan, plus a separate ordered prefix audit.
Every snapshot boundary is AFTER the complete selected epoch and its scheduled
delta update, not immediately after the US tick. All returned timing and counts
describe planned exposures, never observed or accepted DAN spikes.
"""
from __future__ import annotations
from collections import Counter
import hashlib
import json
import evaluation_plan
import trial_plan
SCHEMA = "phase4an-evaluation-policy-draft-v1"
ALGORITHM = "fixed-ordered-CS-prefix-completed-epoch-v1"
POINTS = (0, 1, 2, 5, 10, 20, 50)
CS_FIELDS = frozenset(("cue", "odor", "occurrence", "stream_id", "source_stream_id",
"seed", "seed_bits", "seed_conversion", "role", "start_tick", "stop_tick_exclusive"))
IDENTITY_FIELDS = tuple(sorted(CS_FIELDS - {"stream_id"}))
SAME_POSITION_ARMS = frozenset(("d_zero", "frozen", "random_reward", "cs_only"))
def _require(condition, message):
if not condition:
raise ValueError(message)
def _canonical(value):
return json.dumps(value, sort_keys=True, separators=(",", ":"),
ensure_ascii=False, allow_nan=False)
def _sha(value):
return hashlib.sha256(_canonical(value).encode("utf-8")).hexdigest()
def _registry_index(trial):
manifest = trial["seed_manifest"]
independent = {(r["split"], tuple(r["key"])): r for r in manifest["independent_streams"]}
aliases = {(r["split"], tuple(r["key"])): r for r in manifest["aliases"]}
_require(len(independent) == len(manifest["independent_streams"]) and
len(aliases) == len(manifest["aliases"]) and not (independent.keys() & aliases.keys()),
"Duplicate or overlapping seed registry keys")
return independent, aliases
def _stream_id(master, key):
return "training:" + _sha([trial_plan.SCHEMA, "stream-id", master, "training", list(key)])
def _cs_identity(cs, *, block_id, assay, arm, base_arm, master, registry):
"""Verify actual alias source/seed; omit only the control's own alias ID."""
_require(type(cs) is dict and set(cs) == CS_FIELDS, "CS binding schema differs")
_require(cs["cue"] in trial_plan.CUES and cs["odor"] == trial_plan.ODORS[cs["cue"]] and
type(cs["occurrence"]) is int and 0 <= cs["occurrence"] < 50 and
type(cs["seed"]) is int and 0 <= cs["seed"] < 2**63 and
type(cs["seed_bits"]) is int and cs["seed_bits"] == 63 and cs["role"] == "CS" and
type(cs["start_tick"]) is int and cs["start_tick"] == 0 and
type(cs["stop_tick_exclusive"]) is int and cs["stop_tick_exclusive"] == 1000 and
cs["seed_conversion"] == "exact integer; no truncation or addition",
"CS cue/occurrence/full seed/role/timing differs")
key = (block_id, assay, "CS", cs["cue"], f'{cs["occurrence"]:02d}', arm)
source_key = key[:-1] + (base_arm,)
independent, aliases = registry
source = independent.get(("training", source_key))
_require(source is not None and source["seed"] == cs["seed"] and
cs["source_stream_id"] == _stream_id(master, source_key) and
cs["stream_id"] == _stream_id(master, key), "CS seed/source stream authority differs")
if arm == base_arm:
_require(("training", key) not in aliases and cs["stream_id"] == cs["source_stream_id"],
"Original CS must be its own independent source")
else:
alias = aliases.get(("training", key))
_require(alias is not None and alias["source_split"] == "training" and
alias["source_key"] == list(source_key) and alias["seed"] == cs["seed"] and
alias["independent_replicate"] is False,
"CS alias is not registered to this block/assay/cue/occurrence source")
return {key: cs[key] for key in IDENTITY_FIELDS}
def _prefixes(arm, *, block_id, assay, base_arm, master, registry):
"""Return verified ordered CS records indexed by actual completed epochs."""
identities = []
prefixes = [[]]
for epoch in arm["epochs"]:
if epoch["cs"] is not None:
identities.append(_cs_identity(epoch["cs"], block_id=block_id, assay=assay,
arm=arm["arm"], base_arm=base_arm, master=master, registry=registry))
prefixes.append(list(identities))
if arm["arm"] == "dan_only":
_require(not identities, "DAN-only must contain no delivered CS")
else:
_require(len(identities) == 100, "Every CS-present arm must retain all 100 CS epochs")
counters = Counter()
for row in identities:
_require(row["occurrence"] == counters[row["cue"]], "CS occurrence order is not canonical")
counters[row["cue"]] += 1
_require(dict(counters) == {"A": 50, "B": 50}, "Both original odors require 50 CS occurrences")
return prefixes
def _reinforcement_boundary(arm, point):
"""The whole epoch containing the kth requested US; terminal includes all."""
if point == 0:
return 0
epochs = arm["epochs"]
us_indices = [i for i, e in enumerate(epochs) if e["us"]["requested"]]
_require(len(us_indices) == 50, "Reference arm must have exactly 50 requested US epochs")
return len(epochs) if point == 50 else us_indices[point - 1] + 1
def _exposure(arm, count, cs_prefix):
epochs = arm["epochs"][:count]
return dict(completed_epochs=count, CS_epochs=len(cs_prefix),
CS_counts_by_cue={cue: sum(c["cue"] == cue for c in cs_prefix) for cue in trial_plan.CUES},
ordered_CS_source_identity_sha256=_sha(cs_prefix),
requested_US_epochs=sum(e["us"]["requested"] for e in epochs),
enabled_nonzero_D_update_opportunities=sum(e["update_enabled"] and e["D_command"] != 0 for e in epochs),
elapsed_training_ticks=sum(e["steps"] for e in epochs),
actual_accepted_US_or_DAN_spikes_known=False)
def _suite_policy(block, suite, master, registry):
"""Independent prefix/accounting construction, separate from evaluation_plan."""
assay, block_id = suite["assay"], block["block_id"]
_require(assay in ("normal", "delayed", "reversal"), "Unexpected assay")
expected_arms = trial_plan.REVERSAL_ARMS if assay == "reversal" else trial_plan.ARMS
_require([a["arm"] for a in suite["arms"]] == list(expected_arms), "Missing, reordered or duplicate control arms")
arms = {a["arm"]: a for a in suite["arms"]}
base_name = "reversal" if assay == "reversal" else "paired"
base = arms[base_name]
_require(len(base["epochs"]) == 100, "Base arm must contain all 100 epochs")
prefixes = {name: _prefixes(arm, block_id=block_id, assay=assay, base_arm=base_name,
master=master, registry=registry) for name, arm in arms.items()}
base_full = prefixes[base_name][-1]
for name, arm in arms.items():
_require(len(arm["epochs"]) == (150 if name == "unpaired" else 100), "Unexpected epoch coverage")
if name == "dan_only":
for i, epoch in enumerate(arm["epochs"]):
original = base["epochs"][i]
_require(epoch["cs"] is None and
epoch["yoked_cue_removed"] == original["cs"]["cue"] and
type(epoch["yoked_occurrence_removed"]) is int and
epoch["yoked_occurrence_removed"] == original["cs"]["occurrence"] and
_canonical(epoch["us"]) == _canonical(original["us"]) and
epoch["steps"] == original["steps"],
"DAN-only omitted CS/yoked US/epoch position differs from paired")
else:
_require(_canonical(prefixes[name][-1]) == _canonical(base_full),
"Actual ordered CS cue/occurrence/seed/source sequence differs")
if name != "unpaired":
_require(all(_canonical(a) == _canonical(b)
for a, b in zip(prefixes[name], prefixes[base_name])),
"CS sequence differs at a paired epoch position")
bindings, audits = [], []
for name in expected_arms:
arm = arms[name]
previous = -1
for point in POINTS:
base_count = _reinforcement_boundary(base, point)
expected_prefix = prefixes[base_name][base_count]
if assay == "reversal":
count = _reinforcement_boundary(arm, point)
alignment = "own reinforced-US ordinal; intermediate CS prefixes need not match across arms"
elif name == "unpaired":
if point == 0:
count = 0
elif point == 50:
count = len(arm["epochs"])
else:
matches = [i for i, prefix in enumerate(prefixes[name]) if len(prefix) == len(expected_prefix)]
_require(bool(matches), "Unpaired arm never reaches the required CS prefix")
count = matches[0]
alignment = "earliest ordered-CS-prefix boundary; terminal includes all 150 epochs"
else:
count = base_count
alignment = ("paired yoked epoch positions; zero delivered CS, not equal CS exposure"
if name == "dan_only" else "exact paired completed-epoch and ordered-CS prefix")
_require(count > previous, "Policy snapshot epochs must advance strictly")
previous = count
own_prefix = prefixes[name][count]
same_cs = _canonical(own_prefix) == _canonical(expected_prefix)
if assay != "reversal" and name != "dan_only":
_require(same_cs, "Ordered CS prefix mismatch at a control checkpoint")
if name == "unpaired" and point not in (0, 50):
_require(count > 0 and len(prefixes[name][count - 1]) < len(expected_prefix),
"Unpaired snapshot is not the earliest matching completed-epoch boundary")
actual = _exposure(arm, count, own_prefix)
reference = _exposure(base, base_count, expected_prefix)
binding = dict(block_id=block_id, assay=assay, arm=name,
checkpoint=point, completed_epochs=count)
bindings.append(binding)
audits.append(dict(binding=binding, alignment=alignment,
reference_arm=base_name, reference_completed_epochs=base_count,
boundary="after complete epoch and scheduled delta update; not at the US tick",
terminal_includes_all_epochs=point == 50,
actual=actual, reference=reference,
ordered_CS_prefix_exact=same_cs if name != "dan_only" else None,
CS_exposure_equal=same_cs if name != "dan_only" else False,
DAN_only_yoke_verified=name == "dan_only",
requested_US_count_equal_to_reference=actual["requested_US_epochs"] == reference["requested_US_epochs"],
requested_US_count_difference=actual["requested_US_epochs"] - reference["requested_US_epochs"],
elapsed_ticks_equal_to_reference=actual["elapsed_training_ticks"] == reference["elapsed_training_ticks"],
elapsed_ticks_difference=actual["elapsed_training_ticks"] - reference["elapsed_training_ticks"],
nonzero_D_update_opportunity_difference=actual["enabled_nonzero_D_update_opportunities"] - reference["enabled_nonzero_D_update_opportunities"],
all_US_timing_or_total_exposure_matched_claimed=False))
return bindings, audits
def _assemble(trial):
registry = _registry_index(trial)
bindings, audits = [], []
blocks = trial["plan"]["blocks"]
_require(len(blocks) == 40, "Exactly 40 original training blocks are required")
for block in blocks:
_require(block["evaluation_checkpoints"] == list(POINTS) and
[s["assay"] for s in block["suites"]] == ["normal", "delayed", "reversal"],
"Original assay/checkpoint coverage differs")
for suite in block["suites"]:
suite_bindings, suite_audits = _suite_policy(block, suite, trial["plan"]["master"], registry)
bindings.extend(suite_bindings)
audits.extend(suite_audits)
config = dict(schema=evaluation_plan.CONFIG_SCHEMA, probe_steps=1200,
cs_start_tick=0, cs_stop_tick_exclusive=1000, reversal_checkpoints=list(POINTS),
baseline_execution="execute_each_snapshot", snapshot_binding_policy="explicit_completed_epochs",
snapshot_bindings=bindings)
# Check the unchanged consumer's declared schema/coverage as well as our
# independent ordered-prefix construction. Do not generate its probes here.
evaluation_plan._configuration(config)
evaluation_plan._binding_inventory(trial["plan"], config)
_require(len(bindings) == 4480 and len({_canonical(v) for v in bindings}) == 4480,
"Incomplete or duplicate snapshot coverage")
counts = Counter(row["assay"] for row in bindings)
result = dict(schema=SCHEMA, status="DRAFT-NOT-REGISTERED", algorithm=ALGORITHM,
trial_payload_sha256=trial["payload_sha256"],
seed_manifest_sha256=trial["seed_manifest"]["payload_sha256"],
config_sha256=_sha(config), config=config,
audit=dict(bindings=audits,
coverage=dict(training_blocks=40, training_replicates_per_cell=5,
nonretention_snapshots=4480, snapshots_by_assay=dict(sorted(counts.items())),
original_full_CS_alias_sequences_verified=True,
new_random_streams=0, new_master_seed_selected=False),
exposure_limits=[
"CS-present normal/delayed controls match actual ordered CS prefixes, not just counts.",
"US counts, nonzero-D opportunities and elapsed time are separately reported, not assumed matched.",
"Unpaired boundaries include earlier inserted US; terminal includes all 150 epochs and trailing US.",
"DAN-only has zero delivered CS; yoked positions are not equal CS exposure.",
"Reversal and continuation use their own kth US; intermediate CS prefixes can differ.",
"All snapshots are after a full epoch plus its delta update, not the US tick.",
"Requested US is not accepted DAN activity, dopamine concentration or learning evidence."],
unchanged_retention_scope=dict(arms=["normal/paired", "normal/frozen"],
idle_checkpoints_seconds=[0, 1, 10],
same_continuous_ten_second_lineage=True, probes_fork_and_do_not_advance_idle=True,
retained_snapshot_execution_or_lineage_verified=False),
conditional_evaluation_inventory=dict(nonretention_probes=8960,
retention_snapshots=240, retention_probes=480, all_snapshots=4720,
all_probes=9440, probe_steps=1200, evaluation_steps=11328000,
includes_full_seven_point_reversal=True,
baseline_output_deduplication=False,
actual_execution_count=0)),
policy_structural_checks_complete=True, choices_fixed_before_outcomes=True,
choices_are_new_draft_proposals_not_inherited_gates=True,
protocol_approval_claimed=False, execution_authorized=False, training_executed=False,
inputs_generated=False, input_generator_implementation_verified=False,
source_and_snapshot_authority_verified=False, candidate_generated=False,
learning_pass_claimed=False, scientific_gate_added=False)
result["payload_sha256"] = _sha(result)
return json.loads(_canonical(result))
def build_evaluation_policy(trial_payload):
"""Return fixed DRAFT config/audit; caller provides, we never choose, master."""
trial_plan.validate_trial_plan(trial_payload)
return _assemble(trial_payload)
def validate_evaluation_policy(payload, trial_payload):
"""Exact reconstruction, including JSON types; a replaced self-hash is not authority."""
_require(type(payload) is dict and "payload_sha256" in payload, "Invalid policy payload")
_require(payload["payload_sha256"] == _sha({k: v for k, v in payload.items() if k != "payload_sha256"}),
"Actual evaluation policy SHA mismatch")
expected = build_evaluation_policy(trial_payload)
_require(_canonical(payload) == _canonical(expected), "Policy config/audit/coverage differs from exact reconstruction")
return True
@@ -0,0 +1,494 @@
"""Pure Phase 4AN proposal metrics; no file I/O, simulation or fitted decoder.
The caller must bind complete anatomical population membership (not a plastic
mask subset), finalized execution receipts, input identities and seed lineages
to its preregistration. These functions validate arrays/declared coverage; they
cannot establish provenance by trusting a caller's Boolean or SHA string.
All gates below are NEW DRAFT hypotheses, not a historical learning result.
"""
from __future__ import annotations
import hashlib
import json
import math
from collections.abc import Mapping
import numpy as np
SCHEMA = 'phase4an-metrics-draft-v1'
CUES = ('4-ethylphenol', '3-hexanone')
MODES = ('physiological', 'matched_energy')
DAN_TYPES = ('PAM15', 'PPL103')
CONTROLS = ('D=0', 'frozen', 'random reward', 'unpaired', 'CS-only', 'DAN-only')
ARMS = ('paired',) + CONTROLS
POPULATIONS = ('MBON', 'legacyDownstream', 'nonMBONDownstream', 'MBON01', 'MBON12')
SEPARATION_POPULATIONS = ('MBON', 'legacyDownstream', 'nonMBONDownstream')
POPULATION_SEMANTICS = {
'MBON': 'complete caller-bound anatomical MBON population',
'legacyDownstream': 'complete frozen H downstream membership, including MBONs; never mask-filtered',
'nonMBONDownstream': 'exact legacyDownstream minus MBON; NEW hypothesis readout, not the old H population',
'MBON01': 'complete caller-bound MBON01 type, not plastic-mask targets only',
'MBON12': 'complete caller-bound MBON12 type, not plastic-mask targets only',
}
STATE_KEYS = frozenset(('step', 'neuron_idx', 'counts', 'v', 'g', 'ref'))
def require(condition, message):
if not condition:
raise ValueError(message)
def _hash(value):
return hashlib.sha256(np.ascontiguousarray(value, dtype=value.dtype.newbyteorder('<')).tobytes()).hexdigest()
def _json_hash(value):
raw = json.dumps(value, sort_keys=True, separators=(',', ':'), allow_nan=False).encode('utf-8')
return hashlib.sha256(raw).hexdigest()
def _sha(value):
return type(value) is str and len(value) == 64 and all(c in '0123456789abcdef' for c in value)
def _token(value, name):
require(type(value) is str and bool(value) and value.strip() == value, 'Invalid ' + name)
return value
def _vector(value, dtype, name, size=None):
require(isinstance(value, np.ndarray) and value.dtype == np.dtype(dtype) and value.ndim == 1,
name + ' requires an exact ' + dtype + ' vector')
require(size is None or value.shape == (size,), name + ' shape mismatch')
return value
def _ids(value, name):
_vector(value, 'int32', name)
require(len(value) > 0 and np.all((value >= 0) & (value < 166700)) and np.all(value[1:] > value[:-1]),
name + ' must be nonempty sorted unique global neuron indices')
return value
def _positions(nodes, values, name):
position = np.searchsorted(nodes, values)
require(np.all(position < len(nodes)), name + ' contains absent neurons')
require(np.array_equal(nodes[position], values), name + ' contains absent neurons')
return position
def validate_populations(nodes, populations):
"""Validate full caller-bound type lists; never derive them from an edge mask."""
_ids(nodes, 'nodes')
require(isinstance(populations, Mapping) and set(populations) == set(POPULATIONS),
'Require exactly MBON/legacyDownstream/nonMBONDownstream/MBON01/MBON12 memberships')
for name in POPULATIONS:
_positions(nodes, _ids(populations[name], name), name)
for name in ('MBON01', 'MBON12'):
_positions(populations['MBON'], populations[name], name + ' outside MBON')
require(not len(np.intersect1d(populations['MBON01'], populations['MBON12'])), 'Type populations overlap')
_positions(populations['legacyDownstream'], populations['MBON'], 'MBON outside legacyDownstream')
require(np.array_equal(populations['nonMBONDownstream'],
np.setdiff1d(populations['legacyDownstream'], populations['MBON'])),
'nonMBONDownstream must equal the complete legacyDownstream minus MBON')
payload = {name: populations[name].tolist() for name in POPULATIONS}
return dict(indices=payload, membership_sha256=_json_hash(payload),
population_semantics=dict(POPULATION_SEMANTICS),
complete_anatomical_membership_authority='caller preregistration; never inferred from mask')
def validate_state(state, *, nodes, steps, dt_ms=.1):
"""Check every original state element and all sparse events without mutation."""
_ids(nodes, 'nodes')
require(type(steps) is int and 0 < steps <= np.iinfo(np.int32).max, 'Invalid steps')
require(type(dt_ms) in (int, float) and math.isfinite(dt_ms) and dt_ms > 0, 'Invalid dt_ms')
require(isinstance(state, Mapping) and set(state) == STATE_KEYS, 'State must contain exactly six runtime keys')
n = len(nodes)
for key, dtype in (('counts', 'int32'), ('ref', 'int32'), ('v', 'float64'), ('g', 'float64')):
_vector(state[key], dtype, key, n)
require(np.isfinite(state[key]).all(), 'Nonfinite state: ' + key)
require(np.all((state['counts'] >= 0) & (state['counts'] <= steps)), 'Invalid counts')
require(np.all((state['ref'] >= 0) & (state['ref'] <= 22)), 'Invalid refractory state')
ticks = _vector(state['step'], 'int32', 'step')
events = _vector(state['neuron_idx'], 'int32', 'neuron_idx', len(ticks))
require(np.all((ticks >= 0) & (ticks < steps)), 'Event tick outside trial')
require(len(ticks) < 2 or np.all((ticks[1:] > ticks[:-1]) |
((ticks[1:] == ticks[:-1]) & (events[1:] > events[:-1]))),
'Events must be sorted unique by tick/global neuron')
local = _positions(nodes, events, 'events')
require(np.array_equal(np.bincount(local, minlength=n), state['counts']), 'Sparse events/counts mismatch')
first = np.full(n, -1, dtype=np.int32)
if len(ticks):
first.fill(steps)
np.minimum.at(first, local, ticks)
first[first == steps] = -1
return dict(first_tick=first, local_events=local,
hashes={key: _hash(state[key]) for key in sorted(STATE_KEYS)})
def _first_ms(first, dt_ms):
present = first[first >= 0]
return float(int(present.min()) * dt_ms) if len(present) else None
def _latencies(first, dt_ms):
return [None if int(tick) < 0 else float(int(tick) * dt_ms) for tick in first]
def _population_change(pre, post, checked_pre, checked_post, nodes, local, dt_ms, vectors):
cp, cq = pre['counts'][local].astype(np.int64), post['counts'][local].astype(np.int64)
fp, fq = checked_pre['first_tick'][local], checked_post['first_tick'][local]
lp, lq = _first_ms(fp, dt_ms), _first_ms(fq, dt_ms)
both = (fp >= 0) & (fq >= 0)
ep = np.isin(checked_pre['local_events'], local)
eq = np.isin(checked_post['local_events'], local)
events_same = (np.array_equal(pre['step'][ep], post['step'][eq]) and
np.array_equal(pre['neuron_idx'][ep], post['neuron_idx'][eq]))
try:
with np.errstate(over='raise', invalid='raise'):
dv, dg = post['v'][local] - pre['v'][local], post['g'][local] - pre['g'][local]
vmax, gmax = float(np.max(np.abs(dv))), float(np.max(np.abs(dg)))
except FloatingPointError as error:
raise ValueError('Nonfinite derived final-state difference') from error
result = dict(neurons=len(local), pre_total_spikes=int(cp.sum()), post_total_spikes=int(cq.sum()),
spike_count_l1=int(np.abs(cq - cp).sum()), spike_count_changed_neurons=int(np.count_nonzero(cp != cq)),
events_changed=not events_same, pre_first_latency_ms=lp, post_first_latency_ms=lq,
first_latency_delta_ms=None if lp is None or lq is None else float(lq - lp),
defined_first_latency_changed=bool(lp is not None and lq is not None and lp != lq),
first_latency_both_defined_neurons=int(both.sum()),
first_latency_changed_among_both_defined_neurons=int(np.count_nonzero(both & (fp != fq))),
first_latency_became_defined_neurons=int(np.count_nonzero((fp < 0) & (fq >= 0))),
first_latency_became_undefined_neurons=int(np.count_nonzero((fp >= 0) & (fq < 0))),
final_delta_v_max_abs=vmax, final_delta_g_max_abs=gmax,
final_ref_changed_neurons=int(np.count_nonzero(pre['ref'][local] != post['ref'][local])),
final_state_units='SI volts; refractory integer ticks',
max_abs_scope='maximum across population neurons at terminal tick; NOT a temporal peak')
if vectors:
result.update(neuron_idx=nodes[local].tolist(), pre_counts=cp.tolist(), post_counts=cq.tolist(),
pre_first_latency_ms_per_neuron=_latencies(fp, dt_ms),
post_first_latency_ms_per_neuron=_latencies(fq, dt_ms))
for key in ('v', 'g', 'ref'):
result['pre_final_' + key] = pre[key][local].tolist()
result['post_final_' + key] = post[key][local].tolist()
return result
def functional_summary(pre, post, *, nodes, populations, steps, dt_ms=.1):
"""Full-state validation plus wholebrain and anatomical-group differences.
The population's earliest event defines the legacy-style first latency.
Per-neuron first events and missingness transitions are additional diagnostics.
A missing latency is None, never zero or evidence of a defined latency shift.
"""
membership = validate_populations(nodes, populations)
a = validate_state(pre, nodes=nodes, steps=steps, dt_ms=dt_ms)
b = validate_state(post, nodes=nodes, steps=steps, dt_ms=dt_ms)
groups = {name: _population_change(pre, post, a, b, nodes,
_positions(nodes, populations[name], name), dt_ms, True) for name in POPULATIONS}
return dict(schema=SCHEMA, full_six_key_state_validated=True, steps=steps, dt_ms=float(dt_ms),
nodes=len(nodes), nodes_raw_sha256=_hash(nodes), membership=membership,
pre_state_raw_sha256=a['hashes'], post_state_raw_sha256=b['hashes'],
wholebrain=_population_change(pre, post, a, b, nodes, np.arange(len(nodes)), dt_ms, False),
populations=groups, MBON_final_delta_g_max_abs=groups['MBON']['final_delta_g_max_abs'],
MBON_final_delta_g_units='SI volts', MBON_final_delta_g_scope='terminal across-neuron maximum, not time peak')
def arm_metrics(pre_by_cue, post_by_cue, *, conditioned_cue, nodes, populations, steps=1200, dt_ms=.1):
"""New fixed, unscaled type-count contrast; neither decoder nor feedback."""
require(conditioned_cue in CUES, 'Unknown conditioned odor label')
require(isinstance(pre_by_cue, Mapping) and isinstance(post_by_cue, Mapping) and
set(pre_by_cue) == set(post_by_cue) == set(CUES), 'Both original odor labels are required')
neutral = next(cue for cue in CUES if cue != conditioned_cue)
summaries = {cue: functional_summary(pre_by_cue[cue], post_by_cue[cue], nodes=nodes,
populations=populations, steps=steps, dt_ms=dt_ms) for cue in CUES}
b, means = {}, {}
for phase in ('pre', 'post'):
b[phase], means[phase] = {}, {}
for cue in CUES:
groups = summaries[cue]['populations']
means[phase][cue] = {name: float(np.mean(groups[name][phase + '_counts'])) for name in ('MBON01', 'MBON12')}
b[phase][cue] = means[phase][cue]['MBON12'] - means[phase][cue]['MBON01']
raw_s = (b['post'][conditioned_cue] - b['post'][neutral]) - (b['pre'][conditioned_cue] - b['pre'][neutral])
separation = {}
for name in SEPARATION_POPULATIONS:
values = {}
for phase in ('pre', 'post'):
first = np.asarray(summaries[CUES[0]]['populations'][name][phase + '_counts'], dtype=np.float64)
second = np.asarray(summaries[CUES[1]]['populations'][name][phase + '_counts'], dtype=np.float64)
values[phase] = float(np.linalg.norm(first - second))
separation[name] = dict(values, gain=values['post'] - values['pre'],
vector='all bound member spike counts', population_semantics=POPULATION_SEMANTICS[name])
return dict(schema=SCHEMA, analysis_status='DRAFT_NEW_HYPOTHESIS', conditioned_cue=conditioned_cue,
neutral_cue=neutral, steps=steps, dt_ms=float(dt_ms), rawB=b, rawS=float(raw_s),
type_mean_counts=means, B_definition='mean full MBON12 counts - mean full MBON01 counts',
B_is_decoder=False, S_orientation='conditioned minus neutral; post minus own pre',
separation=separation, cues=summaries,
historical_implementation_equivalence_claimed=False, learning_pass_claimed=False)
def _finite_tree(value):
if isinstance(value, Mapping):
for item in value.values():
_finite_tree(item)
elif isinstance(value, (list, tuple)):
for item in value:
_finite_tree(item)
elif isinstance(value, (float, np.floating)):
require(math.isfinite(float(value)), 'Nonfinite metric value')
def _bound_metric(metric, cue):
require(isinstance(metric, Mapping) and metric.get('schema') == SCHEMA and
metric.get('analysis_status') == 'DRAFT_NEW_HYPOTHESIS', 'Unexpected metric schema')
_finite_tree(metric)
require(metric['conditioned_cue'] == cue and metric['neutral_cue'] == next(c for c in CUES if c != cue),
'Cue role mismatch')
require(metric['steps'] == 1200 and metric['dt_ms'] == .1, 'Evaluation must use the fixed 120 ms probe')
# Derive gate quantities from recorded raw vectors, not a supplied gate Boolean.
for odor in CUES:
summary = metric['cues'][odor]
require(summary.get('full_six_key_state_validated') is True and summary['steps'] == 1200 and
summary['dt_ms'] == .1 and _sha(summary['nodes_raw_sha256']), 'Unvalidated cue state')
membership = summary['membership']['indices']
require(set(membership) == set(POPULATIONS) and summary['membership']['membership_sha256'] == _json_hash(membership),
'Population membership digest mismatch')
require(summary['membership'].get('population_semantics') == POPULATION_SEMANTICS,
'Population semantics must distinguish old H membership from the new non-MBON readout')
for key in ('pre_state_raw_sha256', 'post_state_raw_sha256'):
require(set(summary[key]) == STATE_KEYS and all(_sha(v) for v in summary[key].values()), 'Missing state digest')
for name in POPULATIONS:
group = summary['populations'][name]
ids, count = group['neuron_idx'], group['neurons']
require(type(count) is int and count > 0 and len(ids) == count and ids == membership[name] and
all(type(v) is int and 0 <= v < 166700 for v in ids) and ids == sorted(set(ids)),
'Empty or mismatched anatomical population')
for phase in ('pre', 'post'):
counts = group[phase + '_counts']
latency = group[phase + '_first_latency_ms_per_neuron']
require(len(counts) == len(latency) == count and
all(type(v) is int and 0 <= v <= 1200 for v in counts), 'Invalid count/latency vector')
for c, value in zip(counts, latency):
require((c == 0 and value is None) or (c > 0 and type(value) in (int, float) and
0 <= value < 120 and value == round(value / .1) * .1), 'Missing/nonphysical first latency')
present = [v for v in latency if v is not None]
require(group[phase + '_first_latency_ms'] == (min(present) if present else None),
'Population first latency differs from raw cells')
for key in ('v', 'g', 'ref'):
values = group[phase + '_final_' + key]
require(len(values) == count and all(type(v) in (int, float) and math.isfinite(v) for v in values),
'Invalid final-state vector')
if key == 'ref':
require(all(type(v) is int and 0 <= v <= 22 for v in values), 'Invalid final ref')
l1 = sum(abs(a - b) for a, b in zip(group['pre_counts'], group['post_counts']))
lp, lq = group['pre_first_latency_ms'], group['post_first_latency_ms']
changed = lp is not None and lq is not None and lp != lq
require(type(group['spike_count_l1']) is int and group['spike_count_l1'] == l1 and
group['defined_first_latency_changed'] is changed and
group['first_latency_delta_ms'] == (None if lp is None or lq is None else lq - lp),
'Functional predicate differs from raw counts/defined latency')
try:
with np.errstate(over='raise', invalid='raise'):
for key in ('v', 'g'):
difference = np.asarray(group['post_final_' + key]) - np.asarray(group['pre_final_' + key])
require(group['final_delta_' + key + '_max_abs'] == float(np.max(np.abs(difference))),
'Final-state maximum differs from raw terminal vectors')
except FloatingPointError as error:
raise ValueError('Nonfinite derived terminal metric') from error
require(not set(membership['MBON01']).intersection(membership['MBON12']), 'Type population overlap')
require(set(membership['MBON']) <= set(membership['legacyDownstream']),
'Legacy downstream must retain the complete MBON population')
require(membership['nonMBONDownstream'] == sorted(set(membership['legacyDownstream']) - set(membership['MBON'])),
'nonMBONDownstream differs from exact legacyDownstream minus MBON')
for name, parent in (('MBON01', 'MBON'), ('MBON12', 'MBON'),
('MBON', 'legacyDownstream'), ('nonMBONDownstream', 'legacyDownstream')):
require(set(membership[name]) <= set(membership[parent]), 'Population is not a subset of its bound parent')
lookup = {v: i for i, v in enumerate(membership[parent])}
for phase in ('pre', 'post'):
for field in ('counts', 'first_latency_ms_per_neuron', 'final_v', 'final_g', 'final_ref'):
key = phase + '_' + field
require(summary['populations'][name][key] == [
summary['populations'][parent][key][lookup[v]] for v in membership[name]],
'Overlapping population vectors disagree with the bound parent')
for phase in ('pre', 'post'):
for odor in CUES:
groups = metric['cues'][odor]['populations']
means = []
for name in ('MBON12', 'MBON01'):
counts = groups[name][phase + '_counts']
require(len(counts) > 0 and all(type(v) is int and 0 <= v <= 1200 for v in counts),
'Empty or invalid type count vector')
means.append(sum(counts) / len(counts))
require(metric['type_mean_counts'][phase][odor][name] == means[-1], 'Type mean differs from raw counts')
require(metric['rawB'][phase][odor] == means[0] - means[1], 'Raw B disagrees with full type vectors')
neutral = metric['neutral_cue']
calculated = (metric['rawB']['post'][cue] - metric['rawB']['post'][neutral]) - (
metric['rawB']['pre'][cue] - metric['rawB']['pre'][neutral])
require(type(metric['rawS']) in (int, float) and metric['rawS'] == calculated, 'Raw S disagrees with signed components')
for name in SEPARATION_POPULATIONS:
calculated = {}
for phase in ('pre', 'post'):
a = metric['cues'][CUES[0]]['populations'][name][phase + '_counts']
b = metric['cues'][CUES[1]]['populations'][name][phase + '_counts']
require(metric['cues'][CUES[0]]['membership'] == metric['cues'][CUES[1]]['membership'],
'A/B population mismatch')
calculated[phase] = float(np.linalg.norm(np.asarray(a, dtype=np.float64) - np.asarray(b, dtype=np.float64)))
require(metric['separation'][name]['pre'] == calculated['pre'] and
metric['separation'][name]['post'] == calculated['post'] and
metric['separation'][name]['gain'] == calculated['post'] - calculated['pre'],
'Separation gain differs from raw count vectors')
require(metric['separation'][name].get('population_semantics') == POPULATION_SEMANTICS[name],
'Separation population semantics mismatch')
return metric
def _execution(arm, name, assay, used_receipts):
require(isinstance(arm, Mapping) and set(arm) == {'metrics', 'execution'}, 'Arm needs metrics and actual execution evidence')
e = arm['execution']
required = {'complete', 'recording_verified', 'receipt_sha256', 'assay', 'epoch_duration_ms',
'cs_duration_ms', 'us_tick', 'completed_epochs', 'reinforcement_opportunities', 'terminal_pairings'}
require(isinstance(e, Mapping) and set(e) == required, 'Incomplete execution evidence')
require(e['complete'] is True and e['recording_verified'] is True and _sha(e['receipt_sha256']),
'All arms require an actual verified completed execution receipt')
require(e['receipt_sha256'] not in used_receipts, 'Execution receipt reused across arms/replicates')
used_receipts.add(e['receipt_sha256'])
require(e['assay'] == assay and e['epoch_duration_ms'] == (120. if assay == 'normal' else 180.) and
e['cs_duration_ms'] == 100. and type(e['us_tick']) is int and
e['us_tick'] == (900 if assay == 'normal' else 1500), 'Normal/delayed control timing mismatch')
require(type(e['completed_epochs']) is int and e['completed_epochs'] == (150 if name == 'unpaired' else 100),
'Incomplete arm epoch inventory')
require(type(e['reinforcement_opportunities']) is int and e['reinforcement_opportunities'] == (0 if name == 'CS-only' else 50),
'Wrong arm US opportunity inventory')
require(type(e['terminal_pairings']) is int and e['terminal_pairings'] == 50, 'Only fixed terminal count 50 may gate')
def evaluate_acquisition(replicates, *, assay, mode, dan_type, conditioned_cue, registered_training_streams):
"""Strict new normal/delayed 3-of-5 SAME-training-replicate conjunction.
Each row has replicate_id, replicate_unit='independent_training',
training_stream_ids (all independent source IDs, not aliases),
evaluation_stream_ids, assay/mode/dan_type/conditioned_cue, and seven arms.
Arm = {metrics: arm_metrics(...), execution: complete receipt declaration}.
registered_training_streams maps exactly five predeclared replicate IDs to
their source-ID lists. Aliased controls stay within a row; evaluation cannot
become new training replication. This pure function does NOT verify a file
receipt/seed derivation; the coordinator must independently do so.
"""
require(assay in ('normal', 'delayed') and mode in MODES and dan_type in DAN_TYPES and conditioned_cue in CUES,
'Unknown raw assay/mode/DAN/odor label')
require(isinstance(registered_training_streams, Mapping) and len(registered_training_streams) == 5,
'Require five predeclared independent training replicates')
registered, training_all = {}, set()
for identity, sources in registered_training_streams.items():
_token(identity, 'training replicate identity')
require(isinstance(sources, (tuple, list)) and len(sources) > 0, 'Missing actual training source identities')
values = tuple(_token(v, 'training source identity') for v in sources)
require(len(set(values)) == len(values) and not training_all.intersection(values),
'Training source reused across independent replicates')
training_all.update(values)
registered[identity] = values
require(isinstance(replicates, (tuple, list)) and len(replicates) == 5, 'Exactly five training rows required; do not count evaluation draws')
used_ids, used_eval, receipts, reports = set(), set(), set(), []
for row in replicates:
keys = {'replicate_id', 'replicate_unit', 'training_stream_ids', 'evaluation_stream_ids',
'assay', 'mode', 'dan_type', 'conditioned_cue', 'arms'}
require(isinstance(row, Mapping) and set(row) == keys, 'Invalid replicate schema')
identity = row['replicate_id']
require(identity in registered and identity not in used_ids and row['replicate_unit'] == 'independent_training',
'Unknown/duplicate training identity or evaluation pseudo-replication')
used_ids.add(identity)
require(isinstance(row['training_stream_ids'], (list, tuple)) and
tuple(row['training_stream_ids']) == registered[identity], 'Training lineage does not match preregistration')
require((row['assay'], row['mode'], row['dan_type'], row['conditioned_cue']) == (assay, mode, dan_type, conditioned_cue),
'Cannot pool modes, DAN branches, cue assignments or timing suites')
ev = row['evaluation_stream_ids']
require(isinstance(ev, (tuple, list)) and len(ev) > 0, 'Missing independent evaluation identity')
ev = tuple(_token(v, 'evaluation identity') for v in ev)
require(len(set(ev)) == len(ev) and not training_all.intersection(ev) and not used_eval.intersection(ev),
'Evaluation sources overlap training or another training replicate')
used_eval.update(ev)
require(isinstance(row['arms'], Mapping) and set(row['arms']) == set(ARMS), 'All six actual controls and paired arm are mandatory')
metrics = {}
for name in ARMS:
_execution(row['arms'][name], name, assay, receipts)
metrics[name] = _bound_metric(row['arms'][name]['metrics'], conditioned_cue)
baseline = metrics['paired']
# State baseline and populations must really be matched for each control.
for metric in metrics.values():
for cue in CUES:
a, b = baseline['cues'][cue], metric['cues'][cue]
require(a['membership'] == b['membership'] and a['nodes_raw_sha256'] == b['nodes_raw_sha256'] and
a['pre_state_raw_sha256'] == b['pre_state_raw_sha256'], 'Control population or pre-baseline mismatch')
s = baseline['rawS']
control_s = {name: metrics[name]['rawS'] for name in CONTROLS}
direction = s > 0 if dan_type == 'PAM15' else s < 0
beyond = {name: (s > value if dan_type == 'PAM15' else s < value) for name, value in control_s.items()}
neutral = baseline['neutral_cue']
c = baseline['cues'][conditioned_cue]['populations']['MBON']
n = baseline['cues'][neutral]['populations']['MBON']
functional = c['spike_count_l1'] > 0 or c['defined_first_latency_changed'] is True
predicates = dict(raw_signed_direction=bool(direction), beyond_each_actual_control=all(beyond.values()),
conditioned_final_g_exceeds_neutral=c['final_delta_g_max_abs'] > n['final_delta_g_max_abs'],
conditioned_count_or_defined_population_first_latency_change=bool(functional),
all_MBON_AB_euclidean_gain_positive=baseline['separation']['MBON']['gain'] > 0,
legacyDownstream_AB_euclidean_gain_positive=baseline['separation']['legacyDownstream']['gain'] > 0,
nonMBONDownstream_AB_euclidean_gain_positive=baseline['separation']['nonMBONDownstream']['gain'] > 0)
reports.append(dict(replicate_id=identity, rawS=s, raw_control_S=control_s,
population_separation_gains={name: baseline['separation'][name]['gain'] for name in SEPARATION_POPULATIONS},
beyond_controls= beyond, predicates=predicates, conjunction_pass=all(predicates.values()),
legacy_audit=dict(L_like_functional_with_explicit_missing_latency=bool(
predicates['conditioned_final_g_exceeds_neutral'] and functional),
H_like_MBON_positive_gain=predicates['all_MBON_AB_euclidean_gain_positive'],
H_like_downstream_population='legacyDownstream',
H_like_downstream_positive_gain=predicates['legacyDownstream_AB_euclidean_gain_positive'],
original_implementation_equivalence_claimed=False),
new_hypothesis_audit=dict(nonMBONDownstream_positive_gain=
predicates['nonMBONDownstream_AB_euclidean_gain_positive'],
inherited_from_old_H=False, either_downstream_may_rescue_other=False)))
reports.sort(key=lambda r: r['replicate_id'])
successful = sum(r['conjunction_pass'] for r in reports)
return dict(schema=SCHEMA, status='DRAFT-PROPOSAL-GATE-PASS' if successful >= 3 else 'DRAFT-PROPOSAL-GATE-FAIL',
proposal_gate_pass=successful >= 3, scientific_learning_pass_claimed=False, candidate_generated=False,
assay=assay, mode=mode, dan_type=dan_type, conditioned_cue=conditioned_cue,
required_training_replicates=5, successful_same_replicate_conjunctions=successful, required_majority=3,
actual_control_receipts=len(receipts) - 5, evaluation_draws_count_as_replicates=False,
execution_receipt_authority='caller must verify finalized receipts and seed registry; pure metrics do not read files',
historical_full_H_mean_control_rule_computed=False, replicates=reports)
def legacy_h_mean_control_audit(protocol_mbon_gains):
"""Old H scalar predicate only, kept separate from the new signed gate.
Input uses the ORIGINAL five protocol labels, each with five already-bound
training-replicate MBON Euclidean gains. The caller must explicitly define
any AN-to-old-protocol mapping; this function never picks a DAN/control or
pools AN cells. It is not the entire old H gate or a reproduced old result.
"""
names = ('appetitive-conditioning', 'aversive-conditioning', 'd-zero-control',
'random-reward-control', 'frozen-brain-control')
require(isinstance(protocol_mbon_gains, Mapping) and set(protocol_mbon_gains) == set(names),
'Legacy audit requires exactly the five original H protocol labels')
means = {}
for name in names:
values = protocol_mbon_gains[name]
require(isinstance(values, (list, tuple)) and len(values) == 5 and
all(type(v) in (int, float) and math.isfinite(v) for v in values),
'Legacy audit needs five finite, separately bound training-replicate gains')
means[name] = float(np.mean(values))
require(math.isfinite(means[name]), 'Nonfinite legacy mean')
floor, ceiling = min(means[names[0]], means[names[1]]), max(means[name] for name in names[2:])
return dict(schema=SCHEMA, audit='legacy_H_mean_control_predicate_only', protocol_means=means,
condition_floor=floor, control_ceiling=ceiling, predicate_pass=floor > ceiling,
source_reference='frozen phase4h.py:412-414', full_old_H_gate_computed=False,
scientific_learning_pass_claimed=False, AN_to_legacy_protocol_mapping_authority='caller; no automatic pooling')
def unavailable_assay(assay):
"""Do not turn a terminal delta or fresh reset into a retention/reversal PASS."""
require(assay in ('reversal', 'retention'), 'Unknown unavailable assay')
missing = {'reversal': ['bound acquisition/reversal/no-reversal lineage',
'original pre baseline and original cue orientation', 'terminal reversal count and both cue event outputs'],
'retention': ['validated continuous-state checkpoint/queue/RNG schema',
'bound same idle trajectory at 1 s and 10 s', 'matched frozen continuation and non-feedback probes']}
return dict(schema=SCHEMA, assay=assay, status='NOT-IMPLEMENTED', proposal_gate_pass=None,
scientific_learning_pass_claimed=False, candidate_generated=False, missing=missing[assay])
@@ -0,0 +1,478 @@
"""Unregistered fixed-weight continuation of the reporting-only AN executor.
run_continued(case, out, build, overlay, manifest, checkpoint_dir, mode='light')
also accepts mode='reference'. Inputs/events/counts use segment-local ticks;
summary and receipts expose global_tick_offset/global_next_tick. No training,
weight update, early stopping, or implicit extra tail is performed.
The caller must externally pin spec.parent_checkpoint_manifest_sha256. A fresh
parent's original runtime receipt did NOT bind state.npz: seal_checkpoint adds
that binding, after checking the available original receipts and current bytes.
It cannot retroactively prove that an unsealed state file was never modified.
Full graph/APL/coefficient/preregistered-input authority remains the coordinator's
responsibility, as in runtime_light. No executable/venv path is resolved here.
"""
from __future__ import annotations
import ast
import copy
import hashlib
import importlib.util
from pathlib import Path
import sys
import numpy as np
LIGHT_SHA = '522b8c31ea938890fe15364fccd25ccc5bded54101754909f255532e7ad01aae'
N = 166700
E = 25582938
MANIFEST = 'checkpoint-manifest.json'
RECEIPT = 'continuation-receipt.json'
RESTORE = '_continuation.restore(v, g, ref, counts, ring, apl_ring, apl, upload)'
CLOCK_OLD = 'tick >= 18'
CLOCK_NEW = 'tick + _continuation.global_offset >= 18'
def _light():
path = Path(__file__).resolve().parent / 'runtime_light.py'
raw = path.read_bytes()
if hashlib.sha256(raw).hexdigest() != LIGHT_SHA:
raise ValueError('Fresh runtime source pin changed')
name = 'phase4an_continuation_pinned_light'
existing = sys.modules.get(name)
if existing is not None:
if getattr(existing, '__continuation_pin__', None) != LIGHT_SHA or Path(existing.__file__).resolve() != path:
raise ValueError('Unverified runtime module binding')
return existing
spec = importlib.util.spec_from_file_location(name, path)
module = importlib.util.module_from_spec(spec)
sys.modules[name] = module
try:
exec(compile(raw, str(path), 'exec'), module.__dict__)
except BaseException:
sys.modules.pop(name, None)
raise
module.__continuation_pin__ = LIGHT_SHA
return module
R = _light() # Pure NumPy/stdlib; this does not import Torch or run a brain.
def rotate_spike_ring(raw_ring, phase_tick):
"""Parent's local next-write slot becomes the child's local slot zero."""
R.require(isinstance(raw_ring, np.ndarray) and raw_ring.dtype == np.uint8 and
raw_ring.ndim == 2 and raw_ring.shape[0] == 19, 'Invalid spike ring dtype/shape')
R.require(type(phase_tick) is int and phase_tick >= 0 and np.all(raw_ring <= 1), 'Invalid ring phase/content')
return raw_ring[(np.arange(19) + phase_tick % 19) % 19].copy()
def _tree_hashes(directory):
"""Exact regular-file inventory, no symlink/reparse escapes, no self-hash."""
directory = Path(directory).resolve(strict=True)
R.require(directory.is_dir(), 'Checkpoint must be a directory')
result = {}
for path in sorted(directory.rglob('*')):
R.require(not path.is_symlink() and path.resolve().is_relative_to(directory), 'Checkpoint contains a linked/escaping path')
if path.is_dir():
continue
R.require(path.is_file(), 'Checkpoint contains a nonregular entry')
relative = path.relative_to(directory).as_posix()
if relative == MANIFEST:
continue
R.require('error' not in path.name.lower() and 'failure' not in path.name.lower(), 'Parent has failure/error evidence')
result[relative] = R.digest(path)
return result
def _npz(path, keys):
with np.load(path, allow_pickle=False) as archive:
R.require(len(archive.files) == len(keys) and set(archive.files) == set(keys), 'Unexpected checkpoint array keys: ' + str(path))
return {name: archive[name].copy() for name in keys}
def validate_state(state, terminal, steps, *, fresh):
"""Validate local events/counts and the slots written by this segment."""
R.require(type(steps) is int and 0 < steps <= np.iinfo(np.int32).max, 'Invalid segment length')
for name in ('v', 'g'):
value = state[name]
R.require(value.dtype == np.float64 and value.shape == (N,) and np.isfinite(value).all(), 'Invalid parent ' + name)
for name in ('ref', 'counts'):
value = state[name]
R.require(value.dtype == np.int32 and value.shape == (N,) and np.all(value >= 0), 'Invalid parent ' + name)
R.require(np.all(state['ref'] <= 22), 'Parent refractory counter exceeds frozen 22-step contract')
tick, neuron = state['step'], state['neuron_idx']
R.require(tick.dtype == neuron.dtype == np.int32 and tick.ndim == neuron.ndim == 1 and tick.shape == neuron.shape,
'Invalid sparse event dtype/shape')
R.require(np.all((tick >= 0) & (tick < steps)) and np.all((neuron >= 0) & (neuron < N)), 'Parent event outside segment')
key = tick.astype(np.int64) * N + neuron
R.require(len(key) < 2 or np.all(key[1:] > key[:-1]), 'Parent events must be sorted and unique')
R.require(np.array_equal(np.bincount(neuron, minlength=N), state['counts']), 'Parent counts are not segment-local event counts')
ring = terminal['spike_ring']
R.require(ring.dtype == np.uint8 and ring.shape == (19, N) and np.all(ring <= 1), 'Invalid parent spike ring')
for name, shape in (('apl_state', (2,)), ('apl_pending', (18, 4064))):
value = terminal[name]
R.require(value.dtype == np.float32 and value.shape == shape and np.isfinite(value).all(), 'Invalid terminal ' + name)
phase = terminal['next_tick']
R.require(phase.dtype == np.int64 and phase.shape == () and phase.item() == steps, 'Terminal local phase does not match segment length')
for local_tick in range(max(0, steps - 19), steps):
start, stop = np.searchsorted(tick, (local_tick, local_tick + 1))
expected = np.zeros(N, dtype=np.uint8)
expected[neuron[start:stop]] = 1
R.require(np.array_equal(ring[local_tick % 19], expected), 'Spike ring does not match emitted segment events')
if fresh and steps < 19:
R.require(not np.any(ring[steps:]), 'Fresh parent has nonzero uninitialized spike slots')
return state, terminal
def _verify_receipt_hashes(directory, receipt):
for key, name in (('summary_sha256', 'summary.json'), ('finite_audit_sha256', 'finite-audit.json'),
('terminal_queues_sha256', 'terminal-queues.npz'), ('overlay_audit_sha256', 'overlay-audit.json')):
R.require(receipt.get(key) == R.digest(directory / name), 'Parent receipt hash mismatch: ' + name)
inputs = receipt.get('input_sha256')
R.require(isinstance(inputs, dict) and inputs, 'Parent receipt lacks immutable input bindings')
for name, sha in inputs.items():
R.require(Path(name).is_absolute() and R._sha(sha) and R.digest(name) == sha, 'Parent input/source bytes changed: ' + name)
def _validate_record(directory, seen):
summary = R._read_json(directory / 'summary.json')
finite = R._read_json(directory / 'finite-audit.json')
provenance = R._read_json(directory / 'runtime-provenance.json')
steps = summary.get('steps')
R.require(summary.get('complete') is True and summary.get('pass_all') is True and summary.get('all_finite') is True and
summary.get('completed_ticks') == steps and summary.get('nodes') == N and summary.get('edges') == E and
summary.get('precision') == 'float64' and summary.get('apl_precision') == 'float32' and
summary.get('within_trial_weight_updates') is False, 'Parent is not a complete fixed-weight fullgraph trial')
R.require(finite.get('all_finite') is True and finite.get('first_failure_tick') == -1 and
finite.get('finite_observed_ticks') == steps, 'Parent finite coverage failed')
R.require(provenance.get('complete') is True and provenance.get('weights_updated_during_run') is False and
provenance.get('old_globals_patched') is False, 'Invalid parent runtime receipt')
_verify_receipt_hashes(directory, provenance)
fresh = provenance.get('schema') == 'phase4an-reporting-adapter-v1'
if fresh:
R.require(provenance.get('source_sha256') == LIGHT_SHA and R.digest(provenance['source_file']) == LIGHT_SHA and
summary.get('checkpoint_continuation_used') is False, 'Fresh parent source/continuation scope mismatch')
offset = 0
else:
R.require(provenance.get('schema') == 'phase4an-continuation-runtime-v1' and
provenance.get('source_sha256') == R.digest(__file__) and
R.digest(provenance['source_file']) == R.digest(__file__) and
summary.get('checkpoint_continuation_used') is True, 'Unrecognized continued parent source')
receipt = R._read_json(directory / RECEIPT)
R.require(receipt.get('schema') == 'phase4an-continuation-receipt-v1' and receipt.get('complete') is True,
'Invalid continuation receipt')
R.require(receipt.get('weights_updated_during_run') is False and receipt.get('segment_steps') == steps and
receipt.get('counts_scope') == 'segment_only_not_lifetime' and
provenance.get('state_sha256') == R.digest(directory / 'state.npz'), 'Continued state/scope receipt mismatch')
expected_payload = _tree_hashes(directory)
expected_payload.pop(RECEIPT, None)
R.require(receipt.get('payload_sha256') == expected_payload, 'Continuation receipt does not bind the exact payload')
parent_pin = receipt.get('parent_checkpoint_manifest_sha256')
case_spec = _parent_spec({'summary': summary, 'provenance': provenance})
R.require(R._sha(parent_pin) and provenance.get('parent_checkpoint_manifest_sha256') == parent_pin and
summary.get('parent_checkpoint_manifest_sha256') == parent_pin and
case_spec.get('parent_checkpoint_manifest_sha256') == parent_pin,
'Parent checkpoint pin disagrees across case/summary/provenance/continuation receipts')
R.require(Path(receipt['parent_checkpoint_dir']).is_absolute(), 'Parent checkpoint directory is not absolute')
parent = load_checkpoint(receipt['parent_checkpoint_dir'], parent_pin, _seen=seen)
offset = parent['global_next_tick']
R.require(receipt.get('global_tick_offset') == offset and receipt.get('global_next_tick') == offset + steps and
summary.get('global_tick_offset') == offset and summary.get('global_next_tick') == offset + steps and
provenance.get('global_tick_offset') == offset and provenance.get('global_next_tick') == offset + steps,
'Continuation global clock chain mismatch')
R.require(summary.get('effective_weights_raw_sha256') == parent['summary'].get('effective_weights_raw_sha256'),
'Retention effective weights changed between parent and child')
state = _npz(directory / 'state.npz', ('step', 'neuron_idx', 'counts', 'v', 'g', 'ref'))
terminal = _npz(directory / 'terminal-queues.npz', ('spike_ring', 'apl_state', 'apl_pending', 'next_tick'))
validate_state(state, terminal, steps, fresh=fresh)
initial_padding = max(18 - (offset + steps), 0)
R.require(np.all(terminal['apl_pending'][:initial_padding].view(np.uint32) == 0),
'Unconsumed original APL padding is not positive zero')
if not fresh:
if steps < 19:
restored = rotate_spike_ring(parent['terminal']['spike_ring'], int(parent['terminal']['next_tick']))
R.require(np.array_equal(terminal['spike_ring'][steps:], restored[steps:]), 'Unwritten carried spike slots changed')
if steps < 18:
carried = parent['terminal']['apl_pending'][steps:]
R.require(np.array_equal(terminal['apl_pending'][:18 - steps].view(np.uint32), carried.view(np.uint32)),
'Unconsumed carried APL slots changed')
overlay = R._read_json(directory / 'overlay-audit.json')
R.require(overlay.get('complete') is True and overlay.get('weights_updated_during_run') is False and
overlay.get('effective_weights_raw_sha256') == summary.get('effective_weights_raw_sha256') and
overlay.get('mask_sha256') == R.MASK_SHA and overlay.get('am_helper_sha256') == R.AM_OVERLAY_SHA,
'Parent effective overlay receipt mismatch')
overlay_manifest = R._read_json(directory / 'overlay-manifest.json')
R.require(R.digest(directory / 'overlay-input.npz') == overlay_manifest.get('overlay_sha256') and
R.digest(directory / 'overlay-manifest.json') in provenance['input_sha256'].values() and
overlay_manifest.get('mask_sha256') == R.MASK_SHA and
overlay_manifest.get('base_candidate_sha256') == overlay.get('base_candidate_sha256'),
'Parent copied overlay inputs do not match their original receipt')
timing = R._read_json(directory / 'apl-timing-audit.json')
with (directory / 'apl-pending-tail.float32.npy').open('rb') as stream:
pending = np.load(stream, allow_pickle=False)
R.require(pending.dtype == np.float32 and pending.shape == (18, 4064) and
pending.tobytes() == terminal['apl_pending'].tobytes(), 'Parent terminal/pending output mismatch')
R.require(timing.get('pending_raw_sha256') == R._raw(pending) and timing.get('apl_delay_steps') == 18 and
timing.get('brain_tail_advanced') is False and
timing.get('final_apl_state_u32_bits') == terminal['apl_state'].view(np.uint32).tolist(), 'Parent APL terminal audit mismatch')
return {'directory': directory, 'summary': summary, 'provenance': provenance, 'overlay': overlay,
'state': state, 'terminal': terminal, 'steps': steps, 'global_tick_offset': offset,
'global_next_tick': offset + steps, 'fresh': fresh,
'ancestry_directories': [directory] if fresh else [directory, *parent['ancestry_directories']]}
def seal_checkpoint(checkpoint_dir):
"""Exclusive file-hash seal, not a brain run or proof of pre-seal integrity.
The returned manifest SHA must be externally pinned into the next case spec.
No existing receipt is overwritten. Fresh state.npz is newly bound here.
"""
directory = Path(checkpoint_dir).resolve(strict=True)
R.require(not (directory / MANIFEST).exists(), 'Checkpoint is already sealed')
before = _tree_hashes(directory)
record = _validate_record(directory, {directory})
R.require(before == _tree_hashes(directory), 'Checkpoint changed while sealing')
value = {'schema': 'phase4an-checkpoint-v1', 'complete': True, 'files_sha256': before,
'segment_steps': record['steps'], 'global_tick_offset': record['global_tick_offset'],
'global_next_tick': record['global_next_tick'], 'counts_scope': 'segment_only',
'state_binding_origin': 'checkpoint seal; fresh runtime receipt did not hash state.npz',
'pre_seal_state_integrity_independently_proven': False}
R._save(directory / MANIFEST, value)
return {'path': str(directory / MANIFEST), 'sha256': R.digest(directory / MANIFEST), **value}
def load_checkpoint(checkpoint_dir, expected_sha, *, _seen=None):
directory = Path(checkpoint_dir).resolve(strict=True)
R.require(R._sha(expected_sha) and R.digest(directory / MANIFEST) == expected_sha, 'Parent checkpoint manifest differs from external pin')
seen = set() if _seen is None else set(_seen)
R.require(directory not in seen, 'Checkpoint ancestry cycle')
seen.add(directory)
manifest = R._read_json(directory / MANIFEST)
R.require(manifest.get('schema') == 'phase4an-checkpoint-v1' and manifest.get('complete') is True and
manifest.get('counts_scope') == 'segment_only' and manifest.get('files_sha256') == _tree_hashes(directory),
'Checkpoint file inventory/hash mismatch')
record = _validate_record(directory, seen)
R.require(manifest.get('segment_steps') == record['steps'] and manifest.get('global_tick_offset') == record['global_tick_offset'] and
manifest.get('global_next_tick') == record['global_next_tick'], 'Checkpoint clock metadata mismatch')
R.require(manifest['files_sha256'] == _tree_hashes(directory) and R.digest(directory / MANIFEST) == expected_sha,
'Checkpoint changed while loading')
record.update(manifest=manifest, manifest_sha256=expected_sha)
return record
def derive_continued(source_bytes, mode):
"""One explicit arrival-admission edit plus one pre-loop restore hook."""
tree, inherited = R.derive_ast(source_bytes, mode)
original = copy.deepcopy(tree)
function = tree.body[-1]
body = next(node for node in function.body if isinstance(node, ast.Try)).body
candidates = [i for i, node in enumerate(body) if isinstance(node, ast.Assign) and
len(node.targets) == 1 and isinstance(node.targets[0], ast.Name) and node.targets[0].id == 'apl_ring']
R.require(len(candidates) == 1, 'Expected one APL ring initialization')
body.insert(candidates[0] + 1, ast.parse(RESTORE).body[0])
clocks = [node for node in ast.walk(function) if isinstance(node, ast.If) and ast.unparse(node.test) == CLOCK_OLD]
R.require(len(clocks) == 1, 'Expected exact original local arrival-admission condition')
clocks[0].test = ast.parse(CLOCK_NEW, mode='eval').body
tree = ast.fix_missing_locations(tree)
# Independently reverse exactly the two edits, then require the ENTIRE AST.
rollback = copy.deepcopy(tree)
reverse_body = next(node for node in rollback.body[-1].body if isinstance(node, ast.Try)).body
restores = [i for i, node in enumerate(reverse_body) if ast.unparse(node) == RESTORE]
reverse_clocks = [node for node in ast.walk(rollback) if isinstance(node, ast.If) and ast.unparse(node.test) == CLOCK_NEW]
R.require(len(restores) == len(reverse_clocks) == 1, 'Ambiguous continuation edit rollback')
del reverse_body[restores[0]]
reverse_clocks[0].test = ast.parse(CLOCK_OLD, mode='eval').body
R.require(R._dump(rollback) == R._dump(original), 'Additional unlisted continuation AST edit')
audit = {'base': inherited, 'restore_hook': RESTORE, 'clock_admission_old': CLOCK_OLD,
'clock_admission_new': CLOCK_NEW, 'full_AST_rollback_identical': True,
'kernels_sorting_and_local_ring_math_unchanged': True,
'derived_module_ast_sha256': hashlib.sha256(R._dump(tree).encode()).hexdigest()}
return tree, audit
def audit_carried_sequence(generated, applied, pending, initial_pending):
"""Bitwise FIFO accounting with a carried queue, never a new brain step."""
for name, value in (('generated', generated), ('applied', applied), ('pending', pending), ('initial', initial_pending)):
R.require(isinstance(value, np.ndarray) and value.dtype == np.float32 and value.ndim == 2 and
np.isfinite(value).all() and np.all(value >= 0), 'Invalid continued APL trace: ' + name)
R.require(generated.shape == applied.shape and pending.shape == initial_pending.shape == (18, generated.shape[1]),
'Continued APL sequence shapes differ')
steps = len(generated)
for tick in range(steps + 18):
actual = applied[tick] if tick < steps else pending[tick - steps]
expected = initial_pending[tick] if tick < 18 else generated[tick - 18]
R.require(np.array_equal(actual.view(np.uint32), expected.view(np.uint32)),
'Carried FIFO bitwise accounting failed at local application tick ' + str(tick))
return {'pass_all': True, 'apl_delay_steps': 18, 'steps': steps, 'sample_bits_preserved': True,
'initial_queue_from_parent': True, 'initial_queue_raw_sha256': R._raw(initial_pending),
'full_sequence_bitwise_audit_performed': True, 'brain_tail_advanced': False}
class ContinuedRuntime(R._Runtime):
def __init__(self, *args, parent, **kwargs):
super().__init__(*args, **kwargs)
self.parent = parent
self.global_offset = parent['global_next_tick']
self.restorations = 0
def install(self, *args, **kwargs):
super().install(*args, **kwargs)
R.require(self.effective_sha == self.parent['summary']['effective_weights_raw_sha256'], 'Retention changed effective weights')
def restore(self, v, g, ref, counts, ring, apl_ring, apl, upload):
R.require(self.installations == 1 and self.restorations == 0, 'Restore must occur once after fixed-weight installation')
R.require(not bool(counts.any().item()), 'Segment counts must start at zero')
state, terminal = self.parent['state'], self.parent['terminal']
for target, value, dtype in ((v, state['v'], self.torch.float64), (g, state['g'], self.torch.float64),
(ref, state['ref'], self.torch.int32),
(ring, rotate_spike_ring(terminal['spike_ring'], int(terminal['next_tick'])), self.torch.uint8),
(apl_ring, terminal['apl_pending'], self.torch.float32),
(apl.state, terminal['apl_state'], self.torch.float32)):
target.copy_(upload(value, dtype))
self.restorations += 1
def finish(self, *args, **kwargs):
R.require(self.restorations == 1, 'Continuation was not restored')
return super().finish(*args, **kwargs)
def audit_sequence(self, generated, applied, pending, delay):
R.require(delay == 18, 'Continuation changed D18')
return audit_carried_sequence(generated, applied, pending, self.parent['terminal']['apl_pending'])
def _timing_metadata(self, value):
result = dict(value)
next_tick, steps = self.global_offset + self.spec['steps'], self.spec['steps']
result.pop('initial_padding_positive_zero', None)
result.update(initial_queue_from_parent=True, initial_queue_raw_sha256=R._raw(self.parent['terminal']['apl_pending']),
global_tick_offset=self.global_offset, global_next_tick=next_tick, tick_coordinate='global',
segment_steps=steps, segment_generated_rows_applied=max(steps - 18, 0),
applied_parent_pending_rows=min(steps, 18),
applied_initial_zero_rows=min(steps, max(18 - self.global_offset, 0)),
applied_parent_generated_rows=min(steps, 18) - min(steps, max(18 - self.global_offset, 0)),
pending_segment_generated_rows=min(steps, 18),
pending_parent_generated_rows=max(18 - steps, 0) - max(18 - next_tick, 0),
pending_generated_count=min(next_tick, 18), pending_initial_zero_prefix=max(18 - next_tick, 0),
pending_application_tick_start=next_tick, pending_application_tick_stop_exclusive=next_tick + 18,
pending_generated_tick_start=max(0, next_tick - 18), pending_generated_tick_stop_exclusive=next_tick,
brain_tail_advanced_scope='no extra brain ticks beyond this explicit segment')
if self.mode == 'light':
result.update(generated_trace=None, applied_trace=None, full_sequence_bitwise_audit_performed=False,
scope='Original D18 ring math retained with restored queue; histories not recorded')
return result
def timing(self, steps, pending):
return self._timing_metadata(super().timing(steps, pending))
def summary(self, value):
result = super().summary(value)
result.update(checkpoint_continuation_used=True, global_tick_offset=self.global_offset,
global_next_tick=self.global_offset + self.spec['steps'], event_tick_scope='segment_local',
counts_scope='segment_only_not_lifetime', parent_checkpoint_manifest_sha256=self.parent['manifest_sha256'])
result.update(pending_generated_count=min(result['global_next_tick'], 18),
pending_initial_zero_prefix=max(18 - result['global_next_tick'], 0))
if 'apl_timing' in result:
result['apl_timing'] = self._timing_metadata(result['apl_timing'])
return result
def write_json(self, path, value):
if Path(path).name == 'apl-timing-audit.json':
value = self._timing_metadata(value)
return super().write_json(path, value)
def compile_continued(aj, runtime, mode):
tree, audit = derive_continued(Path(aj.__file__).read_bytes(), mode)
namespace = dict(aj.__dict__)
namespace.update(__name__=__name__ + '.derived_' + mode, __file__=str(Path(__file__).resolve()),
_runtime=runtime, _continuation=runtime, write_json=runtime.write_json,
audit_sequence=runtime.audit_sequence)
audit['diagnostic_namespace_binding'] = 'audit_sequence checks parent_pending + generated, not an assumed zero-initial queue'
exec(compile(tree, str(Path(__file__).resolve()) + '::continued-AJ-AST-' + mode, 'exec'), namespace)
return namespace['run_gpu'], audit
def _parent_spec(parent):
expected = parent['summary']['input_sha256']['spec.json']
paths = [Path(path) for path, sha in parent['provenance']['input_sha256'].items()
if Path(path).name == 'spec.json' and sha == expected]
R.require(len(paths) == 1, 'Ambiguous original parent case spec')
return R._read_json(paths[0])
def run_continued(case_dir, out_dir, build_root, overlay_file, overlay_manifest, checkpoint_dir, *, mode='light'):
R.require(sys.dont_write_bytecode, 'Use python -B; frozen sources must not receive bytecode writes')
R.require(mode in ('light', 'reference'), 'Invalid diagnostic mode')
case_dir, out_dir, build_root = R.execution_paths(case_dir, out_dir, build_root, (checkpoint_dir,))
spec = R._read_json(case_dir / 'spec.json')
parent_sha = spec.get('parent_checkpoint_manifest_sha256')
fresh_admission = dict(spec)
fresh_admission.pop('parent_checkpoint_manifest_sha256', None)
R.validate_spec(fresh_admission, case_dir)
parent = load_checkpoint(checkpoint_dir, parent_sha)
# A continuation may never create output or a build cache inside ANY
# immutable ancestor, not only its immediate checkpoint parent. This check
# precedes overlay imports, Torch imports, compilation and output creation.
case_dir, out_dir, build_root = R.execution_paths(case_dir, out_dir, build_root, parent['ancestry_directories'])
if 'global_tick_offset' in spec:
R.require(type(spec['global_tick_offset']) is int and spec['global_tick_offset'] == parent['global_next_tick'],
'Declared global offset disagrees with checkpoint chain')
R.require(parent['global_next_tick'] + spec['steps'] <= np.iinfo(np.int64).max, 'Absolute tick overflow')
helper, plan, meta, inputs = R.load_overlay(overlay_file, overlay_manifest)
R.require(R.digest(overlay_file) == R.digest(parent['directory'] / 'overlay-input.npz') and
R.digest(overlay_manifest) == R.digest(parent['directory'] / 'overlay-manifest.json'), 'Retention overlay snapshot/manifest changed')
previous = _parent_spec(parent)
for key in ('dt_ms', 'model', 'apl', 'apl_delay_steps'):
R.require(spec[key] == previous[key], 'Continuation changes frozen dynamics: ' + key)
for key in ('graph_dir', 'apl_bundle_dir'):
R.require(Path(spec[key]).resolve(strict=True) == Path(previous[key]).resolve(strict=True), 'Continuation changes input directory: ' + key)
R.require(R.digest(spec['coefficients_path']) == parent['summary']['input_sha256']['coefficients.json'], 'Continuation coefficients changed')
R.require(R.digest(Path(spec['graph_dir']) / 'weights.npy') == parent['summary']['input_sha256']['base_weights.npy'], 'Continuation base weights changed')
aj_dir = Path(meta['aj_model_dir']).resolve(strict=True)
protected = (aj_dir.parent, Path(meta['am_overlay_path']).resolve().parent, spec['graph_dir'], spec['apl_bundle_dir'],
*parent['ancestry_directories'],
*(Path(path).resolve(strict=True).parent for path in
(overlay_file, overlay_manifest, meta['mask_path'], meta['base_candidate_path'], spec['coefficients_path'])))
case_dir, out_dir, build_root = R.execution_paths(case_dir, out_dir, build_root, protected)
for path in (case_dir / 'spec.json', case_dir / 'requests.npz', Path(spec['graph_dir']) / 'weights.npy',
Path(spec['coefficients_path']), Path(checkpoint_dir).resolve() / MANIFEST, Path(R.__file__).resolve()):
inputs[str(path)] = R.digest(path)
source_sha = R.digest(__file__)
R._load('timing_queue', aj_dir / 'timing_queue.py', R.PINS['apl-timing-lab-v1/timing_queue.py'])
aj = R._load('phase4an_continued_sealed_aj_gpu', aj_dir / 'gpu_timing.py', R.AJ_SHA)
runtime = ContinuedRuntime(mode, spec, helper, plan, meta, inputs, aj.torch, overlay_file, overlay_manifest, parent=parent)
function, ast_audit = compile_continued(aj, runtime, mode)
try:
inherited = function(case_dir, out_dir, build_root)
R.require(runtime.finished and runtime.restorations == 1 and runtime.installations == 1 and inherited['complete'], 'Incomplete continuation')
R.require(inherited['effective_weights_raw_sha256'] == runtime.effective_sha, 'Effective weight identity changed')
R.require(source_sha == R.digest(__file__) and all(R.digest(path) == sha for path, sha in inputs.items()), 'Source/input changed during continuation')
load_checkpoint(checkpoint_dir, parent_sha)
summary = R._read_json(out_dir / 'summary.json')
R.require(summary == runtime.summary(inherited), 'Stored continuation summary mismatch')
provenance = {'schema': 'phase4an-continuation-runtime-v1', 'complete': True, 'mode': mode,
'source_file': str(Path(__file__).resolve()), 'source_sha256': source_sha, 'fresh_runtime_sha256': LIGHT_SHA,
'ast': ast_audit, 'input_sha256': inputs, 'old_globals_patched': False, 'weights_updated_during_run': False,
'learning_pass_claimed': False, 'global_tick_offset': runtime.global_offset,
'global_next_tick': runtime.global_offset + spec['steps'],
'parent_checkpoint_manifest_sha256': parent_sha}
for key, name in (('summary_sha256', 'summary.json'), ('finite_audit_sha256', 'finite-audit.json'),
('terminal_queues_sha256', 'terminal-queues.npz'), ('overlay_audit_sha256', 'overlay-audit.json'),
('state_sha256', 'state.npz')):
provenance[key] = R.digest(out_dir / name)
R._save(out_dir / 'runtime-provenance.json', provenance)
receipt = {'schema': 'phase4an-continuation-receipt-v1', 'complete': True,
'parent_checkpoint_dir': str(parent['directory']), 'parent_checkpoint_manifest_sha256': parent_sha,
'global_tick_offset': runtime.global_offset, 'global_next_tick': runtime.global_offset + spec['steps'],
'segment_steps': spec['steps'], 'counts_scope': 'segment_only_not_lifetime',
'payload_sha256': _tree_hashes(out_dir), 'weights_updated_during_run': False,
'clock_admission_edit': CLOCK_NEW, 'checkpoint_continuation_verified_by_GPU_comparison': False}
R._save(out_dir / RECEIPT, receipt)
sealed = seal_checkpoint(out_dir)
return dict(summary, checkpoint_manifest_sha256=sealed['sha256'])
except Exception as error:
if out_dir.is_dir() and not (out_dir / 'continuation-error.json').exists():
R._save(out_dir / 'continuation-error.json', {'complete': False, 'type': type(error).__name__,
'message': str(error), 'learning_update_forbidden': True})
raise
@@ -0,0 +1,559 @@
"""Engineering-only, fixed-weight trial execution derived from frozen AJ AST.
Public APIs run_light(case_dir,out_dir,build_root,overlay_file,overlay_manifest)
and run_reference(...) have identical arguments. Neither trains nor carries a
previous trial's neural state. The reference retains AJ's eager diagnostics;
the light function removes only an explicit diagnostic whitelist. Both accept
an explicit spec.max_steps instead of AJ's 1000-step admission bound.
The parent protocol owns conditioning, cue/tail schedules, frozen input/mask
authority and success criteria. A completed trial is not a learning PASS.
Before any scientific execution, the coordinator must verify externally fixed
hashes for the entire graph, APL bundle, coefficients/model, and preregistered
case spec (including max_steps). Inherited AJ consistency checks and this
adapter's input hashes are not a replacement for those external authorities.
"""
from __future__ import annotations
import ast
import copy
import hashlib
import importlib.util
import json
from pathlib import Path
import re
import sys
import numpy as np
AJ_SHA = 'af1975ba5e5e700208c0a598d3cbe648a3cf10885c6231239b6e4b28f76d173b'
AM_OVERLAY_SHA = '7f891d0ba79b393c96e75fd2dee2eddc51e57d8213eaac2779703bc2c18712b4'
MASK_SHA = '8b0fb6eb2e6c31da4d17dc20d890aa3f367e514f05642de5bad6b42a1611c05b'
V2_SHA = 'abb4574da85f00bd9ef61024c65993ef81d42651ff8d8d63c15dd69c6222471e'
PINS = {
'apl-timing-lab-v1/gpu_timing.py': AJ_SHA,
'apl-timing-lab-v1/timing_queue.py': '54df6d4818e2a0c26e96fb0aaf5d9b71cd964981706da937562d48af79aca14e',
'author-apl-lab-v2/gpu_apl.py': '69f5d893f47dafff4ef7c3186c3df0b9014666b32c8f4e943b57016bec92b18a',
'author-apl-lab-v2/legacy_apl_worker.py': '21ceec65fe42581dc55872010e0025d660e80358905f1cf4df1dadf341a3b291',
'author-runtime-lab-v1/gpu_backend.py': '03758d07572f4b44d272059d9308fac534268ad201b57bdd77d69fee95e67278',
'author-runtime-lab-v1/author_ext.cpp': '208a937d17bbb041b9bf6b25b89264ee566830b60609e58491b44583d8541f15',
'author-runtime-lab-v1/author_ext.cu': '669f166e18ce9a3f63fc723db8b0a8b74f7fa4757194b1bd65b4cf6008a7787e',
'author-fullgraph-lab-v1/oracle_full.py': 'd72003d98f0f1624b12126eff50bd361c43d888f4a64d90ba9247a58976f1961',
}
_LOADED = {}
_INSTALL = '_runtime.install(weights, graph, apl_out_mask, upload, out_dir, source_provenance, device)'
_FINISH = '_runtime.finish(ring, apl_ring, apl, steps, out_dir)'
_MATH_TARGETS = frozenset(('indptr', 'dst', 'global_nodes_gpu', 'base_weights', 'weights',
'local_apl', 'apl_out_mask', 'common_kc', 'local_kc', 'full_kc', 'kc_local_gpu', 'kc_full_gpu',
'v', 'g', 'ref', 'counts', 'requests', 'ring', 'apl_delay', 'apl_ring', 'requested_local_gpu',
'spikes', 'due', 'sources', 'degrees', 'offsets', 'edge_count', 'expanded', 'destination',
'order', 'ordered_edges', 'ordered_targets', 'boundary', 'starts', 'targets', 'full_spikes',
'drive', 'apl_state', 'h', 'slot', 'applied_h', 'delta_si'))
def require(ok, message):
if not ok:
raise ValueError(message)
def digest(path):
h = hashlib.sha256()
with Path(path).open('rb') as stream:
for block in iter(lambda: stream.read(8 << 20), b''):
h.update(block)
return h.hexdigest()
def _read_json(path):
def pairs(items):
out = {}
for key, value in items:
require(key not in out, 'Duplicate JSON key: ' + key)
out[key] = value
return out
return json.loads(Path(path).read_text(encoding='utf-8'), object_pairs_hook=pairs,
parse_constant=lambda value: (_ for _ in ()).throw(ValueError('Nonfinite JSON: ' + value)))
def _save(path, value):
with Path(path).open('x', encoding='utf-8') as stream:
json.dump(value, stream, sort_keys=True, indent=2, allow_nan=False)
stream.write('\n')
def _raw(value):
return hashlib.sha256(np.ascontiguousarray(value, dtype=value.dtype.newbyteorder('<')).tobytes()).hexdigest()
def _sha(value):
return isinstance(value, str) and re.fullmatch('[0-9a-f]{64}', value) is not None
def _load(name, path, expected):
path = Path(path).absolute()
raw = path.read_bytes()
require(hashlib.sha256(raw).hexdigest() == expected, 'Unpinned source: ' + str(path))
if name in sys.modules:
module = sys.modules[name]
require(_LOADED.get(name) == (path, expected, module), 'Existing unverified module binding: ' + name)
return module
spec = importlib.util.spec_from_file_location(name, path)
require(spec is not None and spec.loader is not None, 'Missing module loader')
module = importlib.util.module_from_spec(spec)
sys.modules[name] = module
try:
exec(compile(raw, str(path), 'exec'), module.__dict__)
except BaseException:
sys.modules.pop(name, None)
raise
_LOADED[name] = (path, expected, module)
return module
def validate_spec(spec, case_dir):
"""Admission policy only. No implicit early stopping or checkpoint support."""
require(type(spec.get('max_steps')) is int and 0 < spec['max_steps'] <= np.iinfo(np.int32).max,
'An explicit positive int32-range max_steps is mandatory')
require(type(spec.get('steps')) is int and 0 < spec['steps'] <= spec['max_steps'], 'steps exceeds its declared bound')
require(spec.get('dt_ms') == .1 and spec.get('duration_ms') == spec['steps'] * .1 and
type(spec.get('apl_delay_steps')) is int and spec['apl_delay_steps'] == 18,
'Frozen clock/duration/D18 contract changed')
require(spec.get('record_history') is False, 'Only wholebrain fresh-state trials are supported')
require(not (Path(case_dir) / 'arrays.npz').exists(), 'Initial arrays/checkpoint continuation are not supported in this version')
require(not any('checkpoint' in str(key).lower() or str(key) in ('initial_state', 'resume', 'continuation') for key in spec),
'Checkpoint/continuation configuration is unsupported')
require(spec.get('within_trial_weight_updates', False) is False, 'No within-trial plasticity is supported')
return spec
def load_overlay(overlay_file, manifest_file):
"""New manifest policy, frozen AM array validators; no old-global mutation.
phase4an-overlay-v1 requires overlay_name zero/static25/learned,
overlay_sha256, original mask_path/mask_sha256, external positive f32-mV
base_candidate_path/base_candidate_sha256, aj_model_dir,
am_overlay_path/am_overlay_sha256. Learned snapshots additionally require
parent_snapshot_sha256 and eligibility_v2_source_sha256=the frozen 4J pin.
These hashes bind provenance; the coordinator must verify the parent chain.
NPZ contains ONLY edge_id:int64[14232], plastic_delta_mv:float32[14232].
"""
overlay_file, manifest_file = Path(overlay_file).absolute(), Path(manifest_file).absolute()
manifest_sha = digest(manifest_file)
meta = _read_json(manifest_file)
keys = {'schema', 'overlay_name', 'overlay_sha256', 'mask_path', 'mask_sha256',
'base_candidate_path', 'base_candidate_sha256', 'aj_model_dir', 'am_overlay_path', 'am_overlay_sha256'}
require(isinstance(meta, dict) and keys <= meta.keys() and meta['schema'] == 'phase4an-overlay-v1', 'Invalid overlay manifest schema')
require(meta['overlay_name'] in ('zero', 'static25', 'learned'), 'Unknown overlay snapshot type')
learned_keys = {'parent_snapshot_sha256', 'eligibility_v2_source_sha256'}
allowed = keys | (learned_keys if meta['overlay_name'] == 'learned' else set())
require(not (meta.keys() - allowed), 'Unknown or inapplicable overlay manifest fields')
require(meta['mask_sha256'] == MASK_SHA and meta['am_overlay_sha256'] == AM_OVERLAY_SHA,
'Original mask or frozen AM helper authority changed')
require(_sha(meta['overlay_sha256']) and digest(overlay_file) == meta['overlay_sha256'], 'Overlay SHA mismatch')
inputs = {str(overlay_file): meta['overlay_sha256'], str(manifest_file): manifest_sha}
for label in ('mask', 'base_candidate', 'am_overlay'):
path = Path(meta[label + '_path'])
require(path.is_absolute() and _sha(meta[label + '_sha256']) and digest(path) == meta[label + '_sha256'],
'Missing/mismatched anchored input: ' + label)
inputs[str(path)] = meta[label + '_sha256']
helper = _load('phase4an_sealed_am_overlay', meta['am_overlay_path'], AM_OVERLAY_SHA)
mask = np.load(meta['mask_path'], allow_pickle=False)
base = np.load(meta['base_candidate_path'], allow_pickle=False)
require(mask.dtype == np.int64 and mask.shape == (helper.CANDIDATE_COUNT,), 'Original complete 14,232 mask is required')
with np.load(overlay_file, allow_pickle=False) as archive:
require(len(archive.files) == 2 and set(archive.files) == {'edge_id', 'plastic_delta_mv'}, 'No embedded base or additional overlay arrays allowed')
plan = helper.validate_arrays(archive['edge_id'], archive['plastic_delta_mv'], mask, base)
delta = plan['plastic_delta_mv']
if meta['overlay_name'] == 'zero':
require(not np.any(delta != 0), 'Zero snapshot contains a nonzero delta')
elif meta['overlay_name'] == 'static25':
require(helper.bit_equal(delta, np.multiply(np.float32(-.25), base)), 'static25 must equal the original f32 floor')
else:
require(_sha(meta.get('parent_snapshot_sha256')) and meta.get('eligibility_v2_source_sha256') == V2_SHA,
'Learned snapshot needs parent hash and explicit frozen eligibility-v2 authority')
aj_dir = Path(meta['aj_model_dir'])
require(aj_dir.is_absolute() and aj_dir.name == 'apl-timing-lab-v1', 'AJ must use its real original directory')
for relative, expected in PINS.items():
path = aj_dir.parent / relative
require(digest(path) == expected, 'Sealed dependency changed: ' + relative)
inputs[str(path)] = expected
require(all(digest(path) == expected for path, expected in inputs.items()), 'Input changed while loading')
return helper, plan, meta, inputs
def _dump(node):
return ast.dump(node, annotate_fields=True, include_attributes=False)
def _target_roots(node):
if isinstance(node, ast.Name):
return {node.id}
if isinstance(node, (ast.Tuple, ast.List)):
return set().union(*(_target_roots(value) for value in node.elts))
if isinstance(node, (ast.Subscript, ast.Attribute)):
return _target_roots(node.value)
return set()
def math_projection(function):
"""Ordered/control-structured frozen math statements, excluding diagnostics.
Whitelist explicitly covers state/weight initialization, request mutation,
expansion/stable grouping, original CUDA calls, LegacyAPL and ring/g write.
Tests also require the entire reference loop AST to equal the original.
"""
def block(statements):
result = []
for node in statements:
if isinstance(node, ast.FunctionDef) and node.name == 'upload':
result.append(copy.deepcopy(node))
elif isinstance(node, ast.Assign) and set().union(*(_target_roots(t) for t in node.targets)) & _MATH_TARGETS:
result.append(copy.deepcopy(node))
elif isinstance(node, ast.Expr) and isinstance(node.value, ast.Call):
text = ast.unparse(node.value.func)
if text.startswith('ext.') or text in ('requests.zero_', 'apl.spikes.zero_', 'apl_ring[slot].copy_'):
result.append(copy.deepcopy(node))
elif isinstance(node, (ast.With, ast.Try)):
result.extend(block(node.body))
elif isinstance(node, (ast.If, ast.For)):
body, other = block(node.body), block(node.orelse)
if body or other:
keep = copy.deepcopy(node)
keep.body, keep.orelse = body, other
result.append(keep)
return result
return _dump(ast.Module(body=block(function.body), type_ignores=[]))
def derive_ast(source_bytes, mode):
"""Derive only from exact AJ bytes; return ASTs plus an explicit edit receipt."""
require(mode in ('light', 'reference'), 'Unknown diagnostic mode')
require(hashlib.sha256(source_bytes).hexdigest() == AJ_SHA, 'AJ source pin mismatch')
source = ast.parse(source_bytes)
original = next(node for node in source.body if isinstance(node, ast.FunctionDef) and node.name == 'run_gpu')
loader = copy.deepcopy(next(node for node in source.body if isinstance(node, ast.FunctionDef) and node.name == 'load_inputs'))
bounds = [node for node in ast.walk(loader) if isinstance(node, ast.Compare) and
ast.unparse(node) == '0 < steps <= 1000']
require(len(bounds) == 1, 'Expected one explicit step-admission bound')
bounds[0].comparators[-1] = ast.parse("spec['max_steps']", mode='eval').body
function = copy.deepcopy(original)
body = next(node for node in function.body if isinstance(node, ast.Try)).body
anchor = next(i for i, node in enumerate(body) if ast.unparse(node) == 'effective_hash = hashlib.sha256()')
require(ast.unparse(body[anchor - 1]) == 'if bool(np.any(apl_out_mask)):\n weights[upload(apl_out_mask, torch.bool)] = 0.0',
'Installation no longer follows APL fast-edge zeroing')
body.insert(anchor, ast.parse(_INSTALL).body[0])
sync = [i for i, node in enumerate(body) if ast.unparse(node) == 'torch.cuda.synchronize(device)']
require(len(sync) == 1, 'Expected one post-loop synchronization')
body.insert(sync[0] + 1, ast.parse(_FINISH).body[0])
removed, replacements = [], []
class ReportingOnly(ast.NodeTransformer):
def note(self, node):
removed.append(_dump(node))
return None
def visit_Assign(self, node):
names = set().union(*(_target_roots(t) for t in node.targets))
if names & {'expanded_global', 'accepted_cpu', 'queue', 'apl_cpu', 'actual', 'finite', 'row'}:
return self.note(node)
if names == {'maps'} and isinstance(node.targets[0], ast.Subscript):
return self.note(node)
if names == {'timing'}:
require(ast.unparse(node.value.func) == 'audit_sequence', 'Unexpected timing assignment')
replacements.append(_dump(node))
return ast.parse('timing = _runtime.timing(steps, pending)').body[0]
if names == {'destination'}:
return [node, ast.parse('_runtime.arrivals(expanded, destination, ref)').body[0]]
if ast.unparse(node) == 'g[kc_local_gpu] = g[kc_local_gpu] - delta_si':
return [node, ast.parse('_runtime.after_tick(tick, v, g, ref, drive, apl_state, h, applied_h)').body[0]]
return node
def visit_AugAssign(self, node):
if _target_roots(node.target) & {'due_total', 'accepted_due_total'}:
return self.note(node)
return node
def visit_For(self, node):
text = ast.unparse(node.iter)
if text in ("(('drive', 2), ('state', 2), ('inhibition', 4064))", 'apl_cpu.items()', 'actual.items()'):
return self.note(node)
return self.generic_visit(node)
def visit_If(self, node):
if ast.unparse(node.test) in ("spec['record_history']", 'not finite'):
return self.note(node)
return self.generic_visit(node)
def visit_Expr(self, node):
if isinstance(node.value, ast.Call) and ast.unparse(node.value.func) in ('queue_output.write', 'hash_output.write'):
return self.note(node)
return node
def visit_With(self, node):
targets = [getattr(item.optional_vars, 'id', None) for item in node.items]
if targets == ['queue_output', 'hash_output', 'request_output']:
replacements.append(_dump(node.items[0])); replacements.append(_dump(node.items[1]))
node.items = [node.items[-1]]
return self.generic_visit(node)
if mode == 'light':
function = ReportingOnly().visit(function)
require(len(removed) == 20 and len(replacements) == 3, 'Reporting whitelist changed; stop instead of broad rewriting')
loops = lambda f: [node for node in ast.walk(f) if isinstance(node, ast.For) and
isinstance(node.target, ast.Name) and node.target.id == 'tick']
require(len(loops(original)) == len(loops(function)) == 1, 'Missing/duplicated trial loop')
if mode == 'reference':
require(_dump(loops(original)[0]) == _dump(loops(function)[0]), 'Reference loop must stay entirely unchanged')
original_math, derived_math = math_projection(original), math_projection(function)
require(original_math == derived_math, 'Ordered mathematical AST changed')
extra_calls = [ast.unparse(node) for node in ast.walk(function) if isinstance(node, ast.Call) and
ast.unparse(node.func).startswith('_runtime.')]
expected_calls = [_INSTALL, _FINISH]
if mode == 'light':
expected_calls += ['_runtime.arrivals(expanded, destination, ref)',
'_runtime.after_tick(tick, v, g, ref, drive, apl_state, h, applied_h)',
'_runtime.timing(steps, pending)']
require(sorted(extra_calls) == sorted(expected_calls), 'Unlisted runtime callback added')
tree = ast.fix_missing_locations(ast.Module(body=[loader, function], type_ignores=[]))
sha = lambda value: hashlib.sha256(value.encode('utf-8')).hexdigest()
receipt = {'mode': mode, 'aj_source_sha256': AJ_SHA, 'math_ast_sha256': sha(original_math),
'derived_math_ast_sha256': sha(derived_math), 'mathematical_ast_identical': True,
'reference_entire_tick_loop_identical': mode == 'reference',
'admission_edit': '0 < steps <= 1000 -> 0 < steps <= spec.max_steps',
'pre_loop_installation': _INSTALL, 'post_loop_terminal_recording': _FINISH,
'added_runtime_callbacks': extra_calls,
'removed_diagnostic_statements': removed, 'replaced_diagnostic_nodes': replacements,
'derived_module_ast_sha256': sha(_dump(tree)), 'old_globals_patched': False}
return tree, receipt
class FiniteObserver:
"""GPU scalar diagnostics only; never writes a neuronal/APL input tensor."""
def __init__(self, torch, device):
self.torch = torch
self.ok = torch.ones((), device=device, dtype=torch.bool)
self.first = torch.full((), -1, device=device, dtype=torch.int64)
self.accepted = torch.zeros((), device=device, dtype=torch.int64)
self.due = 0
self.ticks = 0
def arrivals(self, expanded, destination, ref):
self.due += expanded.numel()
self.accepted.add_((ref[destination.long()] == 0).to(self.torch.int64).sum())
def observe(self, tick, *values):
require(tick == self.ticks, 'Finite observer tick coverage changed')
finite = self.torch.ones_like(self.ok)
for value in values:
finite.logical_and_(self.torch.isfinite(value).all())
self.first = self.torch.where((self.first < 0) & (~finite), self.torch.full_like(self.first, tick), self.first)
self.ok.logical_and_(finite)
self.ticks += 1
def result(self):
return {'all_finite': bool(self.ok.item()), 'first_failure_tick': int(self.first.item()),
'finite_observed_ticks': self.ticks, 'arriving_edges': self.due,
'accepted_arriving_edges': int(self.accepted.item())}
def _copy_input(source, target, expected):
with Path(source).open('rb') as src, Path(target).open('xb') as dst:
for block in iter(lambda: src.read(8 << 20), b''):
dst.write(block)
require(digest(target) == expected, 'Copied input differs')
class _Runtime:
def __init__(self, mode, spec, helper, plan, meta, inputs, torch, overlay_file, manifest_file):
self.mode, self.spec, self.helper, self.plan, self.meta, self.inputs = mode, spec, helper, plan, meta, inputs
self.torch, self.overlay_file, self.manifest_file = torch, Path(overlay_file).absolute(), Path(manifest_file).absolute()
self.observer = None
self.installations = 0
self.finished = False
self.result = None
def install(self, weights, graph, apl_mask, upload, out_dir, inherited, device):
require(self.installations == 0, 'Repeated installation is forbidden')
require(len(graph['nodes']) == 166700 and len(graph['edge_id']) == self.helper.EDGE_COUNT,
'Only the intact full graph is permitted')
require(int(apl_mask.sum()) == 5845, 'Original fast APL mask changed')
mapping = self.helper.map_installation(graph['weights'], graph['edge_id'], apl_mask, self.plan)
require(mapping['missing_candidate_count'] == 0, 'All original candidates must be present')
rows = mapping['rows']
if len(rows):
index = upload(rows, self.torch.int64)
nonzero = self.plan['plastic_delta_mv'] != np.float32(0)
change = upload(self.plan['plastic_delta_mv'][nonzero], self.torch.float64) * 0.001
weights[index] = weights[index] + change
require(self.helper.bit_equal(weights[index].cpu().numpy(), mapping['expected_si']), 'Independent SI installation mismatch')
raw_base, expected = self.helper.expected_hashes(graph['weights'], apl_mask, mapping)
actual = hashlib.sha256()
for start in range(0, weights.numel(), 1 << 20):
actual.update(weights[start:start + (1 << 20)].cpu().numpy().astype('<f8', copy=False).tobytes())
require(actual.hexdigest() == expected, 'Effective weight bytes differ from declared overlay')
self.effective_sha = expected
self.observer = FiniteObserver(self.torch, device)
self.installations += 1
_copy_input(self.overlay_file, out_dir / 'overlay-input.npz', self.meta['overlay_sha256'])
_copy_input(self.manifest_file, out_dir / 'overlay-manifest.json', self.inputs[str(self.manifest_file)])
_save(out_dir / 'overlay-audit.json', {'schema': 'phase4an-overlay-install-v1', 'complete': True,
'overlay_name': self.meta['overlay_name'], 'candidate_count': len(self.plan['edge_id']),
'nonzero_delta_count': len(rows), 'am_helper_sha256': AM_OVERLAY_SHA, 'mask_sha256': MASK_SHA,
'base_candidate_sha256': self.meta['base_candidate_sha256'], 'delta_raw_sha256': _raw(self.plan['plastic_delta_mv']),
'base_weights_raw_sha256': raw_base, 'effective_weights_raw_sha256': expected,
'delta_dtype': 'float32', 'delta_units': 'mV', 'state_dtype': 'float64', 'bridge': self.helper.BRIDGE,
'zero_rows_untouched': True, 'weights_updated_during_run': False, 'apl_fast_edges_zeroed': int(apl_mask.sum()),
'parent_snapshot_sha256': self.meta.get('parent_snapshot_sha256'),
'eligibility_v2_source_sha256': self.meta.get('eligibility_v2_source_sha256'),
'parent_chain_verified_by_runtime': False, 'inherited_AJ_provenance_only': inherited})
def arrivals(self, expanded, destination, ref):
self.observer.arrivals(expanded, destination, ref)
def after_tick(self, tick, v, g, ref, drive, state, h, applied):
self.observer.observe(tick, v, g, ref, drive, state, h, applied)
def finish(self, ring, apl_ring, apl, steps, out_dir):
require(not self.finished and self.installations == 1, 'Invalid terminal recording sequence')
if self.mode == 'light':
self.result = self.observer.result()
require(self.result['finite_observed_ticks'] == steps, 'Missing finite checks')
else:
# Reaching this point means AJ's original per-tick finite check passed.
self.result = {'all_finite': True, 'first_failure_tick': -1, 'finite_observed_ticks': steps}
_save(out_dir / 'finite-audit.json', dict(self.result, mode=self.mode,
scope='every completed tick; no neuronal or APL tensor mutated by observer'))
raw_ring, raw_apl = ring.cpu().numpy().copy(), apl_ring.cpu().numpy().copy()
head = steps % len(raw_apl)
pending = np.concatenate((raw_apl[head:], raw_apl[:head]), axis=0)
apl_state = apl.state.cpu().numpy().copy()
require(raw_ring.dtype == np.uint8 and raw_ring.shape == (19, 166700) and
apl_state.dtype == pending.dtype == np.float32 and apl_state.shape == (2,) and pending.shape == (18, 4064),
'Terminal queue dtype/shape changed')
with (out_dir / 'terminal-queues.npz').open('xb') as target:
np.savez_compressed(target, spike_ring=raw_ring, apl_state=apl_state, apl_pending=pending,
next_tick=np.asarray(steps, dtype=np.int64))
self.finished = True
require(self.result['all_finite'], 'Nonfinite value observed; trial is invalid and must not produce a learning update')
def timing(self, steps, pending):
return {'apl_delay_steps': 18, 'steps': steps, 'pending_generated_count': min(steps, 18),
'pending_initial_zero_prefix': max(18 - steps, 0), 'brain_tail_advanced': False,
'full_sequence_bitwise_audit_performed': False,
'scope': 'Original D18 ring AST retained; generated/applied histories not recorded'}
def summary(self, value):
result = dict(value)
result.pop('learning', None)
result.update(diagnostic_mode=self.mode, within_trial_weight_updates=False,
outer_protocol=self.spec.get('outer_protocol'), outer_learning_status='not evaluated by this single-trial executor',
learning_pass_claimed=False, checkpoint_continuation_used=False,
max_steps=self.spec['max_steps'], explicit_steps_executed=self.spec['steps'],
entire_neural_trial_executed=True, hidden_early_stop=False,
full_APL_history_recorded=self.mode == 'reference', full_tick_state_hashes_recorded=self.mode == 'reference',
full_queue_hashes_recorded=self.mode == 'reference',
full_sequence_bitwise_audit_performed=self.mode == 'reference',
first_failure_tick=self.result['first_failure_tick'],
pass_all_scope='single fixed-weight trial finite/recording integrity only; no learning or equivalence claim')
if self.mode == 'light':
result.update(arriving_edges=self.result['arriving_edges'], accepted_arriving_edges=self.result['accepted_arriving_edges'])
if 'apl_timing' in result:
result['apl_timing'] = dict(result['apl_timing'], generated_trace=None, applied_trace=None,
full_sequence_bitwise_audit_performed=False)
return result
def write_json(self, path, value):
path = Path(path)
if path.name == 'summary.json':
value = self.summary(value)
elif path.name == 'apl-timing-audit.json' and self.mode == 'light':
value = dict(value)
value.update(generated_trace=None, applied_trace=None, full_sequence_bitwise_audit_performed=False)
_save(path, value)
def compile_runtime(aj_module, runtime, mode):
"""Fresh namespace; __file__ is this real adapter, not a disguised old file."""
tree, audit = derive_ast(Path(aj_module.__file__).read_bytes(), mode)
namespace = dict(aj_module.__dict__)
namespace.update(__name__=__name__ + '.derived_' + mode, __file__=str(Path(__file__).absolute()),
_runtime=runtime, write_json=runtime.write_json)
exec(compile(tree, str(Path(__file__).absolute()) + '::AJ-AST-' + mode, 'exec'), namespace)
audit.update(derived_namespace_file=namespace['__file__'], original_module_file=str(Path(aj_module.__file__).absolute()),
output_metadata_binding='new write_json only labels recording/within-trial scope; no old global mutated')
return namespace['run_gpu'], audit
def execution_paths(case_dir, out_dir, build_root, protected=()):
"""Resolve directory aliases before containment checks, without any writes.
Only data/source/output directories are canonicalized here. This helper
must not be used for a Python executable: resolving a venv entry changes
interpreter semantics. The coordinator must keep input paths stable during
execution; this is not a defense against concurrent filesystem mutation.
"""
case_dir = Path(case_dir).resolve(strict=True)
require(case_dir.is_dir(), 'Input case must be an existing directory')
out_dir, build_root = (Path(path).resolve(strict=False) for path in (out_dir, build_root))
require(not out_dir.exists(), 'Output must be a new directory')
require(not build_root.exists() or build_root.is_dir(), 'Build cache must be a directory')
roots = [case_dir]
for path in protected:
root = Path(path).resolve(strict=True)
require(root.is_dir(), 'Protected input/source root must be a directory')
roots.append(root)
require(not any(out_dir.is_relative_to(root) or build_root.is_relative_to(root) for root in roots),
'Output/build paths cannot write into frozen source or input directories')
return case_dir, out_dir, build_root
def _run(mode, case_dir, out_dir, build_root, overlay_file, overlay_manifest):
require(sys.dont_write_bytecode, 'Use python -B; frozen dependencies must not receive bytecode writes')
case_dir, out_dir, build_root = execution_paths(case_dir, out_dir, build_root)
spec = validate_spec(_read_json(case_dir / 'spec.json'), case_dir)
helper, plan, meta, inputs = load_overlay(overlay_file, overlay_manifest)
own_sha = digest(__file__)
inputs[str(case_dir / 'spec.json')] = digest(case_dir / 'spec.json')
inputs[str(case_dir / 'requests.npz')] = digest(case_dir / 'requests.npz')
inputs[str(Path(spec['graph_dir']) / 'weights.npy')] = digest(Path(spec['graph_dir']) / 'weights.npy')
aj_dir = Path(meta['aj_model_dir']).resolve(strict=True)
protected = (aj_dir.parent, Path(meta['am_overlay_path']).resolve(strict=True).parent,
Path(spec['graph_dir']), Path(spec['apl_bundle_dir']),
*(Path(path).resolve(strict=True).parent for path in
(overlay_file, overlay_manifest, meta['mask_path'], meta['base_candidate_path'], spec['coefficients_path'])))
case_dir, out_dir, build_root = execution_paths(case_dir, out_dir, build_root, protected)
_load('timing_queue', aj_dir / 'timing_queue.py', PINS['apl-timing-lab-v1/timing_queue.py'])
aj = _load('phase4an_sealed_aj_gpu', aj_dir / 'gpu_timing.py', AJ_SHA)
runtime = _Runtime(mode, spec, helper, plan, meta, inputs, aj.torch, overlay_file, overlay_manifest)
function, ast_audit = compile_runtime(aj, runtime, mode)
try:
inherited = function(case_dir, out_dir, build_root)
require(runtime.finished and runtime.installations == 1 and inherited['complete'], 'Incomplete single-trial execution')
require(inherited['effective_weights_raw_sha256'] == runtime.effective_sha, 'Effective weight identity changed during trial')
require(digest(__file__) == own_sha and all(digest(path) == expected for path, expected in inputs.items()),
'Source or input changed during execution')
summary = _read_json(out_dir / 'summary.json')
require(summary == runtime.summary(inherited), 'Stored summary does not bind actual trial')
_save(out_dir / 'runtime-provenance.json', {'schema': 'phase4an-reporting-adapter-v1', 'complete': True,
'source_file': str(Path(__file__).absolute()), 'source_sha256': own_sha, 'mode': mode, 'ast': ast_audit,
'input_sha256': inputs, 'old_globals_patched': False, 'math_changed': False,
'admission_bound_changed_explicitly': True, 'weights_updated_during_run': False,
'learning_pass_claimed': False, 'parent_chain_verification_owned_by_coordinator': True,
'summary_sha256': digest(out_dir / 'summary.json'), 'finite_audit_sha256': digest(out_dir / 'finite-audit.json'),
'terminal_queues_sha256': digest(out_dir / 'terminal-queues.npz'),
'overlay_audit_sha256': digest(out_dir / 'overlay-audit.json')})
return summary
except Exception as error:
if out_dir.is_dir() and not (out_dir / 'runtime-error.json').exists():
_save(out_dir / 'runtime-error.json', {'complete': False, 'type': type(error).__name__, 'message': str(error),
'learning_update_forbidden': True})
raise
def run_light(case_dir, out_dir, build_root, overlay_file, overlay_manifest):
return _run('light', case_dir, out_dir, build_root, overlay_file, overlay_manifest)
def run_reference(case_dir, out_dir, build_root, overlay_file, overlay_manifest):
return _run('reference', case_dir, out_dir, build_root, overlay_file, overlay_manifest)
@@ -0,0 +1,118 @@
"""Explicit, order-independent random streams for an unregistered protocol.
This module does not select odors, generate neural input, or run experiments.
Every independent stream has a semantic key and a SHA-derived 63-bit seed.
Intentional common-random-number controls must be declared as aliases of a
same-split stream. Aliases are never counted as independent replicates.
"""
from __future__ import annotations
import hashlib
import json
SCHEMA = "phase4an-seed-registry-v1"
SPLITS = frozenset(("training", "evaluation", "design"))
def _require(condition, message):
if not condition:
raise ValueError(message)
def _canonical(value):
return json.dumps(value, sort_keys=True, separators=(",", ":"),
ensure_ascii=False, allow_nan=False)
def _key(key):
_require(isinstance(key, (tuple, list)) and len(key) > 0,
"A nonempty list/tuple semantic key is required")
_require(all(type(v) is str and v and v.strip() == v for v in key),
"Semantic key tokens must be nonempty, unpadded strings")
return tuple(key)
def seed_for(master, split, key):
_require(type(master) is int and 0 <= master < 2**63, "master must be a 63-bit integer")
_require(type(split) is str and split in SPLITS, "Unknown stream split")
key = _key(key)
message = _canonical([SCHEMA, master, split, list(key)]).encode("utf-8")
full_sha = hashlib.sha256(message).hexdigest()
return int(full_sha[:16], 16) & ((1 << 63) - 1), full_sha
class SeedRegistry:
"""Fail closed on duplicate keys, collisions or cross-split aliasing.
No collision rehashing or seed search is performed: such a failure must be
resolved before registration, never after observing a neural response.
"""
def __init__(self, master):
seed_for(master, "design", ("validate-master",))
self.master = master
self._streams = {}
self._aliases = {}
self._seed_to_key = {}
def add(self, split, key):
key = _key(key)
identity = (split, key)
_require(identity not in self._streams and identity not in self._aliases,
"Duplicate semantic stream key")
seed, full_sha = seed_for(self.master, split, key)
_require(seed not in self._seed_to_key, "Seed collision: stop before registration")
row = dict(split=split, key=list(key), seed=seed, derivation_sha256=full_sha)
self._streams[identity] = row
self._seed_to_key[seed] = identity
return seed
def alias(self, split, key, source_split, source_key, *, reason):
"""Declare paired controls, never silently create/reuse a stream."""
key, source_key = _key(key), _key(source_key)
identity, source = (split, key), (source_split, source_key)
_require(split == source_split and split in SPLITS,
"Cross-split random-stream sharing is forbidden")
_require(identity not in self._streams and identity not in self._aliases,
"Duplicate semantic stream key")
_require(source in self._streams, "Alias must target an independent stream, not another alias")
_require(type(reason) is str and bool(reason.strip()), "Intentional sharing requires a reason")
seed = self._streams[source]["seed"]
self._aliases[identity] = dict(split=split, key=list(key), seed=seed,
source_split=source_split, source_key=list(source_key), reason=reason,
independent_replicate=False)
return seed
def manifest(self):
streams = [dict(self._streams[k]) for k in sorted(self._streams)]
aliases = [dict(self._aliases[k]) for k in sorted(self._aliases)]
split_counts = {split: sum(r["split"] == split for r in streams) for split in sorted(SPLITS)}
result = dict(schema=SCHEMA, master=self.master,
algorithm="SHA-256 canonical JSON [schema,master,split,key]; first64 bits masked to63",
independent_streams=streams, aliases=aliases,
independent_count=len(streams), alias_count=len(aliases), split_counts=split_counts,
all_independent_seeds_unique=True, training_evaluation_seed_overlap=0,
aliases_count_as_independent_replicates=False,
training_executed=False, protocol_approval_claimed=False)
result["payload_sha256"] = hashlib.sha256(_canonical(result).encode("utf-8")).hexdigest()
# JSON round-trip makes nested data independent of the mutable registry.
return json.loads(_canonical(result))
def validate_manifest(manifest):
"""Reconstruct every seed/alias and require the complete canonical payload."""
_require(isinstance(manifest, dict) and manifest.get("schema") == SCHEMA, "Invalid seed manifest")
payload = {key: value for key, value in manifest.items() if key != "payload_sha256"}
actual_sha = hashlib.sha256(_canonical(payload).encode("utf-8")).hexdigest()
_require(manifest.get("payload_sha256") == actual_sha, "Actual manifest payload SHA mismatch")
rebuilt = SeedRegistry(manifest["master"])
for row in manifest["independent_streams"]:
rebuilt.add(row["split"], row["key"])
for row in manifest["aliases"]:
rebuilt.alias(row["split"], row["key"], row["source_split"], row["source_key"], reason=row["reason"])
# Python dict equality conflates False/0 and 1/1.0; the serialized contract
# must retain JSON Boolean versus integer versus floating-point identity.
_require(_canonical(manifest) == _canonical(rebuilt.manifest()),
"Seed manifest differs from full independent reconstruction")
return True
@@ -0,0 +1,153 @@
import unittest
import numpy as np
import pandas as pd
import biological_contract as b
def fixture():
types = ["KC-a", "KC-b", "PAM15", "PPL103", "MBON01", "MBON12", "other-DAN"]
classes = ["Kenyon_Cell", "Kenyon_Cell", "DAN", "DAN", "MBON", "MBON", "DAN"]
instances = ["KC-a_L", "KC-b_R", "PAM15(y5B'2a)_L", "PPL103(y2a'1)_R",
"MBON01(y5B'2a)_L", "MBON12(y2a'1)_R", "other-DAN"]
meta = pd.DataFrame(dict(neuron_idx=np.arange(7, dtype=np.int32),
body_id=np.arange(7, dtype=np.uint64) + np.uint64(2**63),
annotation_type=pd.Series(types, dtype=object), annotation_class=pd.Series(classes, dtype=object),
annotation_instance=pd.Series(instances, dtype=object)))
dan = meta.iloc[[2, 3, 6]].drop(columns="annotation_class").copy()
mbon = meta.iloc[[4, 5]].drop(columns="annotation_class").copy()
dan["mb_compartment_verbatim"] = ["y5B'2a", "y2a'1", None]
dan["mb_compartment_mapping_status"] = ["explicit_single_parenthesized_token"] * 2 + ["no_parenthesized_token"]
mbon["mb_compartment_verbatim"] = ["y5B'2a", "y2a'1"]
mbon["mb_compartment_mapping_status"] = "explicit_single_parenthesized_token"
src = np.array([0, 0, 1, 1, 2, 3], np.int32)
dst = np.array([4, 5, 4, 5, 4, 5], np.int32)
raw = np.array([4, 8, 12, 16, 20, 24], np.int64)
def table(ids):
rows = []
for i in ids:
row = dict(edge_id=i, raw_synapse_count=int(raw[i]))
for label, node in (("source", src[i]), ("target", dst[i])):
for key in ("neuron_idx", "body_id", "annotation_type", "annotation_instance"):
row[label + "_" + key] = meta.iloc[node][key]
row["target_compartment_verbatim"] = "y5B'2a" if dst[i] == 4 else "y2a'1"
rows.append(row)
out = pd.DataFrame(rows)
for key in ("source_neuron_idx", "target_neuron_idx"):
out[key] = out[key].astype(np.int32)
for key in ("source_body_id", "target_body_id"):
out[key] = out[key].astype(np.uint64)
# Match the observed frozen parquet object dtype; local pandas 3 may
# infer its new string extension dtype from constructor literals.
for prefix in ("source", "target"):
for suffix in ("annotation_type", "annotation_instance"):
out[prefix + "_" + suffix] = out[prefix + "_" + suffix].astype(object)
out["edge_id"] = out.edge_id.astype(np.int64)
out["raw_synapse_count"] = out.raw_synapse_count.astype(np.int64)
return out
candidates = table(range(4))
candidates["plasticity_polarity"] = np.int64(-1)
confirmed = table([4, 5])
confirmed["source_compartment_verbatim"] = confirmed.target_compartment_verbatim
confirmed["exact_compartment_match"] = True
return meta, dan, mbon, confirmed, candidates, src, dst, raw
class Biology(unittest.TestCase):
def test_complete_graph_binds_every_edge_array_length(self):
meta, *_, src, dst, raw = fixture()
graph = dict(nodes=np.arange(7, dtype=np.int32), edge_id=np.arange(6, dtype=np.int64),
src=src, dst=dst, signed_raw=raw)
b._complete_graph_identity(meta, graph, n=7, e=6)
for names in (("src",), ("dst",), ("signed_raw",), ("src", "dst", "signed_raw")):
changed = dict(graph)
for name in names:
changed[name] = changed[name][:-1]
with self.subTest(names=names), self.assertRaisesRegex(ValueError, "endpoint/weight lengths"):
b._complete_graph_identity(meta, changed, n=7, e=6)
for name in ("nodes", "edge_id"):
changed = dict(graph)
changed[name] = changed[name].reshape(1, -1)
with self.subTest(name=name), self.assertRaises(ValueError):
b._complete_graph_identity(meta, changed, n=7, e=6)
def test_exact_id_compartment_and_real_edge_contract(self):
values = fixture()
populations, audit = b.audit_interface_tables(*values)
self.assertEqual(populations["PAM15"].tolist(), [2])
self.assertEqual(populations["MBON12"].tolist(), [5])
self.assertEqual(audit["whole_DAN_mapping_unknown"], 1)
self.assertFalse(audit["global_unknowns_filled"])
self.assertEqual(audit["all_candidate_edges_retained"], 4)
self.assertEqual(audit["interfaces"]["PAM15"]["candidate_contacts"], 16)
self.assertEqual(audit["interfaces"]["PPL103"]["confirmed_DAN_MBON_contacts"], 24)
self.assertGreater(audit["interfaces"]["PAM15"]["DAN_members"][0]["body_id"], 2**63)
self.assertTrue(pd.isna(values[1].iloc[-1].mb_compartment_verbatim))
def test_unknown_selected_mapping_is_not_filled_from_type(self):
for field, new_value in (("mb_compartment_verbatim", None),
("mb_compartment_mapping_status", "unknown")):
values = fixture()
values[1].loc[2, field] = new_value
with self.assertRaisesRegex(ValueError, "Unknown or mismatched"):
b.audit_interface_tables(*values)
def test_changed_body_id_or_type_or_instance_rejected(self):
for field, new_value in (("body_id", np.uint64(5)), ("annotation_type", "PAM_OTHER"),
("annotation_instance", "inferred")):
values = fixture()
values[1].loc[2, field] = new_value
with self.assertRaisesRegex(ValueError, "Identity field mismatch"):
b.audit_interface_tables(*values)
def test_unmapped_dan_cannot_be_silently_removed(self):
values = list(fixture())
values[1] = values[1].iloc[:2].copy()
with self.assertRaisesRegex(ValueError, "complete exact-annotation"):
b.audit_interface_tables(*values)
def test_missing_actual_edge_fails_not_guessed(self):
values = list(fixture())
values[3] = values[3].iloc[:1].copy()
with self.assertRaisesRegex(ValueError, "unknown or mismatched actual target"):
b.audit_interface_tables(*values)
def test_edge_identity_and_contacts_and_polarity(self):
changes = ((3, "source_body_id", np.uint64(8)),
(3, "target_annotation_type", "MBON_OTHER"),
(4, "raw_synapse_count", np.int64(1)),
(4, "plasticity_polarity", np.int64(1)))
for which, field, value in changes:
values = fixture()
values[which].loc[values[which].index[0], field] = value
with self.subTest(field=field), self.assertRaises(ValueError):
b.audit_interface_tables(*values)
def test_false_or_nonboolean_confirmed_flag_rejected(self):
for value in (False, "True"):
values = fixture()
values[3]["exact_compartment_match"] = value
with self.assertRaisesRegex(ValueError, "unknown compartment"):
b.audit_interface_tables(*values)
def test_graph_endpoint_and_sign_change_rejected(self):
for where in (5, 6, 7):
values = fixture()
values[where][0] = -1 if where == 7 else 6
with self.assertRaises(ValueError):
b.audit_interface_tables(*values)
def test_schema_and_duplicate_identity_rejected(self):
values = list(fixture())
values[0] = values[0].drop(columns="annotation_class")
with self.assertRaises(ValueError):
b.audit_interface_tables(*values)
values = fixture()
values[0].loc[1, "body_id"] = values[0].loc[0, "body_id"]
with self.assertRaises(ValueError):
b.audit_interface_tables(*values)
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,298 @@
"""Tiny temporary CPU fixtures, never scientific events or a real brain run."""
import copy
import json
from pathlib import Path
import tempfile
from types import SimpleNamespace
import unittest
from unittest.mock import patch
import numpy as np
import conditioning_epoch as m
import conditioning_update as update
from test_conditioning_update import BRIDGE, SOURCE
def save(path, value):
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(value, sort_keys=True, allow_nan=False), encoding='utf-8')
class TinyFixture:
def __init__(self, root):
self.root = root
self.inputs = root / 'inputs'; self.inputs.mkdir()
self.case = self.inputs / 'case'; self.case.mkdir()
self.graph = self.inputs / 'graph'; self.graph.mkdir()
self.apl = self.inputs / 'apl'; self.apl.mkdir()
self.checkpoint = root / 'runtime'; self.checkpoint.mkdir()
self.authority_dir = root / 'authority'; self.authority_dir.mkdir()
self.output_parent = root / 'outputs'; self.output_parent.mkdir()
self.authority_path = self.authority_dir / 'epoch.json'
self.mask = np.array([1, 3], np.int64)
self.kc = np.array([1, 3], np.int32)
self.base = np.array([.275, .55], np.float32)
self.before = np.zeros(2, np.float32)
graph = dict(nodes=np.arange(5, dtype=np.int32), edge_id=np.arange(6, dtype=np.int64),
src=np.array([0, 1, 1, 3, 3, 4], np.int32), dst=np.array([1, 2, 4, 2, 4, 0], np.int32),
signed_raw=np.ones(6, np.int64), ptr=np.array([0, 1, 3, 3, 5, 6], np.int64), weights=np.full(6, .000275))
for name, array in graph.items():
np.save(self.graph / (name + '.npy'), array)
for name, array in dict(input_matrix=np.zeros((2, 5), np.float32),
output_matrix=np.zeros((2, 2), np.float32), kc_globals=self.kc, apl_globals=np.array([0, 4], np.int32)).items():
np.save(self.apl / (name + '.npy'), array)
np.save(self.inputs / 'kc.npy', self.kc)
np.save(self.inputs / 'mask.npy', self.mask)
np.save(self.inputs / 'base.npy', self.base)
np.savez_compressed(self.inputs / 'mapping.npz', edge_id=self.mask,
source_neuron_idx=graph['src'][self.mask], target_neuron_idx=graph['dst'][self.mask],
target_compartment_verbatim=np.array(["y5B'2a", 'a1']))
np.savez_compressed(self.inputs / 'before.npz', edge_id=self.mask, plastic_delta_mv=self.before)
save(self.inputs / 'coeff.json', {'synthetic': True})
source_root = Path(m.__file__).resolve().parent
source_files = dict(runtime_light_source=source_root / 'runtime_light.py',
runtime_continuation_source=source_root / 'runtime_continuation.py',
conditioning_update_source=source_root / 'conditioning_update.py',
eligibility_bridge_source=BRIDGE.resolve(), phase4j_source=SOURCE.resolve(),
am_overlay_source=source_root.parent / 'wholebrain-plastic-interface-v1/overlay.py')
metadata = dict(schema='phase4an-overlay-v1', overlay_name='zero',
overlay_sha256=m._digest(self.inputs / 'before.npz'), mask_path=str(self.inputs / 'mask.npy'),
mask_sha256=m._digest(self.inputs / 'mask.npy'), base_candidate_path=str(self.inputs / 'base.npy'),
base_candidate_sha256=m._digest(self.inputs / 'base.npy'), aj_model_dir=str(self.graph),
am_overlay_path=str(source_files['am_overlay_source']), am_overlay_sha256=m._digest(source_files['am_overlay_source']))
save(self.inputs / 'before.json', metadata)
spec = dict(name='synthetic', steps=4, max_steps=4, dt_ms=.1, duration_ms=.4,
apl_delay_steps=18, record_history=False, within_trial_weight_updates=False,
graph_dir=str(self.graph), apl_bundle_dir=str(self.apl), coefficients_path=str(self.inputs / 'coeff.json'))
save(self.case / 'spec.json', spec)
np.savez_compressed(self.case / 'requests.npz', tick=np.array([0], np.int32), neuron_idx=np.array([1], np.int32))
state = dict(step=np.array([0], np.int32), neuron_idx=np.array([1], np.int32),
counts=np.array([0, 1, 0, 0, 0], np.int32), v=np.full(5, -.052), g=np.zeros(5), ref=np.zeros(5, np.int32))
np.savez_compressed(self.checkpoint / 'state.npz', **state)
(self.checkpoint / 'overlay-input.npz').write_bytes((self.inputs / 'before.npz').read_bytes())
(self.checkpoint / 'overlay-manifest.json').write_bytes((self.inputs / 'before.json').read_bytes())
summary = dict(nodes=5, edges=6, diagnostic_mode='light', effective_weights_raw_sha256='e' * 64)
save(self.checkpoint / 'summary.json', summary)
provenance = dict(input_sha256={str(path): m._digest(path) for path in
(self.case / 'spec.json', self.case / 'requests.npz', self.inputs / 'before.npz', self.inputs / 'before.json')})
save(self.checkpoint / 'runtime-provenance.json', provenance)
save(self.checkpoint / 'overlay-audit.json', dict(delta_raw_sha256=self.raw(self.before), effective_weights_raw_sha256='e' * 64))
paths = dict(spec=self.case / 'spec.json', requests=self.case / 'requests.npz',
overlay_before=self.inputs / 'before.npz', overlay_manifest_before=self.inputs / 'before.json',
state=self.checkpoint / 'state.npz', kc=self.inputs / 'kc.npy', canonical_kc=self.apl / 'kc_globals.npy',
candidate_map=self.inputs / 'mapping.npz', mask=self.inputs / 'mask.npy', base_candidate=self.inputs / 'base.npy', **source_files)
self.value = dict(schema=m.SCHEMA, epoch_id='synthetic/epoch-0', diagnostic_mode='light',
command=dict(steps=4, compartment="y5B'2a", us_tick=1, d_command=1., frozen=False, trace_backend='numpy'),
checkpoint_dir=str(self.checkpoint), checkpoint_manifest_sha256=None,
files={role: dict(path=str(path), sha256=m._digest(path)) for role, path in paths.items()},
frozen_inputs={str(path): m._digest(path) for path in [
*(self.graph / name for name in m.GRAPH_FILES), *(self.apl / name for name in m.APL_FILES), self.inputs / 'coeff.json']})
self.reseal()
@staticmethod
def raw(array):
return __import__('hashlib').sha256(array.tobytes()).hexdigest()
def reseal(self):
files = {path.name: m._digest(path) for path in self.checkpoint.iterdir() if path.name != 'checkpoint-manifest.json'}
save(self.checkpoint / 'checkpoint-manifest.json', dict(files_sha256=files))
self.value['checkpoint_manifest_sha256'] = m._digest(self.checkpoint / 'checkpoint-manifest.json')
for entry in self.value['files'].values():
entry['sha256'] = m._digest(entry['path'])
for path in self.value['frozen_inputs']:
self.value['frozen_inputs'][path] = m._digest(path)
save(self.authority_path, self.value)
def load_checkpoint(self, directory, pin):
self.asserted_loads += 1
directory = Path(directory)
m._require(m._digest(directory / 'checkpoint-manifest.json') == pin, 'synthetic seal pin mismatch')
manifest = m._read(directory / 'checkpoint-manifest.json')
actual = {path.name: m._digest(path) for path in directory.iterdir() if path.name != 'checkpoint-manifest.json'}
m._require(manifest['files_sha256'] == actual, 'synthetic sealed inventory mismatch')
return dict(fresh=True, global_tick_offset=0, steps=4, summary=m._read(directory / 'summary.json'),
provenance=m._read(directory / 'runtime-provenance.json'), manifest=manifest,
state=m._npz(directory / 'state.npz', ('step', 'neuron_idx', 'counts', 'v', 'g', 'ref')),
overlay=m._read(directory / 'overlay-audit.json'))
def load_overlay(self, file, manifest):
meta = m._read(manifest)
m._require(meta['overlay_sha256'] == m._digest(file), 'synthetic overlay SHA mismatch')
value = m._npz(file, ('edge_id', 'plastic_delta_mv'))
value['base_candidate_mv'] = np.load(meta['base_candidate_path'], allow_pickle=False)
if meta['overlay_name'] == 'learned':
expected = {'schema', 'overlay_name', 'overlay_sha256', 'mask_path', 'mask_sha256',
'base_candidate_path', 'base_candidate_sha256', 'aj_model_dir', 'am_overlay_path', 'am_overlay_sha256',
'parent_snapshot_sha256', 'eligibility_v2_source_sha256'}
m._require(set(meta) == expected, 'learned manifest fields differ from real runtime contract')
return None, value, meta, {}
def run(self, out=None):
self.asserted_loads = 0
parsed = m._authority(self.authority_path, m._digest(self.authority_path))
c = SimpleNamespace(load_checkpoint=self.load_checkpoint)
r = SimpleNamespace(PINS={}, validate_spec=lambda value, _: value,
load_overlay=self.load_overlay, _raw=self.raw)
return m._coordinate(*parsed, self.output_parent / 'result' if out is None else out,
(c, r, update), (5, 6, 2, 2))
class EpochPostprocessing(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.f = TinyFixture(Path(self.tmp.name).resolve())
def tearDown(self):
self.tmp.cleanup()
def test_exclusive_complete_output_and_distinct_parent_hashes(self):
chain = self.f.run()
output = self.f.output_parent / 'result'
self.assertEqual({p.name for p in output.iterdir()}, {'delta-after.npz', 'overlay-manifest.json', 'update-receipt.json', 'chain-receipt.json'})
meta = m._read(output / 'overlay-manifest.json')
self.assertEqual(meta['parent_snapshot_sha256'], self.f.value['files']['overlay_before']['sha256'])
self.assertNotEqual(meta['parent_snapshot_sha256'], self.f.value['checkpoint_manifest_sha256'])
self.assertEqual(chain['before_overlay_manifest_sha256'], self.f.value['files']['overlay_manifest_before']['sha256'])
self.assertEqual(chain['source_neural_checkpoint_manifest_sha256'], self.f.value['checkpoint_manifest_sha256'])
self.assertEqual(self.f.asserted_loads, 2)
self.assertFalse(chain['weights_installed'])
self.assertFalse(chain['registration_authority_independently_proven'])
after = m._npz(output / 'delta-after.npz', ('edge_id', 'plastic_delta_mv'))
self.assertLess(after['plastic_delta_mv'][0], 0)
self.assertEqual(after['plastic_delta_mv'][1].view(np.uint32), 0)
with self.assertRaisesRegex(ValueError, 'Output must be new'):
self.f.run()
def test_every_external_sha_before_processing(self):
with self.assertRaisesRegex(ValueError, 'Authority differs'):
m._authority(self.f.authority_path, '0' * 64)
with (self.f.inputs / 'mapping.npz').open('ab') as stream:
stream.write(b'changed')
with self.assertRaisesRegex(ValueError, 'SHA mismatch'):
self.f.run()
self.assertFalse((self.f.output_parent / 'result').exists())
def test_sealed_event_substitution_and_unbound_case_are_rejected(self):
path = self.f.checkpoint / 'runtime-provenance.json'
provenance = m._read(path)
provenance['input_sha256'][str(self.f.case / 'requests.npz')] = 'f' * 64
save(path, provenance)
self.f.reseal()
with self.assertRaisesRegex(ValueError, 'did not bind authorized requests'):
self.f.run()
def test_changed_runtime_overlay_cannot_match_only_its_own_seal(self):
(self.f.checkpoint / 'overlay-input.npz').write_bytes(b'different')
self.f.reseal()
with self.assertRaisesRegex(ValueError, 'used a different overlay'):
self.f.run()
def test_candidate_endpoint_and_canonical_kc_must_really_match(self):
path = self.f.inputs / 'mapping.npz'
mapping = m._npz(path, ('edge_id', 'source_neuron_idx', 'target_neuron_idx', 'target_compartment_verbatim'))
mapping['target_neuron_idx'][0] = 4
np.savez_compressed(path, **mapping)
self.f.reseal()
with self.assertRaisesRegex(ValueError, 'endpoints differ'):
self.f.run()
mapping['target_neuron_idx'][0] = 2
np.savez_compressed(path, **mapping)
np.save(self.f.inputs / 'kc.npy', np.array([1, 4], np.int32))
self.f.reseal()
with self.assertRaisesRegex(ValueError, 'KC differs'):
self.f.run()
def test_command_steps_and_full_frozen_inventory_required(self):
self.f.value['command']['steps'] = 5
self.f.reseal()
with self.assertRaisesRegex(ValueError, 'Command steps differ'):
self.f.run()
self.f.value['command']['steps'] = 4
del self.f.value['frozen_inputs'][str(self.f.graph / 'signed_raw.npy')]
self.f.reseal()
with self.assertRaisesRegex(ValueError, 'authority is incomplete'):
self.f.run()
def test_protected_directories_and_existing_output_fail_before_write(self):
for path in (self.f.checkpoint / 'new', self.f.inputs / 'new', self.f.graph / 'new',
self.f.authority_dir / 'new', self.f.checkpoint):
with self.subTest(path=path), self.assertRaisesRegex(ValueError, 'Output must be new'):
self.f.run(path)
def test_partial_is_preserved_and_not_retryable(self):
with patch.object(self.f, 'load_overlay', side_effect=[
self.f.load_overlay(self.f.inputs / 'before.npz', self.f.inputs / 'before.json'),
ValueError('synthetic output validation failed')]):
with self.assertRaisesRegex(ValueError, 'output validation'):
self.f.run()
output = self.f.output_parent / 'result'
self.assertTrue((output / 'postprocess-error.json').is_file())
self.assertTrue((output / 'delta-after.npz').is_file())
self.assertFalse((output / 'chain-receipt.json').exists())
with self.assertRaisesRegex(ValueError, 'Output must be new'):
self.f.run()
def test_actual_pinned_dependencies_load_without_gpu(self):
_, paths, pins, _, _ = m._authority(self.f.authority_path, m._digest(self.f.authority_path))
c, r, u = m._production_dependencies(paths, pins)
self.assertEqual(Path(r.__file__).resolve(), paths['runtime_light_source'])
self.assertEqual(c.N, 166700)
self.assertEqual(u.BRIDGE_SHA256, update.BRIDGE_SHA256)
for role in m.SOURCE_PINS:
self.assertEqual(m._digest(paths[role]), m.SOURCE_PINS[role])
def test_production_dimensions_cannot_be_set_by_authority(self):
self.assertEqual(m.DIMENSIONS, (166700, 25582938, 4064, 14232))
self.f.value['small_test_mode'] = True
self.f.reseal()
with self.assertRaisesRegex(ValueError, 'fields/schema'):
m.postprocess_epoch(self.f.authority_path, m._digest(self.f.authority_path), self.f.output_parent / 'result')
def test_public_entry_rejects_tiny_population_even_with_valid_fixture_seal(self):
self.f.asserted_loads = 0
c = SimpleNamespace(load_checkpoint=self.f.load_checkpoint)
r = SimpleNamespace(PINS={}, validate_spec=lambda value, _: value,
load_overlay=self.f.load_overlay, _raw=self.f.raw)
with patch.object(m, '_production_dependencies', return_value=(c, r, update)):
with self.assertRaisesRegex(ValueError, 'Runtime population/mode'):
m.postprocess_epoch(self.f.authority_path, m._digest(self.f.authority_path), self.f.output_parent / 'result')
self.assertFalse((self.f.output_parent / 'result').exists())
def test_authorized_event_file_cannot_be_an_unsealed_copy(self):
other = self.f.inputs / 'state-copy.npz'
other.write_bytes((self.f.checkpoint / 'state.npz').read_bytes())
self.f.value['files']['state'] = dict(path=str(other), sha256=m._digest(other))
self.f.reseal()
with self.assertRaisesRegex(ValueError, 'Events must come from this checkpoint'):
self.f.run()
def test_late_input_change_preserves_partial_but_never_complete_chain(self):
original = self.f.load_checkpoint
calls = 0
def late_change(directory, pin):
nonlocal calls
calls += 1
result = original(directory, pin)
if calls == 2:
with (self.f.graph / 'weights.npy').open('ab') as stream:
stream.write(b'changed after outputs were written')
return result
with patch.object(self.f, 'load_checkpoint', side_effect=late_change):
with self.assertRaisesRegex(ValueError, 'SHA mismatch'):
self.f.run()
output = self.f.output_parent / 'result'
self.assertTrue((output / 'postprocess-error.json').is_file())
self.assertFalse((output / 'chain-receipt.json').exists())
def test_zero_d_and_frozen_return_identity_without_installation(self):
for index, (d, frozen) in enumerate(((0., False), (1., True))):
self.f.value['command'].update(d_command=d, frozen=frozen)
self.f.reseal()
self.f.run(self.f.output_parent / str(index))
value = m._npz(self.f.output_parent / str(index) / 'delta-after.npz', ('edge_id', 'plastic_delta_mv'))
self.assertEqual(value['plastic_delta_mv'].tobytes(), self.f.before.tobytes())
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,258 @@
"""Synthetic CPU/mocked-CUDA contract tests; never run a real input generator."""
import ast
import hashlib
import json
from pathlib import Path
import types
import unittest
from unittest.mock import patch
import numpy as np
import conditioning_inputs as c
def req(rows):
values = np.asarray(rows, np.int32).reshape(-1, 2)
return {'tick': values[:, 0].copy(), 'neuron_idx': values[:, 1].copy()}
class FakeTensor:
def __init__(self, values, device):
self.values = np.array(values, copy=True)
self.device, self.dtype = device, self.values.dtype
def __getitem__(self, key): return FakeTensor(self.values[key], self.device)
def __lt__(self, other): return FakeTensor(self.values < other.values, self.device)
def cpu(self): return self
def numpy(self): return self.values.copy()
class FakeTorch:
"""Observable adapter, not a CUDA or random-number emulation."""
float32 = np.dtype(np.float32)
__version__ = 'SYNTHETIC-CONTRACT-MOCK-NOT-TORCH'
version = types.SimpleNamespace(cuda=None)
def __init__(self, uniforms, truncate=False):
self.uniforms = uniforms
self.calls, self.seeds, self.truncate = [], [], truncate
self.cuda = types.SimpleNamespace(is_available=lambda: True,
get_device_name=lambda device: 'SYNTHETIC-NOT-A-REAL-GPU', get_device_capability=lambda device: (0, 0),
get_device_properties=lambda device: types.SimpleNamespace())
def get_default_dtype(self): return self.float32
def device(self, name): return name
def Generator(self, device):
owner = self
class Generator:
def manual_seed(self, value):
owner.seeds.append(value); self.seed = value & 0xffffffff if owner.truncate else value
def initial_seed(self): return self.seed
self.calls.append(('generator', device)); return Generator()
def as_tensor(self, values, dtype, device):
self.calls.append(('probability', np.array(values, copy=True), dtype, device))
return FakeTensor(values, device)
def rand(self, shape, generator, device):
self.calls.append(('rand', shape, generator.initial_seed(), device))
if shape != self.uniforms.shape: raise AssertionError('Unexpected draw shape')
return FakeTensor(self.uniforms, device)
class InputsTests(unittest.TestCase):
def setUp(self):
self.targets = np.array([19, 3], np.int32)
self.rates = np.array([5000, 10000], np.float32)
def test_frozen_probability_dtype_order_signed_zero_and_no_mutation(self):
targets = np.arange(6, dtype=np.int32)
rates = np.array([0., -0., np.nextafter(np.float32(0), np.float32(1)), .1, 123.456, 10000.], np.float32)
before = rates.tobytes(); rates.setflags(write=False)
actual = c.probabilities_f32(targets, rates)
self.assertEqual(actual.dtype, np.float32)
self.assertEqual(actual.tobytes(), (rates * .1 / 1000.).tobytes())
self.assertEqual(rates.tobytes(), before)
def test_uniform_threshold_is_strict_and_bernoulli_not_poisson_counts(self):
p = np.array([0, .5, 1], np.float32)
u = np.array([[0, .5, np.nextafter(np.float32(1), np.float32(0))], [0, .499, 0]], np.float32)
forced = c.forced_from_uniforms(u, p)
np.testing.assert_array_equal(forced, [[False, False, True], [False, True, True]])
self.assertEqual(forced.dtype, np.bool_)
def test_original_PN_column_order_then_sparse_tick_global_sort_only(self):
forced = np.array([[True, True], [True, False], [False, True]], np.bool_)
before = forced.copy(); self.targets.setflags(write=False)
result = c.requests_from_forced(forced, self.targets, cs_start_tick=0, cs_stop_tick=3)
np.testing.assert_array_equal(result['requests']['tick'], [0, 0, 1, 2])
np.testing.assert_array_equal(result['requests']['neuron_idx'], [3, 19, 19, 3])
np.testing.assert_array_equal(forced, before)
self.assertEqual(result['audit']['duplicate_requests_removed'], 0)
def test_half_open_window_masks_full_matrix_including_empty_window(self):
forced = np.ones((5, 2), np.bool_)
result = c.requests_from_forced(forced, self.targets, cs_start_tick=1, cs_stop_tick=4)
self.assertEqual(result['audit']['full_matrix_boolean_hits'], 10)
self.assertEqual(result['audit']['outside_CS_boolean_hits_masked'], 4)
self.assertEqual(set(result['requests']['tick']), {1, 2, 3})
empty = c.requests_from_forced(forced, self.targets, cs_start_tick=2, cs_stop_tick=2)
self.assertEqual(len(empty['requests']['tick']), 0)
self.assertEqual(empty['audit']['outside_CS_boolean_hits_masked'], 10)
def test_full_63bit_seed_single_full_draw_and_mock_scope_receipt(self):
seed = (1 << 62) + (1 << 32) + 19
fake = FakeTorch(np.array([[.2, .6], [.7, .7], [.1, .2], [.8, .1], [.1, .1]], np.float32))
with patch.object(c, '_load_torch', return_value=fake):
result = c.generate_cs_requests(self.targets, self.rates, seed=seed, steps=5, cs_start_tick=1, cs_stop_tick=4,
device='cuda:0', max_uniform_draws=10)
self.assertEqual(fake.seeds, [seed]); draws = [x for x in fake.calls if x[0] == 'rand']
self.assertEqual(draws, [('rand', (5, 2), seed, 'cuda:0')])
a = result['audit']; self.assertEqual(a['seed_hex'], f'{seed:016x}')
self.assertEqual(a['numpy_version'], np.__version__)
self.assertEqual(a['torch_version'], fake.__version__)
self.assertEqual(a['device_name'], 'SYNTHETIC-NOT-A-REAL-GPU')
self.assertFalse(a['CUDA_sequence_equivalence_tested_here'])
self.assertFalse(a['CUDA_high_seed_bit_sequence_influence_tested_here'])
self.assertFalse(a['full_old_L_input_identity_claimed'])
self.assertEqual(a['uniform_scalar_draws'], 10)
self.assertEqual((a['max_uniform_draws'], a['uniform_matrix_bytes']), (10, 40))
self.assertEqual(a['device_identity'], {'uuid': None, 'pci_bus_id': None, 'query_error': None})
np.testing.assert_array_equal(result['requests']['tick'], [1, 2, 2, 3])
np.testing.assert_array_equal(result['requests']['neuron_idx'], [3, 3, 19, 3])
json.dumps(a, allow_nan=False)
def test_high_seed_bits_preserved_and_bad_consumer_detected_before_draw(self):
for seed in (0, 17, (1 << 32) + 17, (1 << 63) - 1):
fake = FakeTorch(np.zeros((1, 2), np.float32))
with patch.object(c, '_load_torch', return_value=fake):
result = c.generate_cs_requests(self.targets, self.rates, seed=seed, steps=1, cs_start_tick=0, cs_stop_tick=1,
device='cuda:0', max_uniform_draws=2)
self.assertEqual(result['audit']['generator_initial_seed'], seed)
self.assertEqual(fake.seeds, [seed])
fake = FakeTorch(np.zeros((1, 2), np.float32), truncate=True)
with patch.object(c, '_load_torch', return_value=fake), self.assertRaisesRegex(ValueError, 'complete semantic seed'):
c.generate_cs_requests(self.targets, self.rates, seed=(1 << 32) + 17, steps=1, cs_start_tick=0, cs_stop_tick=1,
device='cuda:0', max_uniform_draws=2)
self.assertFalse(any(x[0] == 'rand' for x in fake.calls))
def test_invalid_seed_clock_or_rate_never_imports_torch(self):
for seed in (-1, 2**63, True, np.int64(3), 3.):
with patch.object(c, '_load_torch') as loader, self.assertRaises(ValueError):
c.generate_cs_requests(self.targets, self.rates, seed=seed, steps=5, cs_start_tick=0, cs_stop_tick=5,
device='cuda:0', max_uniform_draws=10)
loader.assert_not_called()
for start, stop in ((-1, 1), (2, 1), (0, 6), (False, 1)):
with patch.object(c, '_load_torch') as loader, self.assertRaises(ValueError):
c.generate_cs_requests(self.targets, self.rates, seed=3, steps=5, cs_start_tick=start, cs_stop_tick=stop,
device='cuda:0', max_uniform_draws=10)
loader.assert_not_called()
def test_invalid_arrays_and_no_clipping_or_duplicate_merging(self):
for rates in (np.array([-1, 2], np.float32), np.array([np.nan, 2], np.float32), np.array([np.inf, 2], np.float32),
np.array([10001, 2], np.float32), self.rates.astype(np.float64), self.rates[:1]):
with self.assertRaises(ValueError): c.probabilities_f32(self.targets, rates)
for targets in (np.array([3, 3], np.int32), np.array([-1, 3], np.int32), np.array([166700, 3], np.int32), self.targets.astype(np.int64)):
with self.assertRaises(ValueError): c.probabilities_f32(targets, self.rates)
with self.assertRaises(ValueError): c.probabilities_f32(self.targets, self.rates, dt_ms=.2)
with self.assertRaises(ValueError): c.requests_from_forced(np.ones((2, 2), np.uint8), self.targets, cs_start_tick=0, cs_stop_tick=2)
def test_no_cpu_fallback_or_changed_default_float_dtype(self):
for variant in ('cpu', 'unavailable', 'float64'):
fake = FakeTorch(np.zeros((1, 2), np.float32)); device = 'cuda:0'
if variant == 'cpu': device = 'cpu'
if variant == 'unavailable': fake.cuda.is_available = lambda: False
if variant == 'float64': fake.get_default_dtype = lambda: np.dtype(np.float64)
with patch.object(c, '_load_torch', return_value=fake), self.assertRaises(ValueError):
c.generate_cs_requests(self.targets, self.rates, seed=3, steps=1, cs_start_tick=0, cs_stop_tick=1,
device=device, max_uniform_draws=2)
self.assertFalse(any(x[0] == 'rand' for x in fake.calls))
def test_explicit_device_and_budget_required_and_budget_checked_before_torch(self):
common = dict(seed=3, steps=5, cs_start_tick=0, cs_stop_tick=1)
for missing in ({}, {'device': 'cuda:0'}, {'max_uniform_draws': 10}):
with patch.object(c, '_load_torch') as loader, self.assertRaises(TypeError):
c.generate_cs_requests(self.targets, self.rates, **common, **missing)
loader.assert_not_called()
for budget in (0, -1, 9, True, 10., np.int64(10)):
with patch.object(c, '_load_torch') as loader, self.assertRaises(ValueError):
c.generate_cs_requests(self.targets, self.rates, **common, device='cuda:0', max_uniform_draws=budget)
loader.assert_not_called()
# Full 5x2 matrix admission, not merely the 1-tick active CS window.
fake = FakeTorch(np.zeros((5, 2), np.float32))
with patch.object(c, '_load_torch', return_value=fake):
result = c.generate_cs_requests(self.targets, self.rates, **common, device='cuda:1', max_uniform_draws=10)
self.assertEqual(result['audit']['execution_device'], 'cuda:1')
self.assertEqual(result['audit']['uniform_scalar_draws'], 10)
def test_optional_device_uuid_recorded_as_reported_or_explicitly_unavailable(self):
fake = FakeTorch(np.zeros((1, 2), np.float32))
fake.cuda.get_device_properties = lambda device: types.SimpleNamespace(uuid='SYNTHETIC-UUID', pci_bus_id='SYNTHETIC-PCI')
args = dict(seed=3, steps=1, cs_start_tick=0, cs_stop_tick=1, device='cuda:7', max_uniform_draws=2)
with patch.object(c, '_load_torch', return_value=fake):
audit = c.generate_cs_requests(self.targets, self.rates, **args)['audit']
self.assertEqual(audit['device_identity'], {'uuid': 'SYNTHETIC-UUID', 'pci_bus_id': 'SYNTHETIC-PCI', 'query_error': None})
self.assertFalse(audit['device_identity_verified_against_external_pin'])
def unavailable(device): raise RuntimeError('Synthetic property unavailable')
fake.cuda.get_device_properties = unavailable
with patch.object(c, '_load_torch', return_value=fake):
audit = c.generate_cs_requests(self.targets, self.rates, **args)['audit']
self.assertIsNone(audit['device_identity']['uuid'])
self.assertEqual(audit['device_identity']['query_error'], 'RuntimeError: Synthetic property unavailable')
def test_returned_audited_arrays_are_owned_readonly_and_inputs_not_frozen(self):
forced = np.ones((2, 2), np.bool_)
generated = c.requests_from_forced(forced, self.targets, cs_start_tick=0, cs_stop_tick=2)
merged = c.merge_requests(req([[0, 9]]), req([[1, 8]]), steps=2)
for value in (*generated['requests'].values(), *merged['requests'].values(), merged['source_mask']):
self.assertTrue(value.flags.owndata)
self.assertFalse(value.flags.writeable)
with self.assertRaises(ValueError): value[0] = 0
self.assertTrue(forced.flags.writeable)
self.assertTrue(self.targets.flags.writeable)
def test_empty_targets_and_p0_p1_are_explicit_valid_limits(self):
result = c.requests_from_forced(np.zeros((3, 0), np.bool_), np.empty(0, np.int32), cs_start_tick=0, cs_stop_tick=3)
self.assertEqual(result['requests']['tick'].dtype, np.int32); self.assertEqual(len(result['requests']['tick']), 0)
np.testing.assert_array_equal(c.probabilities_f32(self.targets, np.array([0, 10000], np.float32)), [0, 1])
def test_merging_disjoint_sources_is_sorted_without_mutating_inputs(self):
cs, us = req([[0, 9], [3, 4]]), req([[0, 2], [2, 5]])
for value in [*cs.values(), *us.values()]: value.setflags(write=False)
result = c.merge_requests(cs, us, steps=4)
np.testing.assert_array_equal(result['requests']['tick'], [0, 0, 2, 3])
np.testing.assert_array_equal(result['requests']['neuron_idx'], [2, 9, 5, 4])
np.testing.assert_array_equal(result['source_mask'], [2, 1, 2, 1])
self.assertEqual(result['audit']['explicit_boolean_overlap_collapses'], 0)
def test_collision_rejected_by_default_or_explicitly_audited_boolean_union(self):
cs, us = req([[0, 9], [3, 4]]), req([[0, 9], [2, 5], [3, 4]])
with self.assertRaises(c.RequestConflictError) as error: c.merge_requests(cs, us, steps=4)
self.assertEqual(error.exception.audit['cross_source_conflict_count'], 2)
result = c.merge_requests(cs, us, steps=4, conflict_policy='boolean_union')
np.testing.assert_array_equal(result['source_mask'], [3, 2, 3])
a = result['audit']; self.assertEqual((a['CS_attempts'], a['US_attempts'], a['output_requests']), (2, 3, 3))
self.assertEqual(a['explicit_boolean_overlap_collapses'], 2)
self.assertEqual([x['combined_boolean_requests'] for x in a['conflicts']], [1, 1])
json.dumps(a, allow_nan=False)
def test_within_source_duplicate_unsorted_and_bad_range_never_deduped(self):
for rows in ([[0, 3], [0, 3]], [[1, 3], [0, 3]], [[4, 3]], [[0, -1]]):
with self.assertRaises(ValueError): c.merge_requests(req(rows), req([]), steps=4, conflict_policy='boolean_union')
with self.assertRaises(ValueError): c.merge_requests(req([]), req([]), steps=4, conflict_policy='clip')
def test_frozen_AK_pure_conversion_AST_matches_for_full_window(self):
local = Path(__file__).resolve().parent.parent / 'apl-timing-validation-v1/encoder_inputs.py'
raw = local.read_bytes(); self.assertEqual(hashlib.sha256(raw).hexdigest(), c.AK_ENCODER_SHA)
tree = ast.parse(raw); node = next(n for n in tree.body if isinstance(n, ast.FunctionDef) and n.name == 'requests_from_forced')
namespace = {'np': np}; exec(compile(ast.Module(body=[node], type_ignores=[]), str(local), 'exec'), namespace)
forced = np.array([[True, True], [False, True], [True, False]], np.bool_)
expected = namespace['requests_from_forced'](forced, self.targets)
actual = c.requests_from_forced(forced, self.targets, cs_start_tick=0, cs_stop_tick=3)['requests']
np.testing.assert_array_equal(expected, np.column_stack([actual['tick'], actual['neuron_idx']]))
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,308 @@
"""Synthetic CPU array/receipt mutation tests; no files or neural execution."""
import copy
import hashlib
import json
import unittest
from unittest import mock
import numpy as np
import conditioning_preflight_compare as P
from test_engineering_compare import synthetic_parts
def rehash(receipt):
receipt["payload_sha256"] = P._sha({k: v for k, v in receipt.items() if k != "payload_sha256"})
def pair(*, active=True):
reference = synthetic_parts([64], active=active)[0]
light = synthetic_parts([64], active=active, traces=False, queues=False)[0]
for record, mode in ((reference, "reference"), (light, "light")):
summary = record["summary"]
summary.update(edges=6, explicit_steps_executed=64, within_trial_weight_updates=False,
hidden_early_stop=False, entire_neural_trial_executed=True, record_history=False,
state_units="SI volts", first_failure_tick=-1, counts_match_sparse_events=True,
total_spikes=len(record["state"]["step"]), requested=len(record["requests"]),
accepted_requests=int(record["requests"][:, 3].sum()), rejected_requests=int(record["requests"][:, 4].sum()),
full_APL_history_recorded=mode == "reference", full_tick_state_hashes_recorded=mode == "reference",
full_queue_hashes_recorded=mode == "reference", full_sequence_bitwise_audit_performed=mode == "reference",
arriving_edges=sum(row["count"] for row in reference["queues"]),
accepted_arriving_edges=sum(row["accepted_count"] for row in reference["queues"]),
input_sha256={"base_weights.npy": "a" * 64, "coefficients.json": "b" * 64,
"requests.npz": "c" * 64, "spec.json": "d" * 64})
record["terminal_origin"] = "recorded_terminal_queues"
return reference, light, reference["requests"][:, :2].astype(np.int32)
def delta_fixture():
before = np.zeros(P.CANDIDATE_COUNT, np.float32)
after = before.copy()
after[1] = np.float32(-.001)
receipt = dict(schema="phase4an-conditioning-update-draft-v1",
status="DRAFT-COMPUTED-EPOCH-UPDATE-NOT-INSTALLED",
source_pins={"eligibility_bridge_sha256": P.ELIGIBILITY_BRIDGE_SHA256, "phase4j_sha256": P.PHASE4J_SHA256},
input_arrays={"delta_before": P._evidence(before)}, outputs={"delta_after": P._evidence(after)},
command={"D_scheduled": 1.0, "frozen": False},
semantics={"application": "epoch_boundary_after_all_recorded_ticks; affects subsequent epoch only",
"delta_dtype": "float32", "delta_units": "mV", "D_definition": "artificial scheduled scalar",
"actual_DAN_delivery_observed": False, "actual_DAN_delivery_used_to_gate": False},
counts={"candidates": P.CANDIDATE_COUNT, "actual_bitwise_changed_edges": 1, "actual_value_changed_edges": 1},
weights_installed=False, base_modified=False, brain_dynamics_executed=False,
scientific_learning_pass_claimed=False, candidate_generated=False)
rehash(receipt)
manifest = dict(schema="phase4an-overlay-v1", overlay_name="learned", overlay_sha256="1" * 64,
mask_path="/synthetic/mask.npy", mask_sha256=P.MASK_SHA256,
base_candidate_path="/synthetic/base.npy", base_candidate_sha256="2" * 64,
aj_model_dir="/synthetic/apl-timing-lab-v1", am_overlay_path="/synthetic/overlay.py",
am_overlay_sha256=P.AM_OVERLAY_SHA256, parent_snapshot_sha256="3" * 64,
eligibility_v2_source_sha256=P.PHASE4J_SHA256)
return before, after, after.copy(), receipt, manifest
class EpochPairTests(unittest.TestCase):
def setUp(self):
# Pure three-neuron fixtures; production constants remain full MaleCNS.
self.patches = [mock.patch.object(P, "N", 3), mock.patch.object(P, "E", 6), mock.patch.object(P, "N_KC", 2)]
for patch in self.patches:
patch.start()
self.addCleanup(patch.stop)
def compare(self, reference, light, requests):
return P.compare_epoch_pair(reference, light, expected_requests=requests,
expected_steps=64, expected_effective_weight_sha256="e" * 64)
def test_exact_actual_arrays_with_explicit_light_limits(self):
reference, light, requests = pair()
report = self.compare(reference, light, requests)
self.assertTrue(report["pass_all"], report["failed_checks"])
self.assertTrue(report["numerical_exact"])
self.assertTrue(report["nonvacuous_activity"])
self.assertFalse(report["light_intermediate_states_observed"])
self.assertFalse(report["light_per_tick_V_g_ref_identity_claimed"])
self.assertEqual(report["recorded_pair_availability"],
{"APL_traces": False, "state_hashes": False, "queue_hashes": False})
self.assertFalse(report["external_file_seals_process_and_input_authority_verified_here"])
self.assertFalse(report["learning_pass_claimed"])
self.assertFalse(report["candidate_generated"])
json.dumps(report, allow_nan=False)
def test_equal_quiet_records_are_not_new_activity_or_learning_pass(self):
reference, light, requests = pair(active=False)
report = self.compare(reference, light, requests)
self.assertTrue(report["numerical_exact"])
self.assertFalse(report["nonvacuous_activity"])
self.assertTrue(report["activity_is_descriptive_not_a_new_gate"])
self.assertFalse(report["learning_pass_claimed"])
def test_signed_zero_final_voltage_difference_fails(self):
reference, light, requests = pair()
light["state"]["g"][0] = np.float64(0.)
report = self.compare(reference, light, requests)
self.assertFalse(report["pass_all"])
self.assertIn("pair.state.g", report["failed_checks"])
def test_equal_but_wrong_original_requests_fail(self):
reference, light, requests = pair()
wrong = requests.copy()
wrong[0, 1] = 1
report = self.compare(reference, light, wrong)
self.assertFalse(report["pass_all"])
self.assertIn("reference.input_request_array", report["failed_checks"])
self.assertIn("light.input_request_array", report["failed_checks"])
def test_reference_missing_diagnostics_cannot_be_excused_by_light(self):
for key in ("traces", "state_hashes", "queues"):
with self.subTest(key=key):
reference, light, requests = pair()
reference[key] = {} if key == "traces" else None
report = self.compare(reference, light, requests)
self.assertFalse(report["pass_all"])
self.assertIn("reference.full_diagnostics", report["failed_checks"])
def test_partial_or_corrupt_array_schema_fails_closed(self):
reference, light, requests = pair()
del reference["traces"]["drive"]
with self.assertRaises(ValueError):
self.compare(reference, light, requests)
reference, light, requests = pair()
light["state"]["v"][0] = np.nan
with self.assertRaises(ValueError):
self.compare(reference, light, requests)
def test_reported_modes_population_delay_and_scope_fail_closed(self):
mutations = (("nodes", 2), ("edges", 7), ("apl_delay_steps", 17),
("diagnostic_mode", "reference"), ("first_failure_tick", 0),
("within_trial_weight_updates", True), ("explicit_steps_executed", 63),
("all_finite", 1), ("parent_checkpoint_manifest_sha256", "f" * 64))
for key, value in mutations:
with self.subTest(key=key):
reference, light, requests = pair()
light["summary"][key] = value
self.assertFalse(self.compare(reference, light, requests)["pass_all"])
def test_effective_weight_and_base_coefficient_hash_binding(self):
for mutation in ("effective", "base_weights.npy", "coefficients.json"):
with self.subTest(mutation=mutation):
reference, light, requests = pair()
if mutation == "effective":
for record in (reference, light):
record["summary"]["effective_weights_raw_sha256"] = "f" * 64
else:
light["summary"]["input_sha256"][mutation] = "f" * 64
self.assertFalse(self.compare(reference, light, requests)["pass_all"])
def test_performance_is_not_compared_and_path_only_spec_identity_is_external(self):
reference, light, requests = pair()
reference["summary"].update(wall_seconds=1., peak_allocated_vram_bytes=1)
light["summary"].update(wall_seconds=99., peak_allocated_vram_bytes=999)
light["summary"]["input_sha256"]["spec.json"] = "f" * 64
report = self.compare(reference, light, requests)
self.assertTrue(report["pass_all"])
self.assertFalse(report["external_file_seals_process_and_input_authority_verified_here"])
def test_declared_light_full_diagnostics_or_derived_terminal_is_rejected(self):
for mutation in ("recorded", "origin"):
reference, light, requests = pair()
if mutation == "recorded":
light["summary"]["full_tick_state_hashes_recorded"] = True
else:
light["terminal_origin"] = "AM_spike_ring_derived_from_fresh_events_APL_bits_from_timing"
self.assertFalse(self.compare(reference, light, requests)["pass_all"])
def test_actual_summary_accounting_cannot_be_self_reported_wrong(self):
for key in ("requested", "accepted_requests", "rejected_requests", "total_spikes", "arriving_edges", "accepted_arriving_edges"):
with self.subTest(key=key):
reference, light, requests = pair()
light["summary"][key] += 1
self.assertFalse(self.compare(reference, light, requests)["pass_all"])
def test_invalid_planned_request_seed_arrays_are_not_silently_coerced(self):
reference, light, requests = pair()
for bad in (requests.astype(np.int64), requests[::-1], np.concatenate((requests[:1], requests))):
with self.assertRaises(ValueError):
self.compare(reference, light, bad)
with self.assertRaises(ValueError):
P.compare_epoch_pair(reference, light, expected_requests=requests, expected_steps=True,
expected_effective_weight_sha256="e" * 64)
class DeltaBindingTests(unittest.TestCase):
def audit(self, values):
before, after, installed, receipt, manifest = values
return P.audit_delta_binding(before, after, installed, update_receipt=receipt,
parent_snapshot_sha256="3" * 64, child_manifest=manifest)
def test_exact_array_binding_preserves_explicit_external_authority_limits(self):
values = delta_fixture()
for value in values[:3]:
value.setflags(write=False)
original = [value.tobytes() for value in values[:3]]
result = self.audit(values)
self.assertTrue(result["pass_all"], result["failed_checks"])
self.assertEqual(result["actual_changed_edges"], 1)
for name in ("eligibility_v2_independently_recomputed_here", "full_mask_identity_verified_here",
"file_bytes_and_parent_snapshot_authority_verified_here", "installation_in_runtime_verified_here",
"actual_DAN_delivery_verified_here", "learning_pass_claimed"):
self.assertFalse(result[name])
self.assertEqual([value.tobytes() for value in values[:3]], original)
json.dumps(result, allow_nan=False)
def test_parent_overlay_file_sha_is_not_raw_delta_or_checkpoint_sha(self):
values = delta_fixture()
self.assertNotEqual(values[4]["overlay_sha256"], P._evidence(values[1])["raw_c_order_sha256"])
self.assertTrue(self.audit(values)["pass_all"])
for false_parent in (P._evidence(values[0])["raw_c_order_sha256"], "4" * 64):
values[4]["parent_snapshot_sha256"] = false_parent
result = self.audit(values)
self.assertFalse(result["pass_all"])
self.assertIn("manifest.parent_overlay_file_sha", result["failed_checks"])
def test_installed_delta_signed_zero_and_row_mutation_rejected(self):
for mutation in ("zero", "row"):
values = delta_fixture()
if mutation == "zero":
values[2][0] = np.float32(-0.)
else:
values[2][1], values[2][2] = values[2][2], values[2][1]
result = self.audit(values)
self.assertFalse(result["pass_all"])
self.assertIn("computed_installed_delta_bitwise", result["failed_checks"])
def test_rehashed_receipt_must_still_bind_supplied_arrays(self):
for location, key in (("input_arrays", "delta_before"), ("outputs", "delta_after")):
values = delta_fixture()
values[3][location][key]["raw_c_order_sha256"] = "f" * 64
rehash(values[3])
result = self.audit(values)
self.assertFalse(result["pass_all"])
self.assertIn("receipt." + key, result["failed_checks"])
def test_rehashed_false_count_and_boolean_integer_ambiguity_rejected(self):
for field, value in (("actual_bitwise_changed_edges", 2), ("actual_value_changed_edges", True), ("candidates", float(P.CANDIDATE_COUNT))):
values = delta_fixture()
values[3]["counts"][field] = value
rehash(values[3])
self.assertFalse(self.audit(values)["pass_all"])
def test_stale_receipt_or_changed_frozen_rule_manifest_rejected(self):
for mutation in ("hash", "v2", "mask", "AMhelper", "extra", "wrong_name"):
values = delta_fixture()
if mutation == "hash":
values[3]["payload_sha256"] = "0" * 64
elif mutation == "extra":
values[4]["unregistered_override"] = True
elif mutation == "wrong_name":
values[4]["overlay_name"] = "static25"
else:
key = {"v2": "eligibility_v2_source_sha256", "mask": "mask_sha256", "AMhelper": "am_overlay_sha256"}[mutation]
values[4][key] = "0" * 64
self.assertFalse(self.audit(values)["pass_all"])
def test_zero_D_and_frozen_require_exact_no_update_including_signed_zero(self):
for mode in ("D0", "frozen"):
values = delta_fixture()
values[3]["command"]["D_scheduled" if mode == "D0" else "frozen"] = 0.0 if mode == "D0" else True
rehash(values[3])
result = self.audit(values)
self.assertFalse(result["pass_all"])
self.assertIn("zero_D_or_frozen_identity", result["failed_checks"])
values[1][:] = values[0]
values[2][:] = values[0]
values[3]["outputs"]["delta_after"] = P._evidence(values[1])
values[3]["counts"].update(actual_value_changed_edges=0, actual_bitwise_changed_edges=0)
rehash(values[3])
self.assertTrue(self.audit(values)["pass_all"])
def test_actual_DAN_or_immediate_US_update_claim_does_not_match_interface(self):
for field, value in (("actual_DAN_delivery_used_to_gate", True), ("actual_DAN_delivery_observed", True),
("application", "at US tick"), ("delta_units", "V")):
values = delta_fixture()
values[3]["semantics"][field] = value
rehash(values[3])
self.assertFalse(self.audit(values)["pass_all"])
def test_wrong_dtype_shape_nonfinite_or_positive_delta_rejected(self):
values = delta_fixture()
with self.assertRaises(ValueError):
P.audit_delta_binding(values[0].astype(np.float64), values[1], values[2], update_receipt=values[3],
parent_snapshot_sha256="3" * 64, child_manifest=values[4])
values = list(delta_fixture())
values[0] = values[0][:-1]
with self.assertRaises(ValueError):
self.audit(values)
values = delta_fixture()
values[0][0] = np.nan
with self.assertRaises(ValueError):
self.audit(values)
values = delta_fixture()
values[1][1] = values[2][1] = np.float32(.001)
values[3]["outputs"]["delta_after"] = P._evidence(values[1])
rehash(values[3])
result = self.audit(values)
self.assertFalse(result["pass_all"])
self.assertIn("delta.nonpositive", result["failed_checks"])
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,229 @@
"""Pure/synthetic preflight tests: no actual rates, CUDA draws, or brain run."""
import copy
import hashlib
import io
import json
from pathlib import Path
import tempfile
import types
import unittest
from unittest.mock import patch
import numpy as np
import pandas as pd
import conditioning_preflight_inputs as p
def synthetic_tables():
names = ['synthetic-' + str(i) for i in range(12)]
names[6], names[7] = '4-ethylphenol', '3-hexanone'
ids = ['synthetic-id-' + str(i) for i in range(12)]
panel = pd.DataFrame(dict(odor_id=ids, odor_name=names,
split=['calibration'] * 6 + ['heldout'] * 6, mapped_response_sum=np.arange(1, 13, dtype=float),
formal_label=[''] * 6 + ['ODOR_A', 'ODOR_B'] + [''] * 4))
selection = dict(selected_before_KC_simulation=True, glomeruli_order=['G2', 'G1'],
ODOR_A=dict(odor_id=ids[6], odor_name=names[6]), ODOR_B=dict(odor_id=ids[7], odor_name=names[7]))
vectors = {oid: np.array([.5, .25], np.float32) for oid in ids}
mapping = pd.DataFrame(dict(glomerulus=['G2', 'G2', 'G2', 'G1', 'G1'],
mapping_confidence=['exact', 'alias-normalized', 'exact', 'exact', 'unmatched'],
neuron_idx=[19., 3., 19., 8., 100.]))
config = dict(odor_encoder=dict(max_total_rate_per_glomerulus_hz=400., modes=['physiological', 'matched_energy']))
return selection, panel, vectors, mapping, config
def sparse(rows):
values = np.asarray(rows, np.int32).reshape(-1, 2)
return {'tick': values[:, 0].copy(), 'neuron_idx': values[:, 1].copy()}
class PreflightTests(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.plan = p.build_preflight_plan()
cls.epoch = cls.plan['selected_epochs'][0]
source_parent = Path(__file__).resolve().parent.parent
candidates = (source_parent / 'phase4l.py', source_parent / 'models/male-cns-lif/phase4l.py')
path = next((candidate for candidate in candidates if candidate.is_file()), None)
if path is None:
raise FileNotFoundError('Pinned L source missing at both known server/local paths: ' + repr(candidates))
cls.source_raw = path.read_bytes()
if hashlib.sha256(cls.source_raw).hexdigest() != p.SOURCE_PINS[p.SOURCE_REL]:
raise ValueError('Selected server/local L source differs from the frozen SHA')
cls.drive, cls.ast_sha = p._extract_drive(cls.source_raw, str(path))
cls.tables, _ = p._rate_tables(*synthetic_tables(), cls.drive)
def test_fixed_full_plan_master_first_two_real_epochs_and_12_calls(self):
full = self.plan['full_trial_plan']
self.assertEqual(full['plan']['master'], 20260930)
self.assertEqual(full['payload_sha256'], 'a8e8620875e8213f3cc3b4f153fc5c06e1351bf1d91c7edaf4f18c1e2cdef62e')
self.assertEqual([e['epoch_index'] for e in self.plan['selected_epochs']], [0, 1])
self.assertEqual([e['cs']['cue'] for e in self.plan['selected_epochs']], ['A', 'A'])
self.assertEqual([e['cs']['seed'] for e in self.plan['selected_epochs']], [5717173809506784725, 4342084263790487177])
self.assertTrue(all(e['us']['requested'] for e in self.plan['selected_epochs']))
self.assertEqual(self.plan['total_calls'], 12)
self.assertEqual(sum(r['steps'] for r in self.plan['calls']), 14400)
self.assertEqual(len({r['row_id'] for r in self.plan['calls']}), 12)
self.assertFalse(self.plan['complete_conditioning_authorized'])
def test_plan_rows_share_inputs_but_keep_D_and_freeze_distinct(self):
for row in self.plan['calls']:
self.assertEqual(row['epoch_id'], self.plan['selected_epochs'][row['input_epoch_index']]['epoch_id'])
self.assertEqual(row['d_command'], 0. if row['arm'] == 'D0' else 1.)
self.assertEqual(row['frozen'], row['arm'] == 'frozen')
self.assertEqual(row['previous_delta_row_id'], None if row['input_epoch_index'] == 0 else
f"{row['arm']}--{row['diagnostic_mode']}--epoch-000")
self.assertEqual(self.plan['input_generation_calls'], 2)
def test_whole_plan_validator_rejects_wrong_master_and_row_mutation(self):
# Reuse an already independently rebuilt expected fixture here; the
# production validator calls its full builder, not a count-only gate.
with patch.object(p, 'build_preflight_plan', return_value=self.plan) as rebuild:
self.assertTrue(p.validate_preflight_plan(copy.deepcopy(self.plan)))
bad = copy.deepcopy(self.plan); bad['master'] = 461
with self.assertRaises(ValueError): p.validate_preflight_plan(bad)
bad = copy.deepcopy(self.plan); bad['calls'][0]['d_command'] = 0.
with self.assertRaises(ValueError): p.validate_preflight_plan(bad)
self.assertEqual(rebuild.call_count, 3)
def test_actual_frozen_function_AST_preserves_mapping_order_rates_and_dtype(self):
table = self.tables['A']
np.testing.assert_array_equal(table['targets'], [19, 3, 8])
self.assertEqual(table['targets'].dtype, np.int32)
np.testing.assert_array_equal(table['rates_hz'], np.array([100, 100, 100], np.float32))
self.assertEqual(table['source_panel_row'], 6)
self.assertEqual(table['source_panel_split'], 'heldout') # original static label only
self.assertFalse(table['targets'].flags.writeable)
self.assertFalse(table['rates_hz'].flags.writeable)
self.assertEqual(hashlib.sha256(self.source_raw).hexdigest(), p.SOURCE_PINS[p.SOURCE_REL])
self.assertEqual(len(self.ast_sha), 64)
def test_static_mapping_fails_closed_on_selection_order_schema_or_duplicate_PN(self):
for kind in ('name', 'split', 'vector', 'duplicate'):
args = list(synthetic_tables())
if kind == 'name': args[0]['ODOR_A']['odor_name'] = 'different'
if kind == 'split': args[1].loc[0, 'split'] = 'heldout'
if kind == 'vector': args[2][args[0]['ODOR_A']['odor_id']] = np.array([np.nan, .25], np.float32)
if kind == 'duplicate': args[3].loc[3, 'neuron_idx'] = 19.
with self.assertRaises(ValueError): p._rate_tables(*args, type(self).drive)
def test_loader_checks_all_static_bytes_then_uses_original_AST_no_old_import(self):
selection, panel, vectors, mapping, config = synthetic_tables()
buffer = io.BytesIO(); np.savez(buffer, glomeruli=np.array(selection['glomeruli_order']), **vectors)
payload = {
p.SOURCE_REL: self.source_raw, p.CONFIG_REL: json.dumps(config).encode(),
p.STATIC + 'odor-encoder/frozen-odor-panel.parquet': b'SYNTHETIC-PANEL',
p.STATIC + 'odor-encoder/mapped-glomerular-vectors.npz': buffer.getvalue(),
p.STATIC + 'odor-encoder/selection.json': json.dumps(selection).encode(),
p.STATIC + 'receptor-mapping/olfactory-mapping.parquet': b'SYNTHETIC-MAPPING',
}
pins = {k: hashlib.sha256(v).hexdigest() for k, v in payload.items()}
with tempfile.TemporaryDirectory() as td:
for name, raw in payload.items():
target = Path(td) / name; target.parent.mkdir(parents=True, exist_ok=True); target.write_bytes(raw)
# Synthetic config is JSON (a YAML subset); local runtime lacks
# PyYAML. This mocks parsing only, not any scientific input/RNG.
yaml_mock = types.SimpleNamespace(safe_load=json.loads)
with patch.object(p, 'SOURCE_PINS', pins), patch.object(pd, 'read_parquet', side_effect=[panel, mapping]), \
patch.dict('sys.modules', {'yaml': yaml_mock}):
result = p.load_rate_tables(td)
self.assertEqual(result['audit']['static_sources'], pins)
self.assertFalse(result['audit']['historical_dynamic_results_read'])
self.assertEqual(result['audit']['random_draws'], 0)
self.assertEqual(result['tables']['B']['odor_name'], '3-hexanone')
(Path(td) / p.CONFIG_REL).write_bytes(b'CHANGED')
with patch.object(p, 'SOURCE_PINS', pins), patch.object(pd, 'read_parquet') as parse, \
patch.dict('sys.modules', {'yaml': yaml_mock}), self.assertRaises(ValueError):
p.load_rate_tables(td)
parse.assert_not_called()
def test_generate_delegates_full_shape_full_seed_and_shared_US_without_real_draw(self):
fake = dict(requests=sparse([[2, 19], [999, 8]]), audit={'synthetic_sampler': True})
dan = np.array([80, 90], np.int32)
with patch.object(p.inputs, 'generate_cs_requests', return_value=fake) as sampler:
result = p.generate_epoch_inputs(self.epoch, self.tables, dan, device='cuda:1', max_uniform_draws=3600)
args, kwargs = sampler.call_args
self.assertIs(args[0], self.tables['A']['targets'])
self.assertEqual(kwargs, dict(seed=5717173809506784725, steps=1200, cs_start_tick=0, cs_stop_tick=1000,
device='cuda:1', max_uniform_draws=3600))
np.testing.assert_array_equal(result['requests']['tick'], [2, 900, 900, 999])
np.testing.assert_array_equal(result['requests']['neuron_idx'], [19, 80, 90, 8])
self.assertTrue(result['audit']['US_shared_across_arms'])
self.assertFalse(result['audit']['brain_reset_called'])
def test_explicit_conflict_union_and_default_error_preserve_US_attempts(self):
fake = dict(requests=sparse([[900, 19]]), audit={'synthetic_sampler': True})
with patch.object(p.inputs, 'generate_cs_requests', return_value=fake):
with self.assertRaises(p.inputs.RequestConflictError):
p.generate_epoch_inputs(self.epoch, self.tables, np.array([19], np.int32), device='cuda:0', max_uniform_draws=3600)
value = p.generate_epoch_inputs(self.epoch, self.tables, np.array([19], np.int32),
device='cuda:0', max_uniform_draws=3600, conflict_policy='boolean_union')
self.assertEqual(value['audit']['composition']['cross_source_conflict_count'], 1)
np.testing.assert_array_equal(value['source_mask'], [3])
def test_no_US_epoch_still_draws_CS_but_no_DAN_pulse(self):
epoch = copy.deepcopy(self.epoch)
epoch['us'].update(requested=False, tick=None, request_rule='none'); epoch['D_command'] = 0.
with patch.object(p.inputs, 'generate_cs_requests', return_value=dict(requests=sparse([]), audit={})):
value = p.generate_epoch_inputs(epoch, self.tables, np.array([80], np.int32), device='cuda:0', max_uniform_draws=3600)
self.assertEqual(len(value['us_requests']['tick']), 0)
def test_invalid_targets_and_mutated_rates_rejected_before_sampling(self):
with patch.object(p.inputs, 'generate_cs_requests') as sampler:
for dan in (np.array([2, 2], np.int32), np.array([9, 2], np.int32), np.array([], np.int32), np.array([2], np.int64)):
with self.assertRaises(ValueError):
p.generate_epoch_inputs(self.epoch, self.tables, dan, device='cuda:0', max_uniform_draws=3600)
tables = copy.deepcopy(self.tables); tables['A']['rates_hz'][0] = 3.
with self.assertRaises(ValueError):
p.generate_epoch_inputs(self.epoch, tables, np.array([80], np.int32), device='cuda:0', max_uniform_draws=3600)
sampler.assert_not_called()
def original_spec(self):
return dict(name='old-case', kind='fullgraph', steps=1000, dt_ms=.1, duration_ms=100.,
model={'immutable': 'synthetic model'}, graph_dir='/old/graph', apl_bundle_dir='/old/apl',
coefficients_path='/old/coeff.json', science_python='/old/venv/bin/python',
apl_delay_steps=18, precision='float64', apl_precision='float32', record_history=False,
training=False, validation_cohort='old_cohort', odor_index=0, expected_external_events=123.,
odor_id='old-id', odor_name='old-name', source_panel_row=0, source_panel_split='calibration',
rate_sum_hz=999., target_count=100, seed=22, budget_root='/old/experiment', output_budget_bytes=100)
def test_derive_spec_changes_only_explicit_input_scope_and_clock_budget_fields(self):
old = self.original_spec(); before = copy.deepcopy(old)
value, audit = p.derive_case_spec(old, self.plan['calls'][0], self.epoch, self.tables['A'],
brain_reset_seed=20260930, budget_root='/new/preflight', output_budget_bytes=10000)
self.assertEqual(old, before)
self.assertEqual((value['steps'], value['duration_ms'], value['max_steps']), (1200, 120., 1200))
self.assertEqual(value['seed'], 20260930)
self.assertEqual(value['cs_input_seed'], 5717173809506784725)
for key in ('model', 'graph_dir', 'apl_bundle_dir', 'coefficients_path', 'science_python'):
self.assertEqual(value[key], old[key])
self.assertFalse(value['within_trial_weight_updates'])
self.assertEqual(value['odor_name'], '4-ethylphenol')
self.assertNotIn('validation_cohort', value); self.assertNotIn('training', value)
self.assertEqual(set(audit['removed_stale_input_metadata']), {'validation_cohort', 'training', 'odor_index', 'expected_external_events'})
self.assertFalse(audit['full_conditioning_authorized'])
json.dumps(audit, allow_nan=False)
def test_seed_or_neural_continuation_or_budget_alias_invalid(self):
args = dict(brain_reset_seed=3, budget_root='/new/preflight', output_budget_bytes=10000)
for seed in (-1, 2**31, self.epoch['cs']['seed'], True, np.int32(3)):
with self.assertRaises(ValueError):
p.derive_case_spec(self.original_spec(), self.plan['calls'][0], self.epoch, self.tables['A'], **{**args, 'brain_reset_seed': seed})
for root in ('/old/experiment/new', '/old', '/new/../old', 'relative'):
with self.assertRaises(ValueError):
p.derive_case_spec(self.original_spec(), self.plan['calls'][0], self.epoch, self.tables['A'], **{**args, 'budget_root': root})
old = self.original_spec(); old['parent_checkpoint_manifest_sha256'] = 'a' * 64
with self.assertRaises(ValueError): p.derive_case_spec(old, self.plan['calls'][0], self.epoch, self.tables['A'], **args)
def test_row_command_mismatch_and_rate_label_mismatch_fail_closed(self):
row = copy.deepcopy(self.plan['calls'][0]); row['d_command'] = 0.
with self.assertRaises(ValueError):
p.derive_case_spec(self.original_spec(), row, self.epoch, self.tables['A'],
brain_reset_seed=3, budget_root='/new/preflight', output_budget_bytes=10000)
with self.assertRaises(ValueError):
p.derive_case_spec(self.original_spec(), self.plan['calls'][0], self.epoch, self.tables['B'],
brain_reset_seed=3, budget_root='/new/preflight', output_budget_bytes=10000)
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,109 @@
"""CPU-only coordinator contract tests; no real brain/input sampling."""
import ast
from pathlib import Path
import tempfile
import unittest
from unittest.mock import patch
import numpy as np
import conditioning_preflight_prepare as P
import conditioning_preflight_workflow as W
class CoordinatorTests(unittest.TestCase):
def test_npz_exclusive_roundtrip(self):
with tempfile.TemporaryDirectory() as d:
path = Path(d) / 'x.npz'
a = np.asarray([0, 3], np.int32)
P.save_npz(path, tick=a)
self.assertEqual(W.npz(path)['tick'].tobytes(), a.tobytes())
with self.assertRaises(FileExistsError): P.save_npz(path, tick=a)
def test_exact_pins_and_mutation(self):
with tempfile.TemporaryDirectory() as d:
p = Path(d) / 'input'
p.write_bytes(b'original')
pins = {str(p): P.digest(p)}
P.verify_pins(pins)
p.write_bytes(b'changed')
with self.assertRaisesRegex(ValueError, 'Pinned file'): P.verify_pins(pins)
def test_rows_exact_unique(self):
row = dict(row_id='D1--reference--epoch-000')
self.assertEqual(W.row_by({'plan': {'calls': [row]}}, row['row_id']), row)
for rows in ([], [row, row]):
with self.assertRaises(ValueError): W.row_by({'plan': {'calls': rows}}, row['row_id'])
def test_locations_reject_path_traversal(self):
for value in ('../x', '/x', 'a/b', 'a\\b', '', None):
with self.assertRaises(ValueError): W.locations(value)
def test_plan_descriptor_avoids_huge_payload_duplication(self):
value = dict(full_trial_plan={'large': [1]}, payload_sha256='x', calls=[{'row_id': 'a'}])
self.assertEqual(P.plan_descriptor(value), {'calls': [{'row_id': 'a'}]})
self.assertIn('full_trial_plan', value)
def test_original_AST_rejects_unanchored_bytes(self):
with self.assertRaisesRegex(ValueError, 'L source changed'):
W.original_sampler_nodes(b'def run_rates(): pass')
def test_original_AST_if_local_frozen_source_available(self):
# Both repository checkout and local source mirror are supported; this
# test never reads historical dynamics or executes the extracted code.
candidates = [Path(__file__).resolve().parents[1] / 'phase4l.py',
Path(__file__).resolve().parents[1] / 'models/male-cns-lif/phase4l.py']
source = next((p for p in candidates if p.is_file()), None)
self.assertIsNotNone(source, 'Original L source required for actual AST extraction test')
nodes = W.original_sampler_nodes(source.read_bytes())
self.assertEqual(len(nodes), 4)
self.assertNotIn('reset', ast.unparse(ast.Module(body=nodes, type_ignores=[])))
def test_registration_external_hash_rejection_before_parse(self):
with tempfile.TemporaryDirectory() as d:
exp = Path(d); (exp / 'protocol').mkdir()
P.save(exp / 'protocol/preregistered.json', {'fake': True})
with patch.object(P, 'EXP', exp):
with self.assertRaisesRegex(ValueError, 'external anchor'):
P.read_registration('0' * 64)
def test_initial_overlay_has_no_parent_process_lookup(self):
expected = dict(overlay_file='/immutable/zero.npz', overlay_manifest='/immutable/zero.json')
self.assertIs(W.before_overlay({'initial_overlay': expected}, 'unused', {'previous_delta_row_id': None}), expected)
def test_wrong_parent_arm_rejected_before_runtime_read(self):
parent = dict(row_id='parent', arm='D0', diagnostic_mode='light', input_epoch_index=0)
child = dict(previous_delta_row_id='parent', arm='D1', diagnostic_mode='light', input_epoch_index=1)
with self.assertRaisesRegex(ValueError, 'cross-epoch'):
W.before_overlay({'plan': {'calls': [parent]}}, 'unused', child)
def test_budget_refuses_overrun_without_delete(self):
with tempfile.TemporaryDirectory() as d:
exp = Path(d); p = exp / 'keep'
p.write_bytes(b'1234')
with patch.object(W, 'EXP', exp):
with self.assertRaisesRegex(ValueError, 'Resource cap'):
W.budget({'budget': {'output_budget_bytes': 1, 'minimum_free_reserve_bytes': 1}})
self.assertEqual(p.read_bytes(), b'1234')
def test_partial_log_is_not_retried(self):
with tempfile.TemporaryDirectory() as d:
exp = Path(d); (exp / 'logs').mkdir()
(exp / 'logs/input-generation.stdout.log').write_bytes(b'partial')
with patch.object(W, 'EXP', exp), patch.object(W.subprocess, 'run') as run:
with self.assertRaisesRegex(ValueError, 'never retry'): W.child_process({}, 'sha', 'materialize')
run.assert_not_called()
def test_admission_blocks_stopped_complete_or_sealed(self):
with tempfile.TemporaryDirectory() as d:
exp = Path(d); (exp / 'audit').mkdir()
with patch.object(W, 'EXP', exp):
W.admission()
for name in ('audit/stopped.json', 'audit/execution-complete.json', 'manifest.json'):
p = exp / name; p.write_bytes(b'{}')
with self.assertRaisesRegex(ValueError, 'Stopped/completed'): W.admission()
p.unlink()
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,197 @@
"""Synthetic sparse-event updates only; no recorded trajectories or training."""
import copy
import hashlib
import importlib.util
import json
import os
from pathlib import Path
import tempfile
import unittest
import numpy as np
import conditioning_update as m
def source_path():
# Same PHASE4J_SOURCE and ordered local candidates as frozen AM tests.
explicit = os.environ.get("PHASE4J_SOURCE")
if explicit:
return Path(explicit)
parent = Path(__file__).resolve().parents[1]
paths = (parent / "models/male-cns-lif/phase4j.py", parent / "mb-physiology-v1/phase4j.py")
for path in paths:
if path.is_file():
return path
raise FileNotFoundError("Set PHASE4J_SOURCE to the SHA-pinned original phase4j.py")
SOURCE = source_path()
BRIDGE = Path(os.environ.get("AM_ELIGIBILITY_BRIDGE",
str(Path(__file__).resolve().parents[1] / "wholebrain-plastic-interface-v1/eligibility_bridge.py")))
def args(rows=((0, 1), (3, 3)), us_tick=3, steps=8):
events = np.asarray(rows, np.int32).reshape(-1, 2)
return dict(step=events[:, 0], neuron_idx=events[:, 1], kc=np.array([1, 3, 7], np.int32),
candidate_edge_id=np.array([1, 4, 5, 10, 12], np.int64),
candidate_sources=np.array([1, 1, 3, 7, 3], np.int32),
candidate_targets=np.array([2, 2, 5, 5, 9], np.int32),
candidate_compartments=np.array(["y5B'2a", "a1", "y5B'2a", "y5B'2a", "y2a'1"]),
delta_before=np.zeros(5, np.float32), base_candidate=np.array([.275, .55, .0001, .275, 2.75], np.float32),
steps=steps, n_neurons=10, compartment="y5B'2a", us_tick=us_tick,
d_command=1.0 if us_tick is not None else 0.0, frozen=False,
bridge_source=BRIDGE, phase4j_source=SOURCE, trace_backend="numpy")
class ConditioningUpdate(unittest.TestCase):
def test_pinned_original_calls_and_recomputable_receipt(self):
inputs = args()
result = m.update_epoch(**inputs)
bridge = m._load_bridge(BRIDGE)
traced = bridge.trace_from_events(inputs["step"], inputs["neuron_idx"], inputs["kc"], 8, [3],
phase4j_source=SOURCE, backend="numpy", n_neurons=10)
expected, _ = bridge.apply_eligibility(traced.last_command_trace, inputs["kc"],
inputs["candidate_sources"], inputs["candidate_compartments"], inputs["compartment"],
inputs["delta_before"], inputs["base_candidate"], d=1., phase4j_source=SOURCE, n_neurons=10)
self.assertTrue(bridge.bit_equal(result.delta_after, expected))
self.assertTrue(m.verify_receipt(result.receipt, **inputs))
self.assertEqual(result.receipt["source_pins"]["eligibility_bridge_sha256"], hashlib.sha256(BRIDGE.read_bytes()).hexdigest())
self.assertEqual(result.receipt["trace_audit"]["sampling"], "after_current_tick_spike")
json.dumps(result.receipt, allow_nan=False)
def test_zero_d_and_frozen_preserve_existing_delta(self):
for d, frozen in ((0., False), (1., True), (0., True)):
inputs = args()
inputs["delta_before"][:] = -.25 * inputs["base_candidate"]
inputs.update(d_command=d, frozen=frozen)
result = m.update_epoch(**inputs)
self.assertEqual(result.delta_after.tobytes(), inputs["delta_before"].tobytes())
self.assertEqual(result.receipt["counts"]["original_updated_edges_attempted"], 0)
self.assertEqual(result.receipt["counts"]["actual_bitwise_changed_edges"], 0)
def test_us_before_cs_does_not_use_future_trace(self):
result = m.update_epoch(**args(rows=((2, 1),), us_tick=1))
self.assertFalse(np.any(result.sampled_trace))
self.assertTrue(np.any(result.final_trace))
self.assertFalse(np.any(result.delta_after))
def test_current_tick_and_reentrant_kc_saturate_before_sample(self):
result = m.update_epoch(**args(rows=((0, 1), (1, 1), (3, 1), (3, 3)), us_tick=3))
self.assertEqual(result.sampled_trace.tolist(), [1., 1., 0.])
self.assertEqual(result.delta_after[0].view(np.uint32), np.float32(-.00275).view(np.uint32))
self.assertEqual(result.receipt["counts"]["actual_KC_events"], 4)
def test_non_kc_only_changes_event_evidence_not_update(self):
a = m.update_epoch(**args(rows=((0, 1), (3, 3))))
b = m.update_epoch(**args(rows=((0, 0), (0, 1), (2, 9), (3, 3))))
self.assertEqual(a.delta_after.tobytes(), b.delta_after.tobytes())
self.assertEqual(a.sampled_trace.tobytes(), b.sampled_trace.tobytes())
self.assertNotEqual(a.receipt["input_arrays"], b.receipt["input_arrays"])
def test_no_us_uses_zero_not_final_trace(self):
result = m.update_epoch(**args(us_tick=None))
self.assertFalse(np.any(result.sampled_trace))
self.assertFalse(np.any(np.signbit(result.sampled_trace)))
self.assertTrue(np.any(result.final_trace))
self.assertEqual(result.receipt["counts"]["actual_value_changed_edges"], 0)
def test_attempts_and_fully_saturated_changes_are_distinct(self):
inputs = args(rows=((3, 1), (3, 3), (3, 7)))
inputs["delta_before"][:] = -.25 * inputs["base_candidate"]
result = m.update_epoch(**inputs)
counts = result.receipt["counts"]
self.assertEqual(counts["original_updated_edges_attempted"], 3)
self.assertEqual(counts["actual_value_changed_edges"], 0)
self.assertEqual(counts["selected_at_floor_before"], 3)
self.assertEqual(counts["selected_at_floor_after"], 3)
self.assertEqual(counts["selected_newly_at_floor"], 0)
def test_new_floor_and_off_compartment_identity(self):
inputs = args()
inputs["delta_before"][1] = -.01
result = m.update_epoch(**inputs)
self.assertEqual(result.receipt["counts"]["selected_newly_at_floor"], 1)
self.assertEqual(result.receipt["counts"]["actual_bitwise_changed_edges"], 2)
self.assertEqual(result.delta_after[[1, 3, 4]].tobytes(), inputs["delta_before"][[1, 3, 4]].tobytes())
def test_owned_readonly_results_and_all_input_bytes_unchanged(self):
inputs = args()
before = {k: v.tobytes() for k, v in inputs.items() if isinstance(v, np.ndarray)}
result = m.update_epoch(**inputs)
for key, raw in before.items():
self.assertEqual(inputs[key].tobytes(), raw)
for output in (result.delta_after, result.sampled_trace, result.final_trace):
self.assertTrue(output.flags.owndata)
self.assertFalse(output.flags.writeable)
self.assertTrue(all(not np.shares_memory(output, inputs[key]) for key in before))
inputs["delta_before"][:] = -.000001
self.assertNotEqual(result.delta_after.tobytes(), inputs["delta_before"].tobytes())
def test_scope_claims_do_not_prove_graph_or_dan_delivery(self):
receipt = m.update_epoch(**args()).receipt
for field in ("complete_KC_identity_verified", "complete_mask_identity_verified", "full_graph_authority_verified",
"brain_dynamics_executed", "weights_installed", "base_modified", "training_executed_by_this_function",
"scientific_learning_pass_claimed", "candidate_generated", "numpy_torch_bitwise_comparison_performed",
"frozen_Torch_expression_executed"):
self.assertIs(receipt[field], False)
self.assertFalse(receipt["semantics"]["actual_DAN_delivery_observed"])
self.assertFalse(receipt["semantics"]["actual_DAN_delivery_used_to_gate"])
def test_invalid_events_mapping_commands_and_values_fail_closed(self):
mutations = (
("step", np.array([0, 3], np.int64)), ("steps", True), ("us_tick", False),
("d_command", True), ("d_command", float("nan")), ("frozen", 1), ("trace_backend", "auto"),
("candidate_edge_id", np.array([1, 1, 5, 10, 12], np.int64)),
("candidate_targets", np.array([2, 2, 5, 5, 10], np.int32)),
("candidate_sources", np.array([0, 1, 3, 7, 3], np.int32)),
("base_candidate", np.array([.275, .55, float("inf"), .275, 2.75], np.float32)),
("delta_before", np.full(5, -0., np.float32)),
("kc", np.array([1, 1, 7], np.int32)),
)
for key, value in mutations:
inputs = args()
inputs[key] = value
with self.subTest(key=key), self.assertRaises(ValueError):
m.update_epoch(**inputs)
for rows in (((0, 1), (0, 1)), ((3, 3), (0, 1)), ((8, 1),)):
with self.assertRaises(ValueError):
m.update_epoch(**args(rows=rows))
inputs = args(us_tick=None)
inputs["d_command"] = 1
with self.assertRaises(ValueError):
m.update_epoch(**inputs)
def test_receipt_forgery_fails_after_new_self_hash(self):
inputs = args()
bad = copy.deepcopy(m.update_epoch(**inputs).receipt)
bad["counts"]["actual_value_changed_edges"] = 99
bad["payload_sha256"] = m._sha({k: v for k, v in bad.items() if k != "payload_sha256"})
with self.assertRaisesRegex(ValueError, "exact recomputation"):
m.verify_receipt(bad, **inputs)
def test_source_drift_rejected_before_loading(self):
with tempfile.TemporaryDirectory() as directory:
for field, source in (("bridge_source", BRIDGE), ("phase4j_source", SOURCE)):
path = Path(directory) / (field + ".py")
path.write_bytes(source.read_bytes() + b"\nraise RuntimeError('never execute')\n")
inputs = args()
inputs[field] = path
with self.subTest(field=field), self.assertRaisesRegex(ValueError, "SHA256 mismatch"):
m.update_epoch(**inputs)
@unittest.skipUnless(importlib.util.find_spec("torch") is not None,
"Torch absent: per-call NumPy/Torch exact compatibility NOT established locally")
def test_frozen_torch_cpu_matches_numpy_on_synthetic_events(self):
inputs = args(rows=((0, 1), (1, 1), (3, 3), (7, 7)), us_tick=7)
numpy_result = m.update_epoch(**inputs)
inputs["trace_backend"] = "torch"
torch_result = m.update_epoch(**inputs)
for field in ("delta_after", "sampled_trace", "final_trace"):
self.assertEqual(getattr(numpy_result, field).tobytes(), getattr(torch_result, field).tobytes())
self.assertTrue(torch_result.receipt["frozen_Torch_expression_executed"])
self.assertFalse(torch_result.receipt["numpy_torch_bitwise_comparison_performed"])
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,227 @@
"""Synthetic/mocked dispatch + temporary-array tests; no GPU or brain runs."""
import copy
from contextlib import ExitStack
import hashlib
import json
from pathlib import Path
import tempfile
import unittest
from unittest.mock import Mock, patch
import numpy as np
import engineering_analysis as a
import engineering_plan as p
PIN = 'a' * 64
WEIGHT = 'b' * 64
def raw(value):
return hashlib.sha256(value.tobytes()).hexdigest()
def record(row):
n, k, steps, start = 3, 2, row['steps'], row['start']
state = dict(step=np.empty(0, np.int32), neuron_idx=np.empty(0, np.int32), counts=np.zeros(n, np.int32),
v=np.zeros(n, np.float64), g=np.zeros(n, np.float64), ref=np.zeros(n, np.int32))
reference = row['mode'] == 'reference'
summary = dict(complete=True, pass_all=True, all_finite=True, counts_match_sparse_events=True,
steps=steps, completed_ticks=steps, nodes=n, edges=3, requested=0, total_spikes=0, accepted_requests=0,
rejected_requests=0, arriving_edges=0, accepted_arriving_edges=0, apl_fast_edges_zeroed=1, kc_targets_present=k,
precision='float64', apl_precision='float32', apl_delay_steps=18, diagnostic_mode=row['mode'],
effective_weights_raw_sha256=WEIGHT, within_trial_weight_updates=False, learning_pass_claimed=False,
checkpoint_continuation_used=start > 0, global_tick_offset=start, global_next_tick=start + steps,
wall_seconds=1., diagnostic_simulation_seconds=.5, build_or_cache_seconds=.2,
peak_allocated_vram_bytes=1000, peak_reserved_vram_bytes=2000)
return dict(source=row['row_id'], state=state, steps=steps, global_tick_offset=start, summary=summary,
terminal=dict(spike_ring=np.zeros((19, n), np.uint8), apl_state=np.zeros(2, np.float32),
apl_pending=np.zeros((18, k), np.float32), next_tick=np.asarray(steps, np.int64)),
requests=np.empty((0, 5), np.int64),
traces={name: np.zeros((steps, 2), np.float32) for name in a.compare.TRACE_FILES} if reference else {},
state_hashes=[dict(tick=t, v_sha256=raw(state['v']), g_sha256=raw(state['g']), ref_sha256=raw(state['ref'])) for t in range(steps)] if reference else None,
queues=[dict(tick=t, count=0, accepted_count=0, ordered_edge_sha256='c' * 64,
accepted_mask_sha256='c' * 64, joint_sha256='d' * 64) for t in range(steps)] if reference else None)
def fixture():
plan = p.build_plan(); exp = Path(tempfile.gettempdir()).resolve() / 'synthetic-project' / 'experiments/phase4an-engineering-validation'
reg = dict(schema='phase4an-engineering-registration-v1', plan=plan, training=False,
cases={r['row_id']: dict(case_dir=str(exp / 'cases' / r['row_id'])) for r in plan['rows']},
source_sha256={}, input_sha256={}, selected_case=dict(AM_outputs={'zero': 'mock', 'static25': 'mock'}),
overlays={'zero': {}, 'static25': {}})
verified = {r['row_id']: dict(schema='phase4an-engineering-runtime-record-v1', complete=True, training=False,
row=copy.deepcopy(r), registration_sha256=PIN, checkpoint_manifest_sha256=PIN,
runtime_and_immediate_seal_wall_seconds=2., storage={'state.npz': {'file_bytes': 10}, 'terminal-queues.npz': {'file_bytes': 20}})
for r in plan['rows']}
records = {r['row_id']: record(r) for r in plan['rows']}
reader = Mock(); reader.finish.return_value = []; reader.close.return_value = None
context = dict(reader=reader, records=records,
legacy={stage: record(next(r for r in plan['rows'] if r['overlay'] == stage and r['kind'] == 'fresh' and r['mode'] == 'reference')) for stage in p.OVERLAYS},
graph=dict(ptr=np.array([0, 1, 2, 3], np.int64), dst=np.array([1, 2, 0], np.int32), edge_id=np.arange(3, dtype=np.int64)),
requests=dict(tick=np.empty(0, np.int32), neuron_idx=np.empty(0, np.int32)), storage={r['row_id']: 600 for r in plan['rows']})
return reg, verified, context
class Orchestration(unittest.TestCase):
def mocked(self, reg, verified, context, *, fail=None, live=True, queue_exact=True, joint=True):
calls = []
def ordinary(name):
def f(*args):
calls.append(name)
return dict(numerical_exact=name != fail, checks={}, observed_scope={'mock_only': True})
return f
def chain(mono, parts):
calls.append('chain_exact')
reference = mono['summary']['diagnostic_mode'] == 'reference'
return dict(numerical_exact=fail != ('reference_chain' if reference else 'light_chain'),
pass_all=reference and live, nonvacuous_temporal_coverage=reference and live,
status='PASS' if reference else 'INCOMPLETE-COVERAGE',
coverage={'carried_APL_nonzero_applied_values': 1 if live and reference else 0,
'boundaries': [{'pending_source_spike_events': 1, 'carried_APL_nonzero_pending_values': 1 if joint else 0}]} if reference else None)
with ExitStack() as stack:
for name, value in dict(N=3, E=3, K=2, FAST_APL_EDGES=1).items():
stack.enter_context(patch.object(a, name, value))
stack.enter_context(patch.object(a, '_load_context', return_value=context))
stack.enter_context(patch.object(a, '_materialize', return_value=(np.ones(3, np.float64), {'effective_weights_raw_sha256': WEIGHT})))
stack.enter_context(patch.object(a.compare, 'compare_fresh', side_effect=ordinary('fresh')))
stack.enter_context(patch.object(a.scopes, 'compare_slice', side_effect=ordinary('slice')))
stack.enter_context(patch.object(a.scopes, 'compare_checkpoint', side_effect=ordinary('checkpoint')))
stack.enter_context(patch.object(a.scopes, 'compare_parent_carry', side_effect=ordinary('carry')))
stack.enter_context(patch.object(a, '_pair', side_effect=ordinary('pair')))
stack.enter_context(patch.object(a.compare, 'compare_segments', side_effect=chain))
queue_mock = stack.enter_context(patch.object(a.queue_audit, 'audit_queues', return_value=dict(
queue_ref_request_exact=queue_exact, accepted_nonzero_crossing_witness_verified=live and queue_exact)))
result = a.analyze(reg, PIN, verified)
return result, calls, queue_mock
def test_executes_all_descriptors_and_both_independent_queues(self):
reg, verified, ctx = fixture()
report, calls, queued = self.mocked(reg, verified, ctx)
expected = [r['row_id'] + '/' + str(i) for r in reg['plan']['rows'] for i, _ in enumerate(r['expected_comparisons'])]
self.assertEqual([r['descriptor_id'] for r in report['comparisons']], expected)
self.assertEqual(len(calls), len(expected))
self.assertEqual(report['evaluated_comparisons'], 74)
self.assertEqual(queued.call_count, 2)
self.assertEqual(queued.call_args.kwargs['cuts'], [5, 18, 19, 500])
self.assertEqual(report['status'], a.SUCCESS)
self.assertFalse(report['learning_pass_claimed'])
self.assertEqual(len(report['metrics']), 32)
self.assertTrue(all(row['gpu_utilization'] is None for row in report['metrics']))
self.assertTrue(all(r['light_standalone_coverage'] is None for r in report['overlay_reports'].values()))
json.dumps(report, allow_nan=False)
ctx['reader'].finish.assert_called_once()
ctx['reader'].close.assert_called_once()
def test_numerical_failure_still_evaluates_every_descriptor(self):
for failure in ('fresh', 'slice', 'checkpoint', 'carry', 'pair', 'reference_chain', 'light_chain'):
reg, verified, ctx = fixture()
with self.subTest(failure=failure):
report, calls, _ = self.mocked(reg, verified, ctx, fail=failure)
self.assertEqual(len(calls), 74)
self.assertEqual(report['status'], 'FAIL-NUMERICAL')
self.assertFalse(report['pass_all'])
def test_queue_failure_and_absent_live_coverage_are_distinct(self):
for arguments, expected in [({'queue_exact': False}, 'FAIL-NUMERICAL'),
({'live': False}, 'INCOMPLETE-COVERAGE'), ({'joint': False}, 'INCOMPLETE-COVERAGE')]:
reg, verified, ctx = fixture()
report, _, _ = self.mocked(reg, verified, ctx, **arguments)
self.assertEqual(report['status'], expected)
self.assertFalse(report['pass_all'])
def test_receipt_inventory_alias_and_descriptor_removal_fail_closed(self):
for mutation in ('receipt', 'descriptor', 'pin', 'case', 'row_alias'):
reg, verified, ctx = fixture(); identity = reg['plan']['rows'][0]['row_id']
if mutation == 'receipt':
verified.pop(identity)
elif mutation == 'descriptor':
reg['plan']['rows'][0]['expected_comparisons'] = []
elif mutation == 'pin':
verified[identity]['registration_sha256'] = 'f' * 64
elif mutation == 'row_alias':
verified[identity]['row']['steps'] = True
else:
reg['cases'][identity]['case_dir'] += '-another'
with self.subTest(mutation=mutation), self.assertRaises(ValueError):
self.mocked(reg, verified, ctx)
def test_summary_uses_actual_AJ_requested_names_and_checks_total_accounting(self):
for key in ('requested', 'accepted_requests', 'arriving_edges', 'accepted_arriving_edges', 'edges', 'effective_weights_raw_sha256'):
reg, verified, ctx = fixture(); identity = reg['plan']['rows'][0]['row_id']
ctx['records'][identity]['summary'][key] = 'e' * 64 if key.endswith('sha256') else 1
with self.subTest(key=key):
report, _, _ = self.mocked(reg, verified, ctx)
self.assertEqual(report['status'], 'FAIL-NUMERICAL')
self.assertIn(identity, report['failed_run_checks'])
def test_identical_bytes_give_deterministic_report_and_unknown_is_null(self):
reg, verified, ctx = fixture(); identity = reg['plan']['rows'][0]['row_id']
ctx['records'][identity]['summary'].pop('peak_reserved_vram_bytes')
one, _, _ = self.mocked(reg, verified, ctx)
two, _, _ = self.mocked(reg, verified, ctx)
self.assertEqual(a._canonical(one), a._canonical(two))
row = next(r for r in one['metrics'] if r['row_id'] == identity)
self.assertIsNone(row['peak_reserved_vram_bytes'])
def test_AM_reference_full_comparison_is_not_hidden_by_light(self):
reg, _, ctx = fixture()
row = reg['plan']['rows'][0]; descriptor = row['expected_comparisons'][0]
ctx['records'][row['row_id']]['traces']['drive'][10, 0] = np.float32(1)
report = a._dispatch(descriptor, ctx['records'], ctx['legacy'], row['overlay'])
self.assertFalse(report['numerical_exact'])
self.assertIn('pair.trace.drive', report['failed_checks'])
def test_missing_loaded_population_or_reference_inventory_rejected(self):
reg, verified, ctx = fixture(); ctx['records'].pop(next(iter(ctx['records'])))
with self.assertRaises(ValueError):
self.mocked(reg, verified, ctx)
class FileAndWeightInputs(unittest.TestCase):
def test_reader_rehashes_actual_content_and_never_uses_stat_cache(self):
with tempfile.TemporaryDirectory() as folder:
path = Path(folder).resolve() / 'data.bin'; path.write_bytes(b'AAAA')
reader = a._Reader({str(path): a._sha_file(path)})
reader.bind(path); path.write_bytes(b'BBBB')
with self.assertRaises(ValueError):
reader.bind(path)
with self.assertRaises(ValueError):
reader.finish()
def test_original_pinned_overlay_math_signed_zero_and_nonzero_only_bridge(self):
helper = Path(__file__).resolve().parents[1] / 'wholebrain-plastic-interface-v1/overlay.py'
self.assertEqual(a._sha_file(helper), a.AM_OVERLAY_SHA)
with tempfile.TemporaryDirectory() as folder:
folder = Path(folder).resolve()
mask = folder / 'mask.npy'; base = folder / 'base.npy'
np.save(mask, np.array([0], np.int64)); np.save(base, np.array([1.5], np.float32))
original = np.array([.002, -0., -0., .009], np.float64)
graph = dict(weights=original, edge_id=np.arange(4, dtype=np.int64), apl_out_mask=np.array([False, True, False, False]))
before = original.tobytes()
for stage in ('zero', 'static25'):
delta = np.array([-0. if stage == 'zero' else -.375], np.float32)
overlay = folder / (stage + '.npz'); manifest = folder / (stage + '.json')
np.savez_compressed(overlay, edge_id=np.array([0], np.int64), plastic_delta_mv=delta)
meta = dict(schema='phase4an-overlay-v1', overlay_name=stage, am_overlay_path=str(helper), am_overlay_sha256=a.AM_OVERLAY_SHA,
mask_path=str(mask), mask_sha256=a._sha_file(mask), base_candidate_path=str(base), base_candidate_sha256=a._sha_file(base),
overlay_sha256=a._sha_file(overlay))
manifest.write_text(json.dumps(meta), encoding='utf-8')
paths = [mask, base, overlay, manifest, helper]
reader = a._Reader({str(path): a._sha_file(path) for path in paths})
reg = {'overlays': {stage: dict(overlay_file=str(overlay), overlay_manifest=str(manifest))}}
with patch.object(a, 'MASK_SHA', meta['mask_sha256']), patch.object(a, 'CANDIDATES', 1), patch.object(a, 'FAST_APL_EDGES', 1):
actual, info = a._materialize(reader, reg, graph, stage)
expected = original.copy(); expected[1] = 0.
if stage == 'static25':
expected[0] = np.add(original[0], np.multiply(delta[0].astype(np.float64), np.float64(.001)))
self.assertEqual(actual.tobytes(), expected.tobytes())
self.assertEqual(info['effective_weights_raw_sha256'], raw(expected))
self.assertEqual(info['installed_nonzero_count'], 0 if stage == 'zero' else 1)
self.assertEqual(original.tobytes(), before)
reader.finish(); reader.close()
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,246 @@
"""Synthetic artifact comparisons only; no neural equation or GPU execution."""
import copy
import hashlib
import json
from pathlib import Path
import tempfile
import unittest
import numpy as np
import engineering_compare as c
def sha(value):
return hashlib.sha256(np.ascontiguousarray(value).tobytes()).hexdigest()
def synthetic_parts(lengths, *, active=True, traces=True, queues=True):
n, k = 3, 2
total = sum(lengths)
all_ticks = np.array([value for value in (0, 7, 19, 21, 41, 61) if value < total], np.int32) if active else np.empty(0, np.int32)
all_nodes = np.array([0, 1, 2, 0, 2, 1][:len(all_ticks)], np.int32)
ring, apl_ring = np.zeros((19, n), np.uint8), np.zeros((18, k), np.float32)
phase, offset, result = 0, 0, []
for length in lengths:
ring = ring[(np.arange(19) + phase % 19) % 19].copy()
head = phase % 18
apl_ring = np.concatenate((apl_ring[head:], apl_ring[:head]))
event_tick, event_node, requested, hashes, queue_rows = [], [], [], [], []
generated, applied, drives, apl_states = [], [], [], []
for tick in range(length):
absolute = offset + tick
fired = all_nodes[all_ticks == absolute]
ring[tick % 19] = 0; ring[tick % 19, fired] = 1
for neuron in fired:
event_tick.append(tick); event_node.append(neuron)
requested.append([tick, int(neuron), 1, 1, 0])
due = ring[(tick - 18) % 19] if absolute >= 18 else np.zeros(n, np.uint8)
edges = np.array([int(node) * 2 + i for node in np.flatnonzero(due) for i in (0, 1)], np.int64)
accepted = np.ones(len(edges), np.uint8)
queue_rows.append({'tick': tick, 'count': len(edges), 'accepted_count': len(edges),
'ordered_edge_sha256': sha(edges), 'accepted_mask_sha256': sha(accepted),
'joint_sha256': hashlib.sha256(str(tick).encode() + edges.tobytes() + accepted.tobytes()).hexdigest()})
v = np.array([-.05, -.04, -.03], np.float64)
g = np.array([-0., float(absolute), -.001], np.float64)
ref = np.zeros(n, np.int32)
hashes.append({'tick': tick, 'v_sha256': sha(v), 'g_sha256': sha(g), 'ref_sha256': sha(ref)})
h = np.full(k, np.float32(absolute + 1 if active else 0), np.float32)
application = apl_ring[tick % 18].copy(); apl_ring[tick % 18] = h
generated.append(h); applied.append(application)
drives.append(np.full(2, np.float32(absolute if active else 0), np.float32))
apl_states.append(np.full(2, np.float32(absolute + 2 if active else 0), np.float32))
local_ticks, nodes = np.array(event_tick, np.int32), np.array(event_node, np.int32)
state = {'step': local_ticks, 'neuron_idx': nodes, 'counts': np.bincount(nodes, minlength=n).astype(np.int32), 'v': v, 'g': g, 'ref': ref}
head = length % 18
pending = np.concatenate((apl_ring[head:], apl_ring[:head]))
terminal = {'spike_ring': ring.copy(), 'apl_state': apl_states[-1].copy(), 'apl_pending': pending,
'next_tick': np.asarray(length, np.int64)}
summary = {'complete': True, 'pass_all': True, 'all_finite': True, 'steps': length, 'completed_ticks': length,
'nodes': n, 'precision': 'float64', 'apl_precision': 'float32', 'apl_delay_steps': 18,
'checkpoint_continuation_used': offset > 0, 'global_tick_offset': offset, 'global_next_tick': offset + length,
'effective_weights_raw_sha256': 'e' * 64, 'diagnostic_mode': 'reference' if traces else 'light'}
result.append({'source': 'synthetic', 'steps': length, 'global_tick_offset': offset, 'state': state,
'requests': np.asarray(requested, np.int64).reshape(-1, 5), 'terminal': terminal,
'summary': summary, 'terminal_origin': 'synthetic_recorded',
'traces': {'generated': np.asarray(generated), 'applied': np.asarray(applied),
'drive': np.asarray(drives), 'state': np.asarray(apl_states)} if traces else {},
'state_hashes': hashes if traces else None, 'queues': queue_rows if queues else None})
offset += length; phase = length
return result
def write_output(path, run, *, legacy=False):
path.mkdir()
for name, value in (('summary.json', run['summary']), ('apl-timing-audit.json',
{'final_apl_state_u32_bits': run['terminal']['apl_state'].view(np.uint32).tolist()})):
(path / name).write_text(json.dumps(value), encoding='utf-8')
np.savez_compressed(path / 'state.npz', **run['state'])
np.save(path / 'apl-pending-tail.float32.npy', run['terminal']['apl_pending'])
if legacy:
(path / 'adapter-provenance.json').write_text('{}', encoding='utf-8')
else:
np.savez_compressed(path / 'terminal-queues.npz', **run['terminal'])
with (path / 'requests.csv').open('w', encoding='utf-8', newline='') as stream:
import csv
writer = csv.writer(stream); writer.writerow(c.REQUEST_FIELDS); writer.writerows(run['requests'].tolist())
for name, file in c.TRACE_FILES.items():
if name in run['traces']:
np.save(path / file, run['traces'][name])
for name, file in (('state_hashes', 'tick-state-hashes.jsonl'), ('queues', 'queue.jsonl')):
if run[name] is not None:
(path / file).write_text(''.join(json.dumps(row) + '\n' for row in run[name]), encoding='utf-8')
class Comparisons(unittest.TestCase):
def test_fresh_light_reference_exact_without_fabricating_light_traces(self):
reference = synthetic_parts([64])[0]
light = synthetic_parts([64], traces=False, queues=False)[0]
report = c.compare_fresh(light, reference)
self.assertTrue(report['pass_all'])
self.assertFalse(report['reference_comparisons_available']['APL_traces'])
self.assertFalse(report['temporal_validation_complete'])
def test_light_does_not_hide_reference_vs_AM_interior_trace_mismatch(self):
light = synthetic_parts([64], traces=False, queues=False)[0]
reference = synthetic_parts([64])[0]
legacy = copy.deepcopy(reference)
reference['traces']['drive'][10, 0] += np.float32(1.)
report = c.compare_fresh(light, reference, legacy)
self.assertFalse(report['pass_all'])
self.assertIn('reference_vs_AM.trace.drive', report['failed_checks'])
def test_nonzero_origin_monolithic_and_split_require_same_parent(self):
mono = synthetic_parts([20, 44])[1]
parts = synthetic_parts([20, 5, 39])[1:]
with self.assertRaisesRegex(ValueError, 'same starting parent'):
c.compare_segments(mono, parts)
mono['summary']['parent_checkpoint_manifest_sha256'] = 'a' * 64
parts[0]['summary']['parent_checkpoint_manifest_sha256'] = 'a' * 64
report = c.compare_segments(mono, parts)
self.assertTrue(report['pass_all'], report['failed_checks'])
self.assertEqual(report['global_tick_offset'], 20)
parts[0]['summary']['parent_checkpoint_manifest_sha256'] = 'b' * 64
with self.assertRaisesRegex(ValueError, 'same starting parent'):
c.compare_segments(mono, parts)
def test_monolithic_segments_exact_and_cross_boundary_coverage(self):
mono = synthetic_parts([64])[0]
parts = synthetic_parts([5, 12, 1, 1, 18, 19, 8])
report = c.compare_segments(mono, parts)
self.assertTrue(report['pass_all'], report['failed_checks'])
self.assertTrue(report['reference_comparisons_available']['APL_traces'])
self.assertTrue(report['reference_comparisons_available']['state_hashes'])
self.assertGreater(report['coverage']['observed_carried_arriving_edges'], 0)
self.assertGreater(report['coverage']['carried_APL_nonzero_applied_values'], 0)
self.assertFalse(report['queue_joint_hash_compared'])
self.assertNotEqual(mono['queues'][18]['joint_sha256'], parts[3]['queues'][0]['joint_sha256'])
def test_raw_ring_phase_must_be_normalized(self):
mono = synthetic_parts([64])[0]
parts = synthetic_parts([20, 44])
self.assertFalse(c.bit_equal(mono['terminal']['spike_ring'], parts[-1]['terminal']['spike_ring']))
self.assertTrue(c.bit_equal(c.normalized_ring(mono['terminal']), c.normalized_ring(parts[-1]['terminal'])))
self.assertTrue(c.compare_segments(mono, parts)['pass_all'])
def test_signed_zero_final_state_difference_is_not_equal(self):
left = synthetic_parts([64], traces=False, queues=False)[0]
right = copy.deepcopy(left); right['state']['g'][0] = 0.
report = c.compare_fresh(left, right)
self.assertFalse(report['numerical_exact'])
self.assertIn('pair.state.g', report['failed_checks'])
def test_zero_activity_is_not_a_complete_temporal_pass(self):
mono = synthetic_parts([64], active=False)[0]
parts = synthetic_parts([20, 44], active=False)
report = c.compare_segments(mono, parts)
self.assertTrue(report['numerical_exact'])
self.assertFalse(report['pass_all'])
self.assertFalse(report['temporal_validation_complete'])
self.assertEqual(report['status'], 'INCOMPLETE-COVERAGE')
self.assertFalse(c.compare_fresh(mono, copy.deepcopy(mono))['pass_all'])
def test_missing_observed_queue_coverage_is_reported_unknown_not_zero_or_pass(self):
mono = synthetic_parts([64], traces=False, queues=False)[0]
parts = synthetic_parts([20, 44], traces=False, queues=False)
report = c.compare_segments(mono, parts)
self.assertTrue(report['numerical_exact'])
self.assertIsNone(report['coverage']['observed_carried_arriving_edges'])
self.assertFalse(report['temporal_validation_complete'])
def test_request_acceptance_difference_fails_even_if_events_are_equal(self):
left = synthetic_parts([64])[0]; right = copy.deepcopy(left)
right['requests'][0, 3:] = [0, 1]
report = c.compare_fresh(left, right)
self.assertIn('pair.requests_five_fields', report['failed_checks'])
def test_clock_gaps_and_empty_segment_list_rejected(self):
mono = synthetic_parts([64])[0]
with self.assertRaises(ValueError):
c.compare_segments(mono, [])
parts = synthetic_parts([20, 44]); parts[1]['global_tick_offset'] += 1
with self.assertRaisesRegex(ValueError, 'clock gap'):
c.compare_segments(mono, parts)
def test_partial_traces_hash_coverage_and_bad_dtype_rejected(self):
for mutation in ('trace', 'hash', 'dtype', 'duplicate'):
run = synthetic_parts([64])[0]
if mutation == 'trace':
run['traces'].pop('applied')
elif mutation == 'hash':
run['state_hashes'].pop()
elif mutation == 'dtype':
run['state']['v'] = run['state']['v'].astype(np.float32)
else:
run['state']['step'][1] = run['state']['step'][0]
run['state']['neuron_idx'][1] = run['state']['neuron_idx'][0]
with self.subTest(mutation=mutation), self.assertRaises(ValueError):
c.validate_run(run)
def test_per_step_state_hash_and_queue_mask_mismatch_fail(self):
mono = synthetic_parts([64])[0]; parts = synthetic_parts([20, 44])
parts[1]['state_hashes'][1]['g_sha256'] = '0' * 64
parts[1]['queues'][1]['accepted_mask_sha256'] = '1' * 64
report = c.compare_segments(mono, parts)
self.assertFalse(report['pass_all'])
self.assertIn('concatenated_per_tick_state_hashes', report['failed_checks'])
self.assertIn('concatenated_queue_order_and_acceptance', report['failed_checks'])
def test_effective_weight_change_fails(self):
mono = synthetic_parts([64])[0]; parts = synthetic_parts([20, 44])
parts[1]['summary']['effective_weights_raw_sha256'] = 'f' * 64
self.assertFalse(c.compare_segments(mono, parts)['checks']['unchanged_effective_weights'])
class FileSchemas(unittest.TestCase):
def test_actual_five_request_fields_and_AM_derived_ring_are_labeled(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
run = synthetic_parts([64])[0]
write_output(root / 'AN', run)
write_output(root / 'AM', run, legacy=True)
actual, legacy = c.load_run(root / 'AN'), c.load_run(root / 'AM', allow_legacy_am=True)
try:
self.assertIn('derived_from_fresh_events', legacy['terminal_origin'])
self.assertTrue(c.compare_fresh(actual, copy.deepcopy(actual), legacy)['pass_all'])
with self.assertRaisesRegex(ValueError, 'missing'):
c.load_run(root / 'AM')
finally:
c.close_run(actual); c.close_run(legacy)
def test_missing_reference_file_or_wrong_request_header_rejected(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
run = synthetic_parts([64])[0]
write_output(root / 'partial', run)
(root / 'partial' / 'apl-applied.float32.npy').unlink()
with self.assertRaisesRegex(ValueError, 'incomplete'):
c.load_run(root / 'partial')
write_output(root / 'header', run)
(root / 'header' / 'requests.csv').write_text('tick,neuron_idx,accepted\n', encoding='utf-8')
with self.assertRaisesRegex(ValueError, 'five AJ'):
c.load_run(root / 'header')
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,115 @@
import copy
import unittest
import numpy as np
import engineering_inputs as e
def inputs():
return dict(tick=np.array([0, 4, 5, 17, 18, 18, 19, 499, 500, 999], np.int32),
neuron_idx=np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 166699], np.int32))
def spec():
return dict(name='unchanged', kind='fullgraph', steps=1000, dt_ms=.1, duration_ms=100.,
apl_delay_steps=18, record_history=False, training=False,
model={'threshold_mv': -45, 'w_syn_mv_per_anatomical_synapse': .275},
apl={'gain': .01}, budget_root='/project/old', output_budget_bytes=2048,
graph_dir='/project/frozen-graph', seed=1073741825)
class ExactInputs(unittest.TestCase):
def test_exact_boundaries_reconstruct_all_requests(self):
original = inputs()
intervals = [(0, 5), (5, 18), (18, 19), (19, 500), (500, 1000)]
parts = [e.slice_requests(original, 1000, *span) for span in intervals]
report = e.verify_partition(original, 1000, intervals, parts)
self.assertTrue(report['complete'])
self.assertEqual(parts[2]['tick'].tolist(), [0, 0])
self.assertEqual(parts[-1]['tick'].tolist(), [0, 499])
self.assertEqual(report['random_draws'], 0)
for value in parts:
for key in original:
self.assertFalse(np.shares_memory(original[key], value[key]))
def test_empty_intervals_are_explicit_not_lost(self):
original = inputs()
intervals = [(0, 20), (20, 499), (499, 1000)]
parts = [e.slice_requests(original, 1000, *span) for span in intervals]
self.assertEqual(len(parts[1]['tick']), 0)
self.assertTrue(e.verify_partition(original, 1000, intervals, parts)['complete'])
def test_bad_requests_never_silently_fixed(self):
for mut in ('dtype', 'duplicate', 'order', 'bounds', 'shape', 'extra'):
req = inputs()
if mut == 'dtype': req['tick'] = req['tick'].astype(np.int64)
elif mut == 'duplicate': req['tick'][1], req['neuron_idx'][1] = 0, 1
elif mut == 'order': req['tick'][1] = 10
elif mut == 'bounds': req['neuron_idx'][-1] = 166700
elif mut == 'shape': req['tick'] = req['tick'][:-1]
else: req['seed'] = 1
with self.subTest(mut=mut), self.assertRaises(ValueError):
e.slice_requests(req, 1000, 0, 1000)
def test_partition_gap_overlap_changed_input_and_missing_tail_rejected(self):
req = inputs()
spans = [(0, 500), (500, 1000)]
parts = [e.slice_requests(req, 1000, *x) for x in spans]
for bad in ([(0, 500), (501, 1000)], [(0, 500), (499, 1000)], [(0, 500)]):
with self.assertRaises(ValueError):
e.verify_partition(req, 1000, bad, parts[:len(bad)])
parts[0]['neuron_idx'][0] = 123
with self.assertRaisesRegex(ValueError, 'changes actual requests'):
e.verify_partition(req, 1000, spans, parts)
def test_fresh_prefix_and_continued_specs_preserve_original(self):
original = spec()
before = copy.deepcopy(original)
for start, stop, parent in ((0, 1000, None), (0, 18, None), (18, 19, 'a' * 64)):
derived, audit = e.derive_spec(original, start=start, stop=stop, budget_root='/project/new',
output_budget_bytes=8 * 1024**3, parent_sha=parent)
self.assertEqual(original, before)
self.assertEqual(derived['model'], original['model'])
self.assertEqual(derived['apl'], original['apl'])
self.assertIsNot(derived['model'], original['model'])
self.assertEqual(derived['max_steps'], 1000)
self.assertEqual(derived['steps'], stop - start)
self.assertTrue(audit['all_other_fields_unchanged'])
if start: self.assertEqual(derived['global_tick_offset'], start)
def test_invalid_continuation_root_and_duration_rejected(self):
for root in ('/project/old/sub', '/project', 'relative', '/project/../old', '//project/old', '//project/new'):
with self.assertRaises(ValueError):
e.derive_spec(spec(), start=0, stop=1000, budget_root=root, output_budget_bytes=10)
for start, stop, parent in ((1, 20, None), (0, 10, 'a'*64), (True, 10, None), (0, 1001, None)):
with self.assertRaises(ValueError):
e.derive_spec(spec(), start=start, stop=stop, budget_root='/project/new', output_budget_bytes=10, parent_sha=parent)
original = spec(); original['duration_ms'] = 1.
with self.assertRaises(ValueError):
e.derive_spec(original, start=0, stop=10, budget_root='/project/new', output_budget_bytes=10)
def test_original_clock_type_and_no_within_trial_update_match_runtime(self):
for key, value in (('apl_delay_steps', 18.), ('apl_delay_steps', True),
('within_trial_weight_updates', True), ('budget_root', '//project/old')):
original = spec(); original[key] = value
with self.subTest(key=key, value=value), self.assertRaises(ValueError):
e.derive_spec(original, start=0, stop=10, budget_root='/project/new', output_budget_bytes=10)
def test_overlay_is_declaration_translation_not_self_authorized_learned(self):
original = dict(schema='phase4am-overlay-v1', overlay_name='zero', training=False, fixed_for_whole_trial=True,
overlay_sha256='b'*64, mask_path='/project/mask.npy', mask_sha256=e.MASK_SHA,
base_candidate_path='/project/base.npy', base_candidate_sha256='c'*64,
aj_model_dir='/project/apl-timing-lab-v1', old_extra_field='preserved in original only')
before = copy.deepcopy(original)
result = e.adapt_overlay_manifest(original, '/project/overlay.py')
self.assertEqual(original, before)
self.assertEqual(result['schema'], 'phase4an-overlay-v1')
self.assertNotIn('old_extra_field', result)
self.assertEqual(result['overlay_sha256'], original['overlay_sha256'])
for key, val in (('overlay_name', 'learned'), ('mask_sha256', 'd'*64), ('training', True), ('base_candidate_path', 'local')):
bad = dict(original); bad[key] = val
with self.subTest(key=key), self.assertRaises(ValueError):
e.adapt_overlay_manifest(bad, '/project/overlay.py')
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,152 @@
"""Pure inventory/dependency tests; no I/O, GPU, scientific input or runtime."""
import copy
import json
import unittest
import engineering_plan as p
class FixedPlan(unittest.TestCase):
def test_complete_current_inventory_not_future_supplements(self):
plan = p.build_plan()
self.assertEqual(plan['total_calls'], len(plan['rows']))
self.assertEqual(plan['total_calls'], 32)
self.assertEqual(plan['total_ticks'], sum(row['steps'] for row in plan['rows']))
self.assertEqual(plan['total_ticks'], 11074)
self.assertEqual(plan['overlays'], ['zero', 'static25'])
self.assertEqual(plan['chain_lengths'], [5, 13, 1, 481, 500])
self.assertFalse(plan['training'])
self.assertFalse(plan['candidate_generated'])
self.assertEqual(p.validate_plan(json.loads(json.dumps(plan))), plan)
for overlay in plan['overlays']:
rows = [r for r in plan['rows'] if r['overlay'] == overlay]
self.assertEqual(len(rows), 16)
self.assertEqual(sum(r['steps'] for r in rows), 5537)
self.assertEqual([(r['mode'], r['steps']) for r in rows if r['kind'] == 'fresh'],
[('reference', 1000), ('light', 1000), ('light', 1000)])
self.assertEqual([r['stop'] for r in rows if r['kind'] == 'prefix'], [18, 19, 500])
self.assertFalse(any(r['stop'] > 1000 for r in rows))
def test_all_prerequisites_precede_rows_and_fresh_prefixes_precede_chains(self):
rows = p.build_plan()['rows']
self.assertTrue(all(r['kind'] in ('fresh', 'prefix') for r in rows[:12]))
self.assertTrue(all(r['kind'] == 'chain' for r in rows[12:]))
positions = {r['row_id']: i for i, r in enumerate(rows)}
self.assertEqual(len(positions), 32)
for index, row in enumerate(rows):
if row['parent_row_id'] is not None:
self.assertLess(positions[row['parent_row_id']], index)
for check in row['expected_comparisons']:
reference = check['reference_row_id']
if reference is not None:
self.assertLess(positions[reference], index)
for candidate in check['candidate_row_ids']:
self.assertLessEqual(positions[candidate], index)
self.assertLess(check['start'], check['stop'])
def test_each_chain_is_exact_partition_with_local_phase_distinct_from_global(self):
for overlay in p.OVERLAYS:
for mode in p.MODES:
chain = [r for r in p.build_plan()['rows'] if
(r['overlay'], r['mode'], r['kind']) == (overlay, mode, 'chain')]
self.assertEqual([(r['start'], r['stop']) for r in chain],
[(0, 5), (5, 18), (18, 19), (19, 500), (500, 1000)])
self.assertEqual([r['steps'] % 19 for r in chain], [5, 13, 1, 6, 6])
self.assertEqual([r['stop'] % 19 for r in chain], [5, 18, 0, 6, 12])
self.assertIsNone(chain[0]['parent_row_id'])
for earlier, later in zip(chain, chain[1:]):
self.assertEqual(later['parent_row_id'], earlier['row_id'])
self.assertEqual(later['start'], earlier['stop'])
self.assertEqual(chain[2]['start'], 18) # source tick 0 arrives at this child's local 0.
self.assertEqual(chain[2]['steps'], 1)
last = next(c for c in chain[-1]['expected_comparisons'] if c['check'] == 'chain_exact')
self.assertEqual(last['candidate_row_ids'], [r['row_id'] for r in chain])
self.assertEqual((last['start'], last['stop']), (0, 1000))
def test_prefix5_is_reference_first_segment_without_extra_run(self):
rows = p.build_plan()['rows']
self.assertFalse(any(r['kind'] == 'prefix' and r['stop'] == 5 for r in rows))
for overlay in p.OVERLAYS:
first = {r['mode']: r for r in rows if r['kind'] == 'chain' and
r['overlay'] == overlay and r['start'] == 0}
refs = [c for c in first['light']['expected_comparisons'] if c['check'] == 'checkpoint_exact']
self.assertEqual(len(refs), 1)
self.assertEqual(refs[0]['reference_row_id'], first['reference']['row_id'])
self.assertFalse(any(c['reference_row_id'] == first['reference']['row_id']
for c in first['reference']['expected_comparisons']))
def test_fresh_am_and_light_repeat_comparisons_are_explicit(self):
rows = p.build_plan()['rows']
for overlay in p.OVERLAYS:
fresh = [r for r in rows if r['overlay'] == overlay and r['kind'] == 'fresh']
self.assertEqual(len(fresh), 3)
for row in fresh:
am = [c for c in row['expected_comparisons'] if c['check'] == 'am_exact']
self.assertEqual(len(am), 1)
self.assertEqual(am[0]['external_reference'], 'sealed_AM_selected_case_same_overlay_r0')
self.assertIsNone(am[0]['reference_row_id'])
self.assertEqual(fresh[1]['expected_comparisons'][1]['reference_row_id'], fresh[0]['row_id'])
self.assertEqual(fresh[2]['expected_comparisons'][1]['reference_row_id'], fresh[1]['row_id'])
def test_every_result_is_independent_not_shared_mutable_inventory(self):
original = p.build_plan()
checked = p.validate_plan(original)
checked['rows'][0]['expected_comparisons'][0]['candidate_row_ids'].append('mutation')
checked['chain_lengths'][0] = 99
self.assertEqual(original, p.build_plan())
original['rows'][0]['expected_comparisons'][0]['scope'] = 'mutation'
self.assertNotEqual(original, p.build_plan())
def test_exact_types_reject_bool_int_float_and_sequence_conflation(self):
mutations = (
lambda x: x.update(total_calls=32.0),
lambda x: x.update(training=0),
lambda x: x['rows'][0].update(start=False),
lambda x: x['rows'][14].update(steps=True),
lambda x: x['rows'][14].update(steps=1.0),
lambda x: x['rows'][0]['expected_comparisons'][0].update(start=False),
lambda x: x.update(rows=tuple(x['rows'])),
)
for mutate in mutations:
plan = p.build_plan(); mutate(plan)
with self.subTest(mutation=repr(mutate)), self.assertRaisesRegex(ValueError, 'type'):
p.validate_plan(plan)
def test_unknown_fields_missing_overlays_and_reordering_are_rejected(self):
mutations = (
lambda x: x.update(extra=True),
lambda x: x['rows'][0].update(extra='not allowed'),
lambda x: x['rows'][0]['expected_comparisons'][0].update(tolerance=0),
lambda x: x.update(overlays=['zero']),
lambda x: x['rows'].pop(0),
lambda x: x['rows'].append(copy.deepcopy(x['rows'][-1])),
lambda x: x['rows'].__setitem__(slice(0, 2), x['rows'][1::-1]),
lambda x: x['rows'][0].update(overlay='static25'),
lambda x: x['rows'][0].update(mode='light'),
lambda x: x['rows'][0].update(row_id='arbitrary'),
)
for mutate in mutations:
plan = p.build_plan(); mutate(plan)
with self.subTest(mutation=repr(mutate)), self.assertRaises(ValueError):
p.validate_plan(plan)
def test_gaps_wrong_parents_and_dropped_comparisons_never_pass(self):
mutations = (
lambda x: x['rows'][13].update(start=6, steps=12),
lambda x: x['rows'][13].update(stop=17, steps=12),
lambda x: x['rows'][13].update(parent_row_id=None),
lambda x: x['rows'][12].update(parent_row_id=x['rows'][13]['row_id']),
lambda x: x['rows'][16].update(stop=999, steps=499),
lambda x: x['rows'][0].update(expected_comparisons=[]),
lambda x: x['rows'][2]['expected_comparisons'].pop(),
lambda x: x['rows'][16]['expected_comparisons'][-1]['candidate_row_ids'].pop(),
lambda x: x['rows'][16]['expected_comparisons'][-1].update(scope='final_only'),
)
for mutate in mutations:
plan = p.build_plan(); mutate(plan)
with self.subTest(mutation=repr(mutate)), self.assertRaises(ValueError):
p.validate_plan(plan)
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,216 @@
"""Temporary-directory and small authority tests only; never touch real ROOT."""
import copy
import hashlib
import json
from pathlib import Path
import tempfile
import types
import unittest
from unittest.mock import patch
import engineering_prepare as p
LOCAL = Path(__file__).resolve().parent
AI = LOCAL.parent / 'pn-recruitment-audit-v1/publication.py'
def valid_status():
final = dict(status=p.FINAL_STATUS, primary_trials_per_stage=120, repeats_per_stage=24,
zero_overlay_exact=True, static25_repeat_exact=True, old_integrity_pass=True,
frozen_five_reference_pass=True, frozen_eligibility_and_mask_unchanged=True,
base_weights_unchanged=True, training=False, learned_candidate=False)
scientific = dict(status=p.FINAL_STATUS, primary_trials_per_stage=120, technical_repeats_per_stage=24,
zero_overlay_exact=True, static25_repeat_exact=True, training=False, learned_candidate=False)
old = {'root-' + str(i): {'pass': True, 'checked': 1, 'mismatches': [], 'manifest_sha256': 'a'*64, 'checksums_sha256': 'b'*64}
for i in range(28)}
refs = {'pass_all': True, 'trials': [{'trial': name, 'exact': {key: True for key in ('step', 'neuron_idx', 'counts', 'v', 'g')}} for name in p.REFERENCES]}
return final, scientific, old, copy.deepcopy(old), refs
def planned():
return [dict(name='fixed-' + str(i), cohort='seed_validation' if i < 60 else 'novel_to_aj', repeat=r)
for i in range(120) for r in ((0, 1) if i < 24 else (0,))]
class AuthorityTests(unittest.TestCase):
def test_required_final_status_and_all_28_roots_five_references(self):
p.validate_final_status(*valid_status())
for mutation in ('status', 'hidden-learning', 'root', 'reference', 'root-hash'):
args = copy.deepcopy(valid_status())
if mutation == 'status':
args[0]['status'] = 'INCOMPLETE'
elif mutation == 'hidden-learning':
args[1]['learned_candidate'] = True
elif mutation == 'root':
args[3].pop('root-27')
elif mutation == 'reference':
args[4]['trials'][2]['exact']['g'] = False
else:
args[3]['root-1']['manifest_sha256'] = 'c' * 64
with self.subTest(mutation=mutation), self.assertRaises(ValueError):
p.validate_final_status(*args)
def test_first_actual_planned_primary_is_selected_without_dynamic_ranking(self):
rows = planned()
self.assertEqual(p.validate_planned(rows), rows[0])
for altered in (rows[:-1], rows + [rows[0]], [dict(rows[0], name='../escape'), *rows[1:]],
[rows[1], rows[1], *rows[2:]]):
with self.assertRaises(ValueError):
p.validate_planned(altered)
def test_three_external_sha_anchors_have_no_defaults_or_placeholders(self):
for anchors in ((None, 'a'*64, 'b'*64), ('', '', ''), ('x'*64, 'a'*64, 'b'*64)):
with self.assertRaisesRegex(ValueError, 'actual external'):
p._prepare(Path('not-read'), Path('not-read'), Path('not-read'), *anchors)
def test_path_inventory_rejects_duplicates_and_traversal(self):
for name in ('../bad', '/absolute', 'a\\b', 'a/../b', 'C:/bad'):
with self.assertRaises(ValueError):
p._relative(name)
row = {'path': 'good.json', 'bytes': 3, 'sha256': 'a'*64}
with self.assertRaisesRegex(ValueError, 'Duplicate'):
p._index([row, row])
def test_pinned_AI_verifier_rechecks_complete_synthetic_seal(self):
helper = p.load_module('phase4an_test_AI_publication', AI, p.AI_HELPER_SHA)
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()
payload = root / 'small.json'; payload.write_text('{}', encoding='utf-8')
row = {'path': payload.name, 'bytes': payload.stat().st_size, 'sha256': p.digest(payload)}
p.save(root / 'manifest.json', {'root': str(root), 'files': [row]})
(root / 'checksums.sha256').write_text(row['sha256'] + ' small.json\n' + p.digest(root / 'manifest.json') + ' manifest.json\n', encoding='utf-8')
self.assertTrue(helper.verify_seal(root)['pass_all'])
payload.write_text('{"changed":true}', encoding='utf-8')
with self.assertRaises(ValueError):
helper.verify_seal(root)
def test_finalizer_authority_helper_checks_external_anchors_and_both_full_seals(self):
helper = p.load_module('phase4an_test_AI_publication', AI, p.AI_HELPER_SHA)
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()
model, experiment = root / p.AM_MODEL_REL, root / p.AM_EXP_REL
model.mkdir(parents=True); experiment.mkdir(parents=True)
p.save(model / 'payload.json', {'synthetic': True})
p.save(experiment / 'protocol/preregistered.json', {'planned': []})
p.save(experiment / 'audit/final-status.json', {'status': p.FINAL_STATUS})
for tree in (model, experiment):
rows = [dict(path=path.relative_to(tree).as_posix(), bytes=path.stat().st_size, sha256=p.digest(path))
for path in sorted(tree.rglob('*')) if path.is_file()]
p.save(tree / 'manifest.json', {'root': str(tree), 'files': rows})
(tree / 'checksums.sha256').write_text(''.join(row['sha256'] + ' ' + row['path'] + '\n' for row in rows) +
p.digest(tree / 'manifest.json') + ' manifest.json\n', encoding='utf-8')
prereg = p.digest(experiment / 'protocol/preregistered.json')
anchors = dict(am_model_manifest_sha256=p.digest(model / 'manifest.json'),
am_experiment_manifest_sha256=p.digest(experiment / 'manifest.json'), am_final_sha256=p.digest(experiment / 'audit/final-status.json'),
am_preregistration_sha256=prereg, AI_publication_helper_sha256=p.AI_HELPER_SHA)
with patch.object(p, 'ROOT', root), patch.object(p, 'AM_PREREG_SHA', prereg), patch.object(p, 'load_module', return_value=helper):
self.assertEqual(set(p.verify_authority(anchors)), {'model', 'experiment'})
(model / 'payload.json').write_text('changed', encoding='utf-8')
with self.assertRaises(ValueError):
p.verify_authority(anchors)
def test_old_integrity_bridge_restores_import_path_and_rejects_failed_root(self):
import sys
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve(); expected = valid_status()[3]
before_path = list(sys.path)
def good():
sys.path.insert(0, 'synthetic-old-context-search-path')
return copy.deepcopy(expected)
module = types.SimpleNamespace(ROOT=root, integrity=good)
with patch.object(p, 'load_module', return_value=module):
self.assertEqual(p._old_integrity(root, 'a'*64, expected), expected)
self.assertEqual(sys.path, before_path)
bad = copy.deepcopy(expected); bad['root-1']['pass'] = False
module.integrity = lambda: bad
with patch.object(p, 'load_module', return_value=module), self.assertRaisesRegex(ValueError, '28-root'):
p._old_integrity(root, 'a'*64, expected)
self.assertEqual(sys.path, before_path)
def test_all_144_stage_completions_must_match_sealed_inventory(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve(); experiment = root / p.AM_EXP_REL; experiment.mkdir(parents=True)
rows, index, completed = planned(), {}, []
def add(relative, value):
path = experiment / relative; p.save(path, value)
index[relative] = {'path': relative, 'bytes': path.stat().st_size, 'sha256': p.digest(path)}
for row in rows:
prefix = 'gpu/zero/' + row['name'] + '-r' + str(row['repeat']) + '/'
folder = experiment / prefix.rstrip('/')
case = root / p.AK_REL / 'cases' / row['cohort'] / row['name']
add(prefix + 'summary.json', {'case': row['name'], 'complete': True, 'pass_all': True, 'all_finite': True,
'steps': 1000, 'completed_ticks': 1000})
add(prefix + 'comparison.json', {'pass_all': True, 'overlay_name': 'zero', 'case_dir': str(case), 'right': str(folder),
'mode': 'zero', 'left': str(root / p.AK_REL / 'gpu' / row['cohort'] / (row['name'] + '-r' + str(row['repeat']))),
'checks': {'synthetic_gate': True}, 'failed_checks': [], 'full_queue_reconstructed': True})
inventory = [dict(path=k[len(prefix):], bytes=v['bytes'], sha256=v['sha256']) for k, v in sorted(index.items()) if k.startswith(prefix)]
add(prefix + 'output-hashes.json', inventory)
completed.append(dict(name=row['name'], repeat=row['repeat'], output_hashes_sha256=index[prefix+'output-hashes.json']['sha256'],
comparison_sha256=index[prefix+'comparison.json']['sha256']))
receipt = {'stage': 'zero', 'pass_all': True, 'trials': 144, 'primary_trials': 120, 'technical_repeats': 24,
'training': False, 'completed': completed}
add('audit/zero-complete.json', receipt)
self.assertTrue(p.validate_stage(experiment, index, 'zero', rows, root)['pass_all'])
wrong = copy.deepcopy(rows); wrong[0], wrong[1] = wrong[1], wrong[0]
with self.assertRaisesRegex(ValueError, 'order/identity'):
p.validate_stage(experiment, index, 'zero', wrong, root)
# A missing sealed payload entry cannot be hidden by a PASS flag.
altered = dict(index); altered.pop('gpu/zero/fixed-0-r0/summary.json')
with self.assertRaises(ValueError):
p.validate_stage(experiment, altered, 'zero', rows, root)
class PreparationSafetyTests(unittest.TestCase):
def test_exclusive_layout_allows_only_nonexistent_or_empty_experiment(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()
model = root / 'models/male-cns-lif/olfactory-conditioning-lab-v1'; model.mkdir(parents=True)
experiment = root / 'experiments/phase4an-engineering-validation'; experiment.parent.mkdir()
self.assertEqual(p._layout(root, model, experiment)[2], experiment)
experiment.mkdir(); self.assertEqual(p._layout(root, model, experiment)[2], experiment)
(experiment / 'preserve.txt').write_text('partial', encoding='utf-8')
with self.assertRaisesRegex(ValueError, 'populated'):
p._layout(root, model, experiment)
self.assertEqual((experiment / 'preserve.txt').read_text(), 'partial')
def test_required_runtime_workflow_scope_sources_must_all_exist(self):
with tempfile.TemporaryDirectory() as directory:
model = Path(directory).resolve()
for name in p.SOURCE_FILES:
(model / name).write_text('# pure test source\n', encoding='utf-8')
result = p.source_inventory(model)
self.assertEqual(len(result), 21)
self.assertIn(str(model / 'engineering_workflow.py'), result)
self.assertIn(str(model / 'engineering_scopes.py'), result)
self.assertIn(str(model / 'engineering_analysis.py'), result)
(model / 'engineering_workflow.py').unlink()
with self.assertRaisesRegex(ValueError, 'Required engineering source'):
p.source_inventory(model)
def test_byte_budget_reserve_is_not_a_license_to_delete_partials(self):
with tempfile.TemporaryDirectory() as directory:
experiment = Path(directory).resolve() / 'new'
experiment.mkdir(); (experiment / 'partial').write_bytes(b'x')
with patch.object(p.shutil, 'disk_usage', return_value=types.SimpleNamespace(free=30 * 1024**3)):
self.assertEqual(p._budget(experiment)['used_bytes'], 1)
with patch.object(p.shutil, 'disk_usage', return_value=types.SimpleNamespace(free=5 * 1024**3)):
with self.assertRaisesRegex(ValueError, 'Insufficient'):
p._budget(experiment)
self.assertTrue((experiment / 'partial').is_file())
def test_row_mapping_rejects_unplanned_or_escaping_paths(self):
with tempfile.TemporaryDirectory() as directory:
exp = Path(directory).resolve(); case = exp / 'cases/safe-row'; case.mkdir(parents=True)
reg = {'plan': {'rows': [{'row_id': 'safe-row'}]}, 'cases': {'safe-row': {'case_dir': str(case)}}}
with patch.object(p, 'EXP', exp):
self.assertEqual(p.row_path(reg, 'safe-row'), case)
with self.assertRaises(ValueError):
p.row_path(reg, '../outside')
reg['cases']['safe-row']['case_dir'] = str(exp)
with self.assertRaises(ValueError):
p.row_path(reg, 'safe-row')
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,229 @@
"""Small synthetic CSR/record tests only; no brain, GPU or artifact writes."""
import ast
import copy
import hashlib
import json
from pathlib import Path
import struct
import unittest
import numpy as np
import engineering_queue_audit as audit
def digest(array):
return hashlib.sha256(np.ascontiguousarray(array).tobytes()).hexdigest()
def queue_row(tick, edges, accepted):
# Independent simple whole-payload construction, unlike the chunked audit.
edge_bytes = np.asarray(edges, dtype='<i8').tobytes()
mask_bytes = np.asarray(accepted, dtype=np.uint8).tobytes()
return dict(tick=tick, count=len(edges), accepted_count=int(sum(accepted)),
ordered_edge_sha256=hashlib.sha256(edge_bytes).hexdigest(),
accepted_mask_sha256=hashlib.sha256(mask_bytes).hexdigest(),
joint_sha256=hashlib.sha256(b'phase4af-queue-v1\0' + struct.pack('<qq', tick, len(edges)) + edge_bytes + mask_bytes).hexdigest())
def fixture(*, target_fire=True, weights=None, events=None, cuts=(5, 18, 19, 37)):
ptr = np.array([0, 2, 3, 4, 5], np.int64)
dst = np.array([2, 1, 2, 0, 2], np.int32)
edge_id = np.array([10, 11, 20, 30, 40], np.int64)
weights = np.array([.001, 0., -.002, .003, .004], np.float64) if weights is None else weights
total, n = 60, 4
if events is None:
events = [(0, 0), (0, 1), *(([(18, 2)]) if target_fire else []), (22, 0), (23, 1), (44, 0)]
events = sorted(events)
request_events = sorted(set(events + [(1, 0)]))
ticks, nodes = np.array([x[0] for x in events], np.int32), np.array([x[1] for x in events], np.int32)
requests = {'tick': np.array([x[0] for x in request_events], np.int32),
'neuron_idx': np.array([x[1] for x in request_events], np.int32)}
refs, expanded, masks, request_rows = [], [], [], []
last = np.full(n, -10000, np.int64)
for tick in range(total):
for when, neuron in request_events:
if when == tick:
accepted = int(tick - last[neuron] >= 22)
request_rows.append([tick, neuron, 1, accepted, 1 - accepted])
for when, neuron in events:
if when == tick:
last[neuron] = tick
ref = np.maximum(22 - (tick - last), 0).astype(np.int32)
refs.append(ref)
edges = [j for when, neuron in events if when == tick - 18 for j in range(int(ptr[neuron]), int(ptr[neuron + 1]))]
expanded.append(np.asarray(edges, np.int64))
masks.append((ref[dst[edges]] == 0).astype(np.uint8))
def record(start, stop):
selected = (ticks >= start) & (ticks < stop)
ev_t, ev_n = (ticks[selected] - start).copy(), nodes[selected].copy()
rows = np.array([row for row in request_rows if start <= row[0] < stop], np.int64).reshape(-1, 5)
rows[:, 0] -= start
return {'steps': stop - start, 'global_tick_offset': start,
'summary': {'pass_all': False}, # Never treated as authority.
'state': {'step': ev_t, 'neuron_idx': ev_n, 'counts': np.bincount(ev_n, minlength=n).astype(np.int32),
'v': np.zeros(n, np.float64), 'g': np.zeros(n, np.float64), 'ref': refs[stop - 1].copy()},
'requests': rows,
'queues': [queue_row(t - start, edge_id[expanded[t]], masks[t]) for t in range(start, stop)],
'state_hashes': [{'tick': t - start, 'v_sha256': digest(np.zeros(n, np.float64)),
'g_sha256': digest(np.zeros(n, np.float64)), 'ref_sha256': digest(refs[t])} for t in range(start, stop)]}
intervals = list(zip([0, *cuts], [*cuts, total]))
return {'mono': record(0, total), 'parts': [record(a, b) for a, b in intervals], 'ptr': ptr, 'dst': dst,
'edge_id': edge_id, 'effective_weights': weights, 'requests': requests, 'cuts': list(cuts)}
def run(data):
return audit.audit_queues(data['mono'], data['parts'], **{k: data[k] for k in
('ptr', 'dst', 'edge_id', 'effective_weights', 'requests', 'cuts')})
class QueueAudit(unittest.TestCase):
def test_real_edge_witness_and_local_headers_cross_18_19(self):
data = fixture()
result = run(data)
self.assertTrue(result['pass_all'], result['differences'])
witness = result['first_witness']
self.assertEqual((witness['boundary_tick'], witness['source_spike_tick'], witness['arrival_tick']), (37, 22, 40))
self.assertEqual((witness['source_neuron_idx'], witness['target_neuron_idx'], witness['global_edge_id']), (0, 2, 10))
self.assertEqual(witness['effective_weight_float64_le_hex'], np.asarray(.001, dtype='<f8').tobytes().hex())
self.assertFalse(result['membrane_dynamics_independently_replayed'])
self.assertFalse(result['v_g_hashes_independently_verified'])
self.assertEqual(result['rejected_requests'], 1)
# The tick44 source has arrival62, beyond this 60-tick record; no tail
# simulation or accidental drain is performed by the audit.
self.assertEqual(result['arriving_edges'], 7)
self.assertEqual(result['accepted_arriving_edges'], 2)
json.dumps(result, allow_nan=False)
self.assertNotEqual(data['mono']['queues'][18]['joint_sha256'], data['parts'][2]['queues'][0]['joint_sha256'])
def test_current_threshold_reset_rejects_arrival_and_available_target_accepts(self):
rejected = fixture(); accepted = fixture(target_fire=False)
self.assertEqual(rejected['mono']['queues'][18]['accepted_count'], 0)
self.assertEqual(accepted['mono']['queues'][18]['accepted_count'], 2)
self.assertTrue(run(rejected)['pass_all'])
witness = run(accepted)['first_witness']
self.assertEqual((witness['source_spike_tick'], witness['arrival_tick']), (0, 18))
def test_zero_weight_edges_stay_in_queue_but_cannot_supply_witness(self):
data = fixture(weights=np.zeros(5, np.float64))
result = run(data)
self.assertTrue(result['queue_ref_request_exact'])
self.assertGreater(result['arriving_edges'], 0)
self.assertFalse(result['pass_all'])
self.assertEqual(result['status'], 'INCOMPLETE-NO-ACCEPTED-NONZERO-CROSSING')
self.assertIsNone(result['first_witness'])
def test_target_sorted_or_reversed_queue_fails_even_when_both_outputs_agree(self):
data = fixture()
bad = queue_row(18, [20, 11, 10], [0, 0, 0])
data['mono']['queues'][18] = bad
data['parts'][2]['queues'][0] = queue_row(0, [20, 11, 10], [0, 0, 0])
result = run(data)
self.assertFalse(result['pass_all'])
self.assertTrue(any(d['field'] == 'queue.ordered_edge_sha256' for d in result['differences']))
def test_global_tick_in_segment_joint_header_fails(self):
data = fixture()
data['parts'][2]['queues'][0]['joint_sha256'] = data['mono']['queues'][18]['joint_sha256']
result = run(data)
self.assertEqual(result['difference_count'], 1)
self.assertEqual(result['differences'][0]['field'], 'queue.joint_sha256')
def test_fake_request_acceptance_and_ref_hash_fail_independently(self):
for mutate in ('request', 'hash', 'terminal'):
data = fixture()
if mutate == 'request':
data['parts'][0]['requests'][2, 3:] = [1, 0]
elif mutate == 'hash':
data['parts'][1]['state_hashes'][3]['ref_sha256'] = 'a' * 64
else:
data['parts'][0]['state']['ref'][0] = 0
with self.subTest(mutate=mutate):
self.assertFalse(run(data)['pass_all'])
def test_impossible_spikes_during_ref_fail_even_with_self_consistent_records(self):
data = fixture(events=[(0, 0), (1, 0), (23, 0)])
result = run(data)
self.assertFalse(result['pass_all'])
self.assertTrue(any(d['field'] == 'spike_during_refractory' for d in result['differences']))
def test_original_accepted_request_cannot_disappear_from_all_outputs(self):
data = fixture()
rows = sorted(zip(data['requests']['tick'].tolist(), data['requests']['neuron_idx'].tolist()))
rows.append((4, 3)); rows.sort()
data['requests'] = {'tick': np.array([r[0] for r in rows], np.int32),
'neuron_idx': np.array([r[1] for r in rows], np.int32)}
result = run(data)
self.assertFalse(result['pass_all'])
self.assertTrue(any(d['field'] == 'accepted_request_without_spike' for d in result['differences']))
def test_segment_events_cannot_cancel_or_substitute_nodes(self):
data = fixture()
data['parts'][0]['state']['neuron_idx'][1] = 3
data['parts'][0]['state']['counts'] = np.array([1, 0, 0, 1], np.int32)
result = run(data)
self.assertFalse(result['pass_all'])
self.assertTrue(any(d['field'] == 'events.neuron_idx' for d in result['differences']))
def test_missing_queue_or_wrong_cuts_fail_closed(self):
for mutate in ('missing', 'cuts', 'offset', 'dtype', 'nan'):
data = fixture()
if mutate == 'missing':
data['parts'][0]['queues'] = None
elif mutate == 'cuts':
data['cuts'][1] += 1
elif mutate == 'offset':
data['mono']['global_tick_offset'] = 1
elif mutate == 'dtype':
data['effective_weights'] = data['effective_weights'].astype(np.float32)
else:
data['effective_weights'][0] = np.nan
with self.subTest(mutate=mutate), self.assertRaises(ValueError):
run(data)
def test_negative_nonzero_edge_is_valid_witness_without_valence_claim(self):
data = fixture(target_fire=False)
data['effective_weights'][:2] = 0
result = run(data)
self.assertTrue(result['pass_all'])
self.assertEqual(result['first_witness']['global_edge_id'], 20)
self.assertLess(result['first_witness']['effective_weight_SI_volt'], 0)
self.assertFalse(result['unique_causal_or_necessity_claim'])
def test_input_arrays_are_not_modified(self):
data = fixture(); original = copy.deepcopy(data)
run(data)
for key in ('ptr', 'dst', 'edge_id', 'effective_weights'):
self.assertEqual(data[key].tobytes(), original[key].tobytes())
for key in data['requests']:
self.assertEqual(data['requests'][key].tobytes(), original['requests'][key].tobytes())
self.assertEqual(data['mono']['state']['ref'].tobytes(), original['mono']['state']['ref'].tobytes())
def test_frozen_AF_queue_definition_AST_crosscheck_and_chunk_order(self):
source = Path(__file__).resolve().parents[1] / 'author-fullgraph-lab-v1' / 'oracle_full.py'
raw = source.read_bytes()
self.assertEqual(hashlib.sha256(raw).hexdigest(), audit.AF_QUEUE_SOURCE_SHA256)
tree = ast.parse(raw)
definition = [n for n in tree.body if isinstance(n, ast.FunctionDef) and n.name == 'queue_summary']
self.assertEqual(len(definition), 1)
namespace = dict(np=np, hashlib=hashlib, struct=struct, BLOCK_BYTES=8 << 20, QUEUE_HEADER=b'phase4af-queue-v1\0')
exec(compile(ast.Module(body=definition, type_ignores=[]), str(source), 'exec'), namespace)
data = fixture()
saved = audit.CHUNK
try:
audit.CHUNK = 1 # Exercise edge pass then mask pass across boundaries.
for tick, local, sources, ref in [(18, 0, np.array([0, 1]), np.array([4, 4, 22, 0], np.int32)),
(40, 3, np.array([0]), np.zeros(4, np.int32)),
(0, 0, np.empty(0, np.int32), np.zeros(4, np.int32))]:
rows, _, _ = audit._queue(tick, local, sources, ref, data['ptr'], data['dst'], data['edge_id'], data['effective_weights'], [])
indices = np.array([j for s in sources for j in range(data['ptr'][s], data['ptr'][s + 1])], np.int64)
accepted = (ref[data['dst'][indices]] == 0).astype(np.uint8)
for at, row in zip((tick, local), rows):
self.assertEqual(row, namespace['queue_summary'](at, data['edge_id'][indices], accepted, chunk_bytes=8))
finally:
audit.CHUNK = saved
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,214 @@
"""Pure synthetic records, deliberately not a neural simulation or file test."""
import hashlib
import json
import unittest
import numpy as np
import engineering_scopes as s
def sha(array):
return hashlib.sha256(np.ascontiguousarray(array).tobytes()).hexdigest()
def record(start, stop, *, reference=True, blank_requests=False):
n, k, length = 3, 2, stop - start
ticks = np.array([0, 7, 19, 21, 41, 61], np.int32)
nodes = np.array([0, 1, 2, 0, 2, 1], np.int32)
selected = (ticks >= start) & (ticks < stop)
event_ticks, event_nodes = ticks[selected] - np.int32(start), nodes[selected].copy()
v = np.array([-.05, -.04, -.03], np.float64)
g = np.array([-0., float(stop - 1), -.001], np.float64)
ref = np.zeros(n, np.int32)
state = dict(step=event_ticks, neuron_idx=event_nodes, counts=np.bincount(event_nodes, minlength=n).astype(np.int32),
v=v, g=g, ref=ref)
raw_global = np.zeros((19, n), np.uint8)
for tick, node in zip(ticks, nodes):
if max(0, stop - 19) <= tick < stop:
raw_global[tick % 19, node] = 1
normalized = raw_global[(np.arange(19) + stop % 19) % 19]
raw_local = normalized[(np.arange(19) - length % 19) % 19].copy()
generated_all = np.repeat(np.arange(1, stop + 1, dtype=np.float32)[:, None], k, axis=1)
applied_all = np.concatenate((np.zeros((18, k), np.float32), generated_all))[:stop]
pending = np.zeros((18, k), np.float32)
kept = min(stop, 18); pending[-kept:] = generated_all[-kept:]
apl_state = np.full(2, np.float32(stop + 1), np.float32)
terminal = dict(spike_ring=raw_local, apl_state=apl_state, apl_pending=pending, next_tick=np.asarray(length, np.int64))
trace = dict(generated=generated_all[start:stop].copy(), applied=applied_all[start:stop].copy(),
drive=np.repeat(np.arange(start, stop, dtype=np.float32)[:, None], 2, axis=1),
state=np.repeat(np.arange(start + 2, stop + 2, dtype=np.float32)[:, None], 2, axis=1))
hashes, queues = [], []
for local, global_tick in enumerate(range(start, stop)):
hashes.append(dict(tick=local, v_sha256=sha(v), g_sha256=sha(np.array([-0., float(global_tick), -.001], np.float64)),
ref_sha256=sha(ref)))
edges, accepted = np.array([global_tick], np.int64), np.array([global_tick % 2], np.uint8)
queues.append(dict(tick=local, count=1, accepted_count=int(accepted.sum()), ordered_edge_sha256=sha(edges),
accepted_mask_sha256=sha(accepted), joint_sha256=hashlib.sha256(
str(local).encode() + edges.tobytes() + accepted.tobytes()).hexdigest()))
requests = np.array([[int(t), int(i), 1, 1, 0] for t, i in zip(event_ticks, event_nodes)], np.int64).reshape(-1, 5)
if blank_requests:
requests = np.empty((0, 5), np.int64)
summary = dict(complete=True, pass_all=True, all_finite=True, steps=length, completed_ticks=length, nodes=n,
precision='float64', apl_precision='float32', apl_delay_steps=18,
checkpoint_continuation_used=start > 0, global_tick_offset=start, global_next_tick=stop,
effective_weights_raw_sha256='e' * 64, diagnostic_mode='reference' if reference else 'light',
full_APL_history_recorded=reference, full_tick_state_hashes_recorded=reference,
full_queue_hashes_recorded=reference)
return dict(source='synthetic', state=state, terminal=terminal, requests=requests, steps=length,
global_tick_offset=start, summary=summary, traces=trace if reference else {},
state_hashes=hashes if reference else None, queues=queues if reference else None,
terminal_origin='synthetic_recorded')
class SliceTests(unittest.TestCase):
def test_reference_slice_all_observations_exact_and_inputs_unchanged(self):
candidate, whole = record(20, 44), record(0, 64)
before = candidate['terminal']['spike_ring'].tobytes(), whole['requests'].tobytes()
report = s.compare_slice(candidate, whole)
self.assertTrue(report['numerical_exact'], report['failed_checks'])
self.assertTrue(report['observed_scope']['candidate_per_tick_state_hashes'])
self.assertTrue(report['observed_scope']['candidate_per_tick_queue_fields'])
self.assertFalse(report['observed_scope']['queue_joint_hashes_compared'])
self.assertNotEqual(candidate['queues'][0]['joint_sha256'], whole['queues'][20]['joint_sha256'])
self.assertEqual(before, (candidate['terminal']['spike_ring'].tobytes(), whole['requests'].tobytes()))
json.dumps(report, allow_nan=False)
def test_light_uses_final_raw_hash_but_does_not_fabricate_optional_history(self):
report = s.compare_slice(record(20, 44, reference=False), record(0, 64))
self.assertTrue(report['numerical_exact'])
self.assertTrue(report['observed_scope']['final_state_against_reference_tick_hash'])
for name in ('candidate_APL_histories', 'candidate_per_tick_state_hashes', 'candidate_per_tick_queue_fields'):
self.assertFalse(report['observed_scope'][name])
self.assertNotIn('slice.per_tick_state_hashes', report['checks'])
self.assertTrue(any('unobserved' in item for item in report['limitations']))
def test_short_fresh_positive_zero_padding_and_prefix_events(self):
self.assertTrue(s.compare_slice(record(0, 5), record(0, 64))['numerical_exact'])
candidate = record(0, 5, reference=False)
candidate['terminal']['apl_pending'][0, 0] = np.float32(-0.)
report = s.compare_slice(candidate, record(0, 64))
self.assertFalse(report['checks']['slice.APL_pending'])
def test_blank_requests_are_valid_but_wrong_acceptance_is_not_equal(self):
self.assertTrue(s.compare_slice(record(20, 44, blank_requests=True), record(0, 64, blank_requests=True))['numerical_exact'])
candidate = record(20, 44)
candidate['requests'][0, 3:] = [0, 1]
report = s.compare_slice(candidate, record(0, 64))
self.assertFalse(report['checks']['slice.requests_five_fields'])
def test_wrong_count_or_final_hash_binding_is_invalid_not_a_pass(self):
for mutation in ('count', 'terminal', 'last_hash'):
candidate = record(20, 44)
if mutation == 'count': candidate['state']['counts'][0] += 1
elif mutation == 'terminal': candidate['state']['g'][1] += 1.
else: candidate['state_hashes'][-1]['g_sha256'] = '0' * 64
with self.subTest(mutation=mutation), self.assertRaises(ValueError):
s.compare_slice(candidate, record(0, 64))
def test_signed_zero_light_terminal_vs_reference_hash_is_detected(self):
candidate = record(20, 44, reference=False)
candidate['state']['g'][0] = 0.
report = s.compare_slice(candidate, record(0, 64))
self.assertFalse(report['checks']['slice.final_g_raw_hash'])
def test_interior_trace_state_hash_and_queue_mismatch_are_not_hidden(self):
candidate = record(20, 44)
candidate['traces']['drive'][1, 0] += np.float32(1.)
candidate['state_hashes'][1]['v_sha256'] = '0' * 64
candidate['queues'][1]['accepted_mask_sha256'] = '1' * 64
report = s.compare_slice(candidate, record(0, 64))
for key in ('slice.APL_trace.drive', 'slice.per_tick_state_hashes', 'slice.queue_order_and_acceptance'):
self.assertFalse(report['checks'][key])
def test_missing_reference_observations_and_invalid_bounds_fail_closed(self):
for mutation in ('light_mono', 'light_labeled_full_mono', 'mono_hashes', 'candidate_traces', 'partial_trace', 'out_of_bounds', 'nonzero_mono'):
candidate, mono = record(20, 44), record(0, 64)
if mutation == 'light_mono': mono = record(0, 64, reference=False)
elif mutation == 'light_labeled_full_mono': mono['summary']['diagnostic_mode'] = 'light'
elif mutation == 'mono_hashes': mono['state_hashes'] = None
elif mutation == 'candidate_traces': candidate['traces'] = {}
elif mutation == 'partial_trace': candidate['traces'].pop('drive')
elif mutation == 'out_of_bounds': candidate = record(60, 65)
else: mono = record(1, 65)
with self.subTest(mutation=mutation), self.assertRaises(ValueError):
s.compare_slice(candidate, mono)
def test_summary_type_scope_nonfinite_and_paths_rejected(self):
for mutation in ('bool', 'failed', 'offset', 'mode', 'weight', 'nan'):
candidate = record(20, 44)
if mutation == 'bool': candidate['summary']['steps'] = True
elif mutation == 'failed': candidate['summary']['pass_all'] = False
elif mutation == 'offset': candidate['summary']['global_tick_offset'] = 0
elif mutation == 'mode': candidate['summary']['diagnostic_mode'] = 'unknown'
elif mutation == 'weight': candidate['summary'].pop('effective_weights_raw_sha256')
else: candidate['state']['g'][0] = np.nan
with self.subTest(mutation=mutation), self.assertRaises(ValueError):
s.compare_slice(candidate, record(0, 64))
with self.assertRaises(ValueError): s.compare_slice('/never/open/this', record(0, 64))
class CheckpointTests(unittest.TestCase):
def test_local_phase_normalization_without_segment_cumulative_counts_confusion(self):
candidate, prefix = record(20, 64), record(0, 64)
self.assertFalse(s.ec.bit_equal(candidate['terminal']['spike_ring'], prefix['terminal']['spike_ring']))
self.assertFalse(s.ec.bit_equal(candidate['state']['counts'], prefix['state']['counts']))
report = s.compare_checkpoint(candidate, prefix)
self.assertTrue(report['numerical_exact'], report['failed_checks'])
self.assertFalse(report['observed_scope']['counts_compared'])
self.assertFalse(report['observed_scope']['per_tick_state_histories_compared'])
def test_signed_zero_different_stop_or_effective_weights_cannot_pass(self):
for mutation in ('zero', 'stop', 'weight'):
candidate, prefix = record(20, 64, reference=False), record(0, 64, reference=False)
if mutation == 'zero': candidate['state']['g'][0] = 0.
elif mutation == 'stop': prefix = record(0, 63, reference=False)
else: candidate['summary']['effective_weights_raw_sha256'] = 'f' * 64
with self.subTest(mutation=mutation):
self.assertFalse(s.compare_checkpoint(candidate, prefix)['numerical_exact'])
class ParentCarryTests(unittest.TestCase):
def test_short_parent_carry_observed_and_json_safe(self):
report = s.compare_parent_carry(record(20, 25), record(0, 20))
self.assertTrue(report['numerical_exact'], report['failed_checks'])
self.assertEqual(report['observed_scope']['unwritten_spike_slot_count'], 14)
self.assertEqual(report['observed_scope']['unconsumed_APL_row_count'], 13)
self.assertEqual(report['observed_scope']['applied_parent_APL_rows_compared'], 5)
json.dumps(report, allow_nan=False)
def test_unwritten_or_unconsumed_corruption_detected_even_if_terminal_is_self_consistent(self):
for mutation in ('ring', 'pending', 'applied'):
candidate = record(20, 25)
if mutation == 'ring': candidate['terminal']['spike_ring'][10, 0] ^= np.uint8(1)
elif mutation == 'pending': candidate['terminal']['apl_pending'][0, 0] += np.float32(1.)
else: candidate['traces']['applied'][0, 0] += np.float32(1.)
with self.subTest(mutation=mutation):
self.assertFalse(s.compare_parent_carry(candidate, record(0, 20))['numerical_exact'])
def test_light_missing_optional_applied_trace_is_not_fabricated(self):
report = s.compare_parent_carry(record(20, 25, reference=False), record(0, 20))
self.assertTrue(report['numerical_exact'])
self.assertFalse(report['observed_scope']['applied_parent_APL_prefix_checked'])
self.assertIsNone(report['observed_scope']['applied_parent_APL_rows_compared'])
self.assertNotIn('parent.applied_APL_prefix', report['checks'])
def test_long_segment_discloses_no_unwritten_slots_and_wrong_parent_clock_fails(self):
report = s.compare_parent_carry(record(20, 44), record(0, 20))
self.assertTrue(report['numerical_exact'])
self.assertFalse(report['observed_scope']['unwritten_spike_slots_checked'])
self.assertFalse(report['observed_scope']['unconsumed_APL_prefix_checked'])
self.assertEqual(report['observed_scope']['applied_parent_APL_rows_compared'], 18)
self.assertFalse(s.compare_parent_carry(record(20, 44), record(0, 19))['numerical_exact'])
with self.assertRaises(ValueError): s.compare_parent_carry(record(0, 5), record(0, 20))
def test_long_light_carry_scope_explicitly_has_no_observed_carry_array(self):
report = s.compare_parent_carry(record(20, 44, reference=False), record(0, 20))
self.assertTrue(report['numerical_exact'])
self.assertFalse(report['observed_scope']['any_carry_values_directly_compared'])
self.assertTrue(any('clock/shape/weight identity only' in line for line in report['limitations']))
self.assertFalse(report['complete_dynamical_equivalence_proven'])
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,207 @@
import copy
import hashlib
import json
from pathlib import Path
import tempfile
import types
import unittest
from unittest.mock import patch
import numpy as np
import engineering_workflow as w
from test_engineering_inputs import spec
def template(row):
return dict(schema='phase4an-engineering-case-template-v1', row_id=row['row_id'], original_spec=spec(),
start=row['start'], stop=row['stop'], budget_root='/project/new', output_budget_bytes=8 * 1024**3)
class Workflow(unittest.TestCase):
def test_only_exact_plan_row_and_safe_output_identity(self):
reg = {'plan': w.PLAN.build_plan()}
row = reg['plan']['rows'][0]
self.assertEqual(w.row_from(reg, row['row_id']), row)
for bad in ('unknown', '../escape', '/absolute'):
with self.assertRaises(ValueError): w.row_from(reg, bad)
for bad in ('../escape', '/absolute', 'a/b', 'a\\b', '', 'a:b'):
with self.assertRaises(ValueError): w.locations(bad)
reg['plan']['rows'][0]['steps'] = True
with self.assertRaises(ValueError): w.row_from(reg, row['row_id'])
def test_real_parent_is_required_only_for_later_segments(self):
for row in w.PLAN.build_plan()['rows']:
before = template(row)
pin = None if row['parent_row_id'] is None else 'a' * 64
derived, audit = w.derive_case(before, row, pin)
self.assertEqual(derived['steps'], row['steps'])
self.assertEqual(before, template(row))
self.assertEqual(audit['input_interval'], [row['start'], row['stop']])
with self.assertRaises(ValueError):
w.derive_case(before, row, 'a' * 64 if pin is None else None)
def test_forged_template_and_boolean_interval_rejected(self):
row = w.PLAN.build_plan()['rows'][0]
for key, value in (('schema', 'wrong'), ('row_id', 'wrong'), ('start', False), ('stop', 18)):
changed = template(row); changed[key] = value
with self.subTest(key=key), self.assertRaises(ValueError): w.derive_case(changed, row, None)
changed = template(row); changed['extra'] = True
with self.assertRaises(ValueError): w.derive_case(changed, row, None)
def test_json_comparison_does_not_conflate_booleans_and_numbers(self):
self.assertFalse(w.same_json({'n': True}, {'n': 1}))
self.assertFalse(w.same_json({'n': 1.}, {'n': 1}))
with self.assertRaises(ValueError): w.canonical({'n': float('nan')})
def test_actual_precision_flags_are_checked_not_merely_recorded(self):
component = dict(cuda_matmul_allow_tf32=False, cudnn_allow_tf32=True, deterministic_algorithms=False,
cudnn_deterministic=False, cudnn_benchmark=False, float32_matmul_precision='highest',
CUBLAS_WORKSPACE_CONFIG=None, device_name='NVIDIA GeForce RTX 4090', torch='2.test', cuda='13.0',
numpy='2.test', precision_configuration_changed_by_component=False)
recorded = dict(gpu_uuid=w.GPU, settings_changed_by_environment_recorder=False,
matmul_allow_tf32=False, cudnn_allow_tf32=True, deterministic_algorithms=False,
cudnn_deterministic=False, cudnn_benchmark=False, float32_matmul_precision='highest',
CUBLAS_WORKSPACE_CONFIG=None, device_name='NVIDIA GeForce RTX 4090', torch_version='2.test',
torch_cuda_version='13.0', numpy_version='2.test')
w.verify_environment(recorded, component)
for key, value in [('cuda_matmul_allow_tf32', True), ('float32_matmul_precision', 'high'),
('precision_configuration_changed_by_component', True), ('cudnn_allow_tf32', False)]:
altered = dict(component, **{key: value})
with self.subTest(key=key), self.assertRaises(ValueError): w.verify_environment(recorded, altered)
with self.assertRaises(ValueError): w.verify_environment(dict(recorded, matmul_allow_tf32=0), component)
def test_npz_real_compression_measured_not_assumed(self):
with tempfile.TemporaryDirectory() as name:
root = Path(name)
with (root / 'compressed.npz').open('xb') as stream:
np.savez_compressed(stream, v=np.zeros(50, np.float64))
measured = w.npz_storage(root / 'compressed.npz')
self.assertGreater(measured['uncompressed_entry_bytes'], measured['compressed_entry_bytes'])
self.assertEqual(measured['entries'][0]['compression'], 8)
with (root / 'stored.npz').open('xb') as stream:
np.savez(stream, v=np.zeros(50, np.float64))
with self.assertRaises(ValueError): w.npz_storage(root / 'stored.npz')
def test_resource_limit_preserves_existing_payload(self):
with tempfile.TemporaryDirectory() as name, patch.object(w, 'EXP', Path(name)):
path = Path(name) / 'payload.bin'
path.write_bytes(b'not deleted')
reg = {'budget': {'output_budget_bytes': 1, 'minimum_free_reserve_bytes': 1}}
with self.assertRaises(ValueError): w.disk_budget(reg)
self.assertEqual(path.read_bytes(), b'not deleted')
def _temporary_gate(self, root):
for name in ('audit', 'logs', 'receipts', 'runs', 'cases'):
(root / name).mkdir()
reg = {'plan': w.PLAN.build_plan(), 'budget': {'output_budget_bytes': 1 << 20,
'minimum_free_reserve_bytes': 1}}
return reg, types.SimpleNamespace(read_registration=lambda sha: copy.deepcopy(reg))
def test_controller_refuses_partial_without_rerun_or_deletion(self):
with tempfile.TemporaryDirectory() as name, patch.object(w, 'EXP', Path(name)):
root = Path(name); reg, prep = self._temporary_gate(root)
paths = w.locations(reg['plan']['rows'][0]['row_id'])
paths['stdout'].write_bytes(b'original evidence')
with (patch.object(w, 'preparation', return_value=prep), patch.object(w, 'preflight') as pre,
patch.object(w.subprocess, 'run') as runner):
with self.assertRaisesRegex(ValueError, 'Partial row'):
w.run_registered('a' * 64)
runner.assert_not_called(); pre.assert_not_called()
self.assertEqual(paths['stdout'].read_bytes(), b'original evidence')
self.assertFalse(paths['execution'].exists())
self.assertTrue((root / 'audit/stopped.json').exists())
def test_nonzero_worker_exit_cannot_become_completed_receipt(self):
with tempfile.TemporaryDirectory() as name, patch.object(w, 'EXP', Path(name)):
root = Path(name); reg, prep = self._temporary_gate(root)
paths = w.locations(reg['plan']['rows'][0]['row_id'])
failure = types.SimpleNamespace(returncode=1)
with (patch.object(w, 'preparation', return_value=prep), patch.object(w, 'preflight'),
patch.object(w.subprocess, 'run', return_value=failure) as runner):
with self.assertRaisesRegex(ValueError, 'Worker failed'):
w.run_registered('a' * 64)
self.assertEqual(runner.call_count, 1)
self.assertTrue(paths['stdout'].exists() and paths['stderr'].exists())
self.assertFalse(paths['execution'].exists())
self.assertTrue((root / 'audit/stopped.json').exists())
def test_stopped_inventory_cannot_auto_resume(self):
with tempfile.TemporaryDirectory() as name, patch.object(w, 'EXP', Path(name)):
root = Path(name); _, prep = self._temporary_gate(root)
(root / 'audit/stopped.json').write_text('{}')
with patch.object(w, 'preparation', return_value=prep), patch.object(w.subprocess, 'run') as runner:
with self.assertRaisesRegex(ValueError, 'Stopped/completed'):
w.run_registered('a' * 64)
runner.assert_not_called()
def _completion(self, reg, root):
entries = []
for row in reg['plan']['rows']:
path = w.locations(row['row_id'])['execution']
path.write_bytes(row['row_id'].encode())
entries.append(dict(row_id=row['row_id'], execution_sha256=w.R.digest(path), reverified_existing=False))
return dict(complete=True, registration_sha256='a' * 64, expected_calls=32, executed_ticks=11074,
training=False, learning_pass_claimed=False, completed=entries)
def test_inventory_rechecks_every_actual_receipt(self):
with tempfile.TemporaryDirectory() as name, patch.object(w, 'EXP', Path(name)):
root = Path(name); reg, _ = self._temporary_gate(root)
complete = self._completion(reg, root)
w.R._save(root / 'audit/execution-complete.json', complete)
with patch.object(w, 'verify_execution', side_effect=lambda r, s, row: dict(row=row)) as verify:
records = w.verified_inventory(reg, 'a' * 64)
self.assertEqual(verify.call_count, 32)
self.assertEqual(list(records), [row['row_id'] for row in reg['plan']['rows']])
def test_inventory_rejects_reorder_missing_and_changed_receipt(self):
for corruption in ('reorder', 'missing', 'changed', 'claimed_training', 'wrong_ticks'):
with self.subTest(corruption=corruption), tempfile.TemporaryDirectory() as name, patch.object(w, 'EXP', Path(name)):
root = Path(name); reg, _ = self._temporary_gate(root)
complete = self._completion(reg, root)
if corruption == 'reorder': complete['completed'].reverse()
if corruption == 'missing': complete['completed'].pop()
if corruption == 'changed': w.locations(reg['plan']['rows'][0]['row_id'])['execution'].write_bytes(b'changed')
if corruption == 'claimed_training': complete['training'] = True
if corruption == 'wrong_ticks': complete['executed_ticks'] += 1
w.R._save(root / 'audit/execution-complete.json', complete)
with patch.object(w, 'verify_execution'), self.assertRaises(ValueError):
w.verified_inventory(reg, 'a' * 64)
def test_seal_contains_nested_manifests_and_self_checksum(self):
with tempfile.TemporaryDirectory() as name, patch.object(w, 'EXP', Path(name)):
root = Path(name)
(root / 'runs' / 'one').mkdir(parents=True)
(root / 'runs/one/manifest.json').write_bytes(b'nested evidence')
(root / 'payload').write_bytes(b'payload')
result = w.seal_experiment()
manifest = json.loads((root / 'manifest.json').read_text())
self.assertEqual([row['path'] for row in manifest['files']], ['payload', 'runs/one/manifest.json'])
lines = (root / 'checksums.sha256').read_text().splitlines()
self.assertIn(result['manifest_sha256'] + ' manifest.json', lines)
self.assertEqual(len(lines), 3)
self.assertEqual(result['checksums_sha256'], hashlib.sha256((root / 'checksums.sha256').read_bytes()).hexdigest())
before = (root / 'manifest.json').read_bytes()
with self.assertRaises(ValueError): w.seal_experiment()
self.assertEqual((root / 'manifest.json').read_bytes(), before)
def test_seal_rejects_bytecode_instead_of_silently_losing_it(self):
with tempfile.TemporaryDirectory() as name, patch.object(w, 'EXP', Path(name)):
root = Path(name); (root / '__pycache__').mkdir()
(root / '__pycache__/a.pyc').write_bytes(b'cache')
with self.assertRaises(ValueError): w.seal_experiment()
self.assertFalse((root / 'manifest.json').exists())
def test_finalize_rejects_recomputed_difference_before_sealing(self):
with tempfile.TemporaryDirectory() as name, patch.object(w, 'EXP', Path(name)):
root = Path(name); _, prep = self._temporary_gate(root)
w.R._save(root / 'audit/engineering-analysis.json', dict(status='saved'))
with patch.object(w, 'evaluate', return_value=({}, {'status': 'different'})), patch.object(w, 'seal_experiment') as seal:
with self.assertRaisesRegex(ValueError, 'Recomputed analysis'):
w.finalize('a' * 64)
seal.assert_not_called()
self.assertFalse((root / 'audit/final-status.json').exists())
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,195 @@
"""Synthetic planning tests only; no input generation, files or brain runs."""
import copy
from collections import Counter
import unittest
from unittest.mock import patch
import evaluation_plan as m
import trial_plan
def synthetic_config(trial, reversal_points=(0, 50)):
"""An explicit TEST choice, not a proposed production default.
Controls here use a simple fixed epoch fraction; the planner never supplies
this policy. Only paired/reversal checkpoints track actual scheduled USs.
"""
config = dict(schema=m.CONFIG_SCHEMA, probe_steps=1200, cs_start_tick=0,
cs_stop_tick_exclusive=1000, reversal_checkpoints=list(reversal_points),
baseline_execution="execute_each_snapshot", snapshot_binding_policy="explicit_completed_epochs",
snapshot_bindings=[])
for block in trial["plan"]["blocks"]:
for suite in block["suites"]:
points = list(reversal_points) if suite["assay"] == "reversal" else block["evaluation_checkpoints"]
for arm in suite["arms"]:
for point in points:
if point in (0, 50):
completed = 0 if point == 0 else len(arm["epochs"])
elif arm["arm"] in ("paired", "reversal", "no_reversal"):
rewarded = [i + 1 for i, e in enumerate(arm["epochs"]) if e["us"]["requested"]]
completed = rewarded[point - 1]
else:
completed = point * len(arm["epochs"]) // 50
config["snapshot_bindings"].append(dict(block_id=block["block_id"], assay=suite["assay"],
arm=arm["arm"], checkpoint=point, completed_epochs=completed))
return config
class EvaluationPlan(unittest.TestCase):
@classmethod
def setUpClass(cls):
# Existing synthetic master from the trial-plan tests, not a real choice.
cls.trial = trial_plan.build_trial_plan(461)
cls.config = synthetic_config(cls.trial)
cls.plan = m.build_evaluation_plan(cls.trial, cls.config)
def test_public_full_authority_reconstruction(self):
self.assertTrue(m.validate_evaluation_plan(self.plan, self.trial, self.config))
def test_exact_inventory_budget_and_no_execution_claim(self):
b = self.plan["budget"]
self.assertEqual(b["probes_by_assay"], dict(normal=3920, delayed=3920, reversal=320, retention=480))
self.assertEqual(b["evaluation_probes"], 8640)
self.assertEqual(b["snapshot_count"], 4320)
self.assertEqual(b["evaluation_steps"], 10368000)
self.assertEqual(b["planned_scientific_steps_including_evaluation"], 117968000)
self.assertEqual(b["independent_evaluation_inputs"], 80)
for name in ("execution_authorized", "training_executed", "inputs_generated", "candidate_generated",
"learning_pass_claimed", "scientific_gate_added", "external_source_and_snapshot_authority_verified"):
self.assertIs(self.plan[name], False)
def test_full_cells_and_all_control_coverage(self):
cells = {}
for p in self.plan["probes"]:
cells.setdefault((p["mode"], p["dan_type"], p["original_reinforced_cue"]), set()).add(p["training_replicate"])
self.assertEqual(p["odor"], trial_plan.ODORS[p["cue"]])
self.assertFalse(p["weight_updates_enabled"])
self.assertFalse(p["US_requested"])
self.assertFalse(p["feedback_to_training_or_idle"])
self.assertFalse(p["adds_training_replicate"])
self.assertEqual(len(cells), 8)
self.assertTrue(all(v == {1, 2, 3, 4, 5} for v in cells.values()))
first = self.trial["plan"]["blocks"][0]["block_id"]
normal = [p for p in self.plan["probes"] if p["block_id"] == first and p["assay"] == "normal"]
self.assertEqual(Counter(p["arm"] for p in normal), Counter({arm: 14 for arm in trial_plan.ARMS}))
def test_stream_keys_disjoint_and_reused_as_paired_probes(self):
non_eval = {r["seed"] for r in self.trial["seed_manifest"]["independent_streams"] if r["split"] != "evaluation"}
grouped = {}
for p in self.plan["probes"]:
grouped.setdefault((p["block_id"], p["cue"]), set()).add((p["input"]["seed"], p["input"]["stream_id"]))
self.assertEqual(p["input"]["semantic_key"], [p["block_id"], "evaluation", "CS", p["cue"], "00", "base-probe"])
self.assertEqual(p["input"]["seed_bits"], 63)
seeds = {next(iter(values))[0] for values in grouped.values()}
self.assertEqual(len(grouped), 80)
self.assertTrue(all(len(v) == 1 for v in grouped.values()))
self.assertEqual(len(seeds), 80)
self.assertFalse(seeds & non_eval)
self.assertTrue(any(seed > 2**32 for seed in seeds))
def test_reversal_original_baseline_and_orientation(self):
for s in self.plan["snapshots"]:
if s["assay"] == "reversal":
self.assertEqual(s["baseline_snapshot_id"], s["block_id"] + "/normal/paired/checkpoint-00")
if s["checkpoint"] == 0:
self.assertEqual(s["delta_source"], {"kind": "fork", "source": s["block_id"] + "/normal/paired/terminal-delta"})
self.assertEqual(s["neural_state_source"]["kind"], "common_frozen_reset")
def test_retention_same_idle_lineage_independent_forks(self):
grouped = {}
for s in self.plan["snapshots"]:
if s["assay"] != "retention":
continue
src = s["neural_state_source"]
grouped.setdefault((s["block_id"], s["arm"]), []).append(src)
self.assertFalse(src["reset_or_analytic_time_advance"])
self.assertFalse(s["snapshot_mutated_by_planning"])
self.assertEqual(s["baseline_snapshot_id"], s["block_id"] + "/normal/" + s["arm"] + "/checkpoint-00")
self.assertEqual(len(grouped), 80)
for rows in grouped.values():
self.assertEqual([r["elapsed_ticks"] for r in rows], [0, 10000, 100000])
self.assertEqual(len({r["idle_lineage_id"] for r in rows}), 1)
self.assertEqual(rows[0]["kind"], "actual_terminal_checkpoint")
self.assertEqual(rows[1]["kind"], "continuous_idle_checkpoint")
def test_missing_choices_no_hidden_default(self):
for key in m.CONFIG_FIELDS:
bad = copy.deepcopy(self.config)
del bad[key]
with self.subTest(key=key), self.assertRaisesRegex(ValueError, "missing="):
m.build_evaluation_plan(self.trial, bad)
self.assertIn("snapshot_bindings", m.required_choices())
def test_bad_configuration_types_and_unimplemented_reuse(self):
for key, value in (("probe_steps", 1200.0), ("cs_start_tick", False),
("cs_stop_tick_exclusive", True), ("cs_stop_tick_exclusive", 1201),
("reversal_checkpoints", [False, 50]), ("reversal_checkpoints", [50]),
("baseline_execution", "reuse_identical"), ("snapshot_binding_policy", "guess_from_paired")):
bad = copy.deepcopy(self.config)
bad[key] = value
with self.subTest(key=key, value=value), self.assertRaises(ValueError):
m._assemble(self.trial, bad)
def test_binding_coverage_order_types_and_pairing_boundary(self):
changes = (
lambda rows: rows.pop(), lambda rows: rows.reverse(),
lambda rows: rows[0].update(checkpoint=False),
lambda rows: rows[0].update(completed_epochs=False),
lambda rows: rows[1].update(completed_epochs=0),
lambda rows: rows[6].update(completed_epochs=99),
lambda rows: rows[1].update(completed_epochs=100),
lambda rows: rows[0].update(extra="unknown"),
)
for index, change in enumerate(changes):
bad = copy.deepcopy(self.config)
change(bad["snapshot_bindings"])
with self.subTest(index=index), self.assertRaises(ValueError):
m._assemble(self.trial, bad)
def test_explicit_alternative_curve_changes_budget_not_seed_inventory(self):
config = synthetic_config(self.trial, trial_plan.PROBE_CHECKPOINTS)
result = m._assemble(self.trial, config)
self.assertEqual(result["budget"]["evaluation_probes"], 9440)
self.assertEqual(result["budget"]["probes_by_assay"]["reversal"], 1120)
self.assertEqual(result["budget"]["independent_evaluation_inputs"], 80)
config["cs_stop_tick_exclusive"] = 1200
alternate = m._assemble(self.trial, config)
self.assertNotEqual(result["config_sha256"], alternate["config_sha256"])
self.assertTrue(all(p["cs_stop_tick_exclusive"] == 1200 for p in alternate["probes"]))
def test_payload_mutations_rejected_even_when_self_hash_updated(self):
changes = (
lambda p: p["probes"].pop(), lambda p: p["probes"].reverse(),
lambda p: p["probes"][0]["input"].update(seed=1),
lambda p: p["probes"][0].update(training_replicate=True),
lambda p: p["probes"][0].update(adds_training_replicate=True),
lambda p: p.update(execution_authorized=True),
)
# Full trial authority was validated in setUpClass and again above.
# Isolate evaluation reconstruction here, avoiding six repeated rebuilds
# of the large, unchanged 68k-epoch synthetic training schedule.
with patch.object(m.trial_plan, "validate_trial_plan", return_value=True):
for index, change in enumerate(changes):
bad = copy.deepcopy(self.plan)
change(bad)
bad["payload_sha256"] = m._sha({k: v for k, v in bad.items() if k != "payload_sha256"})
with self.subTest(index=index), self.assertRaises(ValueError):
m.validate_evaluation_plan(bad, self.trial, self.config)
def test_parent_authority_failure_propagates(self):
with patch.object(m.trial_plan, "validate_trial_plan", side_effect=ValueError("invalid full parent")):
with self.assertRaisesRegex(ValueError, "invalid full parent"):
m.build_evaluation_plan(self.trial, self.config)
def test_no_mutable_alias_to_caller(self):
result = m._assemble(self.trial, self.config)
result["config"]["snapshot_bindings"][0]["completed_epochs"] = 900
result["snapshots"][0]["delta_source"]["kind"] = "corruption"
result["probes"][0]["input"]["semantic_key"][0] = "corruption"
self.assertEqual(self.config["snapshot_bindings"][0]["completed_epochs"], 0)
self.assertEqual(self.trial["plan"]["blocks"][0]["suites"][0]["arms"][0]["delta_start"]["kind"], "zero")
self.assertEqual(m._assemble(self.trial, self.config), self.plan)
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,262 @@
"""Synthetic plans only; no neural input generation, files, remote or execution."""
import copy
import unittest
from unittest import mock
import evaluation_plan
import evaluation_policy as policy
import trial_plan
class EvaluationPolicyTests(unittest.TestCase):
@classmethod
def setUpClass(cls):
# Explicit synthetic fixture seed, never a proposed production master.
cls.trial = trial_plan.build_trial_plan(461)
cls.result = policy.build_evaluation_policy(cls.trial)
cls.registry = policy._registry_index(cls.trial)
cls.rows = cls.result["audit"]["bindings"]
def suite(self, assay="normal"):
block = self.trial["plan"]["blocks"][0]
suite = next(s for s in block["suites"] if s["assay"] == assay)
return block, copy.deepcopy(suite)
def check_suite(self, block, suite, registry=None):
return policy._suite_policy(block, suite, self.trial["plan"]["master"],
self.registry if registry is None else registry)
def test_fixed_config_and_complete_consumer_inventory(self):
config = self.result["config"]
self.assertEqual(set(config), evaluation_plan.CONFIG_FIELDS)
self.assertEqual(config["probe_steps"], 1200)
self.assertEqual(config["cs_start_tick"], 0)
self.assertEqual(config["cs_stop_tick_exclusive"], 1000)
self.assertEqual(config["reversal_checkpoints"], [0, 1, 2, 5, 10, 20, 50])
self.assertEqual(config["baseline_execution"], "execute_each_snapshot")
self.assertEqual(len(config["snapshot_bindings"]), 4480)
# The original public consumer performs its own full authority check.
inventory = evaluation_plan.build_evaluation_plan(self.trial, config)
self.assertEqual(inventory["budget"]["evaluation_probes"], 9440)
self.assertEqual(inventory["budget"]["snapshot_count"], 4720)
self.assertEqual(inventory["budget"]["evaluation_steps"], 11328000)
self.assertEqual(inventory["budget"]["planned_scientific_steps_including_evaluation"], 118928000)
self.assertEqual(inventory["budget"]["probes_by_assay"],
{"normal": 3920, "delayed": 3920, "reversal": 1120, "retention": 480})
self.assertFalse(inventory["execution_authorized"])
self.assertFalse(inventory["budget"]["baseline_output_deduplication"])
def test_each_binding_audited_in_exact_order(self):
bindings = self.result["config"]["snapshot_bindings"]
self.assertEqual([r["binding"] for r in self.rows], bindings)
identities = [(r["block_id"], r["assay"], r["arm"], r["checkpoint"]) for r in bindings]
self.assertEqual(len(identities), len(set(identities)))
self.assertEqual({r["checkpoint"] for r in bindings if r["assay"] == "reversal"}, set(policy.POINTS))
self.assertEqual(self.result["audit"]["coverage"]["snapshots_by_assay"],
{"normal": 1960, "delayed": 1960, "reversal": 560})
def test_five_independent_replicates_not_relabelled_from_probes(self):
coverage = self.result["audit"]["coverage"]
self.assertEqual(coverage["training_blocks"], 40)
self.assertEqual(coverage["training_replicates_per_cell"], 5)
self.assertEqual(coverage["new_random_streams"], 0)
self.assertFalse(coverage["new_master_seed_selected"])
self.assertEqual(self.result["seed_manifest_sha256"], self.trial["seed_manifest"]["payload_sha256"])
def test_normal_and_delayed_controls_match_actual_ordered_source_prefixes(self):
selected = [r for r in self.rows if r["binding"]["assay"] in ("normal", "delayed") and
r["binding"]["arm"] != "dan_only"]
self.assertEqual(len(selected), 3360)
for row in selected:
self.assertIs(row["ordered_CS_prefix_exact"], True)
self.assertEqual(row["actual"]["ordered_CS_source_identity_sha256"],
row["reference"]["ordered_CS_source_identity_sha256"])
self.assertEqual(row["actual"]["CS_counts_by_cue"], row["reference"]["CS_counts_by_cue"])
if row["binding"]["arm"] != "unpaired":
self.assertEqual(row["binding"]["completed_epochs"], row["reference_completed_epochs"])
self.assertTrue(row["elapsed_ticks_equal_to_reference"])
def test_unpaired_earliest_matching_prefix_and_full_terminal_including_tail(self):
any_earlier_US = False
any_trailing_US = False
for block in self.trial["plan"]["blocks"]:
for suite in block["suites"][:2]:
arms = {a["arm"]: a for a in suite["arms"]}
paired, unpaired = arms["paired"], arms["unpaired"]
rows = [r for r in self.rows if r["binding"]["block_id"] == block["block_id"] and
r["binding"]["assay"] == suite["assay"] and r["binding"]["arm"] == "unpaired"]
for row in rows:
point = row["binding"]["checkpoint"]
count = row["binding"]["completed_epochs"]
needed = row["reference_completed_epochs"]
if point == 0:
self.assertEqual(count, 0)
elif point == 50:
self.assertEqual(count, 150)
self.assertEqual(row["actual"]["CS_epochs"], 100)
self.assertEqual(row["actual"]["requested_US_epochs"], 50)
self.assertEqual(row["elapsed_ticks_difference"], 50 * paired["epochs"][0]["steps"])
any_trailing_US |= unpaired["epochs"][-1]["cs"] is None
else:
actual_positions = [i + 1 for i, e in enumerate(unpaired["epochs"]) if e["cs"] is not None]
self.assertEqual(count, actual_positions[needed - 1])
self.assertIsNotNone(unpaired["epochs"][count - 1]["cs"])
self.assertEqual(row["actual"]["requested_US_epochs"], count - needed)
any_earlier_US |= count > needed
self.assertTrue(any_earlier_US)
self.assertTrue(any_trailing_US)
def test_DAN_only_is_yoked_zero_CS_not_equal_exposure(self):
rows = [r for r in self.rows if r["binding"]["arm"] == "dan_only"]
self.assertEqual(len(rows), 560)
for row in rows:
self.assertEqual(row["actual"]["CS_epochs"], 0)
self.assertIsNone(row["ordered_CS_prefix_exact"])
self.assertFalse(row["CS_exposure_equal"])
self.assertTrue(row["DAN_only_yoke_verified"])
self.assertTrue(row["requested_US_count_equal_to_reference"])
self.assertTrue(row["elapsed_ticks_equal_to_reference"])
self.assertEqual(row["binding"]["completed_epochs"], row["reference_completed_epochs"])
def test_reversal_each_own_US_boundary_and_original_terminal(self):
saw_different_middle_CS_prefix = False
for block in self.trial["plan"]["blocks"]:
suite = block["suites"][2]
arms = {a["arm"]: a for a in suite["arms"]}
for row in [r for r in self.rows if r["binding"]["block_id"] == block["block_id"] and
r["binding"]["assay"] == "reversal"]:
binding = row["binding"]
arm = arms[binding["arm"]]
point, count = binding["checkpoint"], binding["completed_epochs"]
if point in (0, 50):
self.assertEqual(count, 0 if point == 0 else 100)
self.assertTrue(row["ordered_CS_prefix_exact"])
else:
ordinal_positions = [i + 1 for i, e in enumerate(arm["epochs"]) if e["us"]["requested"]]
self.assertEqual(count, ordinal_positions[point - 1])
self.assertTrue(arm["epochs"][count - 1]["us"]["requested"])
saw_different_middle_CS_prefix |= not row["ordered_CS_prefix_exact"]
self.assertEqual(row["actual"]["requested_US_epochs"], point)
self.assertEqual(row["actual"]["enabled_nonzero_D_update_opportunities"], point)
self.assertTrue(saw_different_middle_CS_prefix)
def test_intermediate_US_time_and_update_mismatches_are_explicit_not_failed(self):
middle = [r for r in self.rows if r["binding"]["checkpoint"] not in (0, 50)]
self.assertTrue(any(r["binding"]["arm"] == "random_reward" and r["requested_US_count_difference"] != 0 for r in middle))
self.assertTrue(any(r["binding"]["arm"] == "unpaired" and r["elapsed_ticks_difference"] > 0 for r in middle))
self.assertTrue(any(r["binding"]["arm"] == "frozen" and r["nonzero_D_update_opportunity_difference"] < 0 for r in middle))
self.assertTrue(all(r["all_US_timing_or_total_exposure_matched_claimed"] is False for r in self.rows))
self.assertTrue(all(r["actual"]["actual_accepted_US_or_DAN_spikes_known"] is False for r in self.rows))
def test_all_boundaries_follow_complete_epoch_update_not_US_tick(self):
for row in self.rows:
self.assertEqual(row["boundary"], "after complete epoch and scheduled delta update; not at the US tick")
steps = 1800 if row["binding"]["assay"] == "delayed" else 1200
self.assertEqual(row["actual"]["elapsed_training_ticks"], row["binding"]["completed_epochs"] * steps)
self.assertEqual(row["terminal_includes_all_epochs"], row["binding"]["checkpoint"] == 50)
def test_control_seed_or_source_alias_mutations_fail_without_relying_on_counts(self):
for field, change in (("seed", lambda v: v + 1),
("source_stream_id", lambda v: v + "x"),
("stream_id", lambda v: v + "x"),
("start_tick", lambda v: False)):
with self.subTest(field=field):
block, suite = self.suite()
cs = suite["arms"][1]["epochs"][0]["cs"]
cs[field] = change(cs[field])
with self.assertRaises(ValueError):
self.check_suite(block, suite)
def test_reordered_cues_with_same_counts_and_valid_individual_aliases_fail(self):
block, suite = self.suite()
epochs = suite["arms"][1]["epochs"]
index = next(i for i in range(1, len(epochs)) if epochs[i]["cs"]["cue"] != epochs[0]["cs"]["cue"])
epochs[0]["cs"], epochs[index]["cs"] = epochs[index]["cs"], epochs[0]["cs"]
with self.assertRaisesRegex(ValueError, "CS occurrence order|ordered CS|CS sequence"):
self.check_suite(block, suite)
def test_corrupt_alias_registry_is_not_excused_by_equal_CS_seed(self):
block, suite = self.suite()
independent, aliases = self.registry
changed = copy.deepcopy(aliases)
cs = suite["arms"][1]["epochs"][0]["cs"]
key = (block["block_id"], "normal", "CS", cs["cue"], f'{cs["occurrence"]:02d}', "d_zero")
changed[("training", key)]["source_key"][-1] = "frozen"
with self.assertRaisesRegex(ValueError, "alias is not registered"):
self.check_suite(block, suite, (independent, changed))
def test_DAN_yoked_cue_US_or_added_CS_mutations_fail(self):
for mutation in ("cue", "US", "CS"):
with self.subTest(mutation=mutation):
block, suite = self.suite()
paired = suite["arms"][0]
dan = suite["arms"][-1]
epoch = dan["epochs"][0]
if mutation == "cue":
epoch["yoked_cue_removed"] = "B" if epoch["yoked_cue_removed"] == "A" else "A"
elif mutation == "US":
epoch["us"]["nominal_slot_tick"] += 1
else:
epoch["cs"] = copy.deepcopy(paired["epochs"][0]["cs"])
with self.assertRaises(ValueError):
self.check_suite(block, suite)
def test_missing_reordered_or_duplicate_control_arms_fail(self):
for mutation in ("missing", "reorder", "duplicate"):
with self.subTest(mutation=mutation):
block, suite = self.suite()
if mutation == "missing":
suite["arms"].pop()
elif mutation == "reorder":
suite["arms"][0], suite["arms"][1] = suite["arms"][1], suite["arms"][0]
else:
suite["arms"].append(suite["arms"][0])
with self.assertRaisesRegex(ValueError, "control arms"):
self.check_suite(block, suite)
def test_result_not_aliased_no_execution_or_new_scientific_gate(self):
fields = ("protocol_approval_claimed", "execution_authorized", "training_executed", "inputs_generated",
"input_generator_implementation_verified", "source_and_snapshot_authority_verified",
"candidate_generated", "learning_pass_claimed", "scientific_gate_added")
self.assertTrue(all(self.result[field] is False for field in fields))
self.assertEqual(self.result["status"], "DRAFT-NOT-REGISTERED")
self.assertTrue(self.result["choices_are_new_draft_proposals_not_inherited_gates"])
self.assertTrue(self.result["audit"]["unchanged_retention_scope"]["same_continuous_ten_second_lineage"])
self.assertFalse(self.result["audit"]["unchanged_retention_scope"]["retained_snapshot_execution_or_lineage_verified"])
# Config and audit may describe the same row, but neither is a caller alias.
block = self.trial["plan"]["blocks"][0]
self.assertIsNot(self.result["config"]["reversal_checkpoints"], block["evaluation_checkpoints"])
self.assertEqual(self.result["trial_payload_sha256"], self.trial["payload_sha256"])
def test_validation_rejects_rehashed_config_type_and_audit_mutations(self):
# Separate reconstruction is supplied by the already independently built
# fixture to keep mutation tests small; public full reconstruction is
# exercised once below, not hidden behind this mocked trust boundary.
with mock.patch.object(policy, "build_evaluation_policy", return_value=self.result):
for mutation in ("point", "type", "audit", "authorization"):
with self.subTest(mutation=mutation):
changed = copy.deepcopy(self.result)
if mutation == "point":
changed["config"]["snapshot_bindings"][1]["completed_epochs"] += 1
elif mutation == "type":
changed["config"]["cs_start_tick"] = False
elif mutation == "audit":
changed["audit"]["bindings"][0]["actual"]["CS_epochs"] += 1
else:
changed["execution_authorized"] = True
changed["payload_sha256"] = policy._sha({k: v for k, v in changed.items() if k != "payload_sha256"})
with self.assertRaisesRegex(ValueError, "exact reconstruction"):
policy.validate_evaluation_policy(changed, self.trial)
def test_public_validation_full_reconstruction(self):
self.assertTrue(policy.validate_evaluation_policy(self.result, self.trial))
def test_trial_authority_hash_rejected_before_planning(self):
changed = dict(self.trial)
changed["payload_sha256"] = "0" * 64
with self.assertRaisesRegex(ValueError, "payload SHA"):
policy.build_evaluation_policy(changed)
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,318 @@
"""Synthetic arrays/declared receipts only; no real recordings or science run."""
import copy
import hashlib
import json
import unittest
import numpy as np
import metrics as m
NODES = np.arange(6, dtype=np.int32)
POPS = {'MBON': np.array([0, 1, 2], np.int32), 'legacyDownstream': np.arange(6, dtype=np.int32),
'nonMBONDownstream': np.array([3, 4, 5], np.int32),
'MBON01': np.array([0], np.int32), 'MBON12': np.array([1], np.int32)}
def state(counts=(1, 1, 0, 1, 0, 0), *, times=None, g=None):
events = [(10 + 20 * k if times is None else times[i][k], i) for i, count in enumerate(counts) for k in range(count)]
events.sort()
return dict(step=np.array([t for t, _ in events], np.int32),
neuron_idx=np.array([i for _, i in events], np.int32), counts=np.array(counts, np.int32),
v=np.full(6, -.052, np.float64), g=np.zeros(6, np.float64) if g is None else np.array(g, np.float64),
ref=np.zeros(6, np.int32))
def effect(*, direction=1, gain=True, g_effect=True, conditioned=m.CUES[0], neutral_g=0.):
pre = {cue: state() for cue in m.CUES}
post = {cue: state() for cue in m.CUES}
counts = [1, 1, 0, 1, 0, 0]
counts[1 if direction > 0 else 0] += abs(direction)
if gain:
counts[3] += abs(direction)
g = np.zeros(6, np.float64)
if g_effect:
g[0] = .002
post[conditioned] = state(counts, g=g)
neutral = next(c for c in m.CUES if c != conditioned)
post[neutral]['g'][0] = neutral_g
return m.arm_metrics(pre, post, conditioned_cue=conditioned, nodes=NODES, populations=POPS)
def inventory(*, assay='normal', dan='PAM15', paired=None):
registration = {f'rep{i}': [f'training-source-{i}'] for i in range(5)}
rows = []
for i in range(5):
arms = {}
for name in m.ARMS:
metric = effect(direction=0, g_effect=False) if name != 'paired' else (
effect(direction=1 if dan == 'PAM15' else -1) if paired is None else copy.deepcopy(paired[i]))
evidence = dict(complete=True, recording_verified=True,
receipt_sha256=hashlib.sha256(f'{i}:{name}:{assay}'.encode()).hexdigest(), assay=assay,
epoch_duration_ms=120. if assay == 'normal' else 180., cs_duration_ms=100.,
us_tick=900 if assay == 'normal' else 1500, completed_epochs=150 if name == 'unpaired' else 100,
reinforcement_opportunities=0 if name == 'CS-only' else 50, terminal_pairings=50)
arms[name] = dict(metrics=metric, execution=evidence)
rows.append(dict(replicate_id=f'rep{i}', replicate_unit='independent_training',
training_stream_ids=registration[f'rep{i}'], evaluation_stream_ids=[f'evaluation-source-{i}'],
assay=assay, mode='physiological', dan_type=dan, conditioned_cue=m.CUES[0], arms=arms))
return rows, registration
def evaluate(rows, registration, *, assay='normal', dan='PAM15'):
return m.evaluate_acquisition(rows, assay=assay, mode='physiological', dan_type=dan,
conditioned_cue=m.CUES[0], registered_training_streams=registration)
class StateTests(unittest.TestCase):
def test_every_key_validated_and_no_input_mutation(self):
a, b = state(), state((1, 2, 0, 2, 0, 0), g=[.001, -.004, 0, 0, 0, 0])
before = {key: value.tobytes() for key, value in a.items()}
r = m.functional_summary(a, b, nodes=NODES, populations=POPS, steps=1200)
self.assertEqual(r['MBON_final_delta_g_max_abs'], .004)
self.assertEqual(r['MBON_final_delta_g_units'], 'SI volts')
self.assertIn('not time peak', r['MBON_final_delta_g_scope'])
self.assertEqual(r['wholebrain']['spike_count_l1'], 2)
self.assertEqual(before, {key: value.tobytes() for key, value in a.items()})
json.dumps(r, allow_nan=False)
def test_first_event_minimum_not_last_and_zero_tick_is_defined(self):
a = state((3, 0, 0, 0, 0, 0), times=[[0, 20, 100], [], [], [], [], []])
r = m.validate_state(a, nodes=NODES, steps=1200)
self.assertEqual(r['first_tick'].tolist(), [0, -1, -1, -1, -1, -1])
def test_missing_latencies_never_zero_filled(self):
a, b = state((0, 0, 0, 0, 0, 0)), state((1, 0, 0, 0, 0, 0))
r = m.functional_summary(a, b, nodes=NODES, populations=POPS, steps=1200)['populations']['MBON']
self.assertIsNone(r['pre_first_latency_ms'])
self.assertIsNone(r['first_latency_delta_ms'])
self.assertFalse(r['defined_first_latency_changed'])
self.assertEqual(r['first_latency_became_defined_neurons'], 1)
self.assertGreater(r['spike_count_l1'], 0)
def test_nonfinite_input_all_neurons_not_only_scored_targets(self):
for key in ('v', 'g'):
for value in (np.nan, np.inf, -np.inf):
with self.subTest(key=key, value=value):
a = state(); a[key][5] = value
with self.assertRaisesRegex(ValueError, 'Nonfinite'):
m.functional_summary(a, state(), nodes=NODES, populations=POPS, steps=1200)
def test_nonfinite_derived_difference_rejected(self):
a, b = state(), state()
a['g'][0], b['g'][0] = -1e308, 1e308
with self.assertRaisesRegex(ValueError, 'Nonfinite derived'):
m.functional_summary(a, b, nodes=NODES, populations=POPS, steps=1200)
def test_bad_state_dtype_shape_counts_order_bounds_rejected(self):
variants = []
a = state(); a['counts'] = a['counts'].astype(np.int64); variants.append(a)
a = state(); a['counts'][0] += 1; variants.append(a)
a = state(); a['step'][0] = -1; variants.append(a)
a = state(); a['neuron_idx'][0] = 99; variants.append(a)
a = state(); a['ref'][0] = 23; variants.append(a)
a = state(); a['v'] = a['v'][:-1]; variants.append(a)
a = state(); a['neuron_idx'][1] = 0; variants.append(a)
a = state(); a['extra'] = np.zeros(1); variants.append(a)
for a in variants:
with self.subTest(keys=list(a)), self.assertRaises(ValueError):
m.validate_state(a, nodes=NODES, steps=1200)
def test_empty_wrong_duplicate_and_overlapping_memberships_rejected(self):
for key, value in [('MBON12', []), ('MBON01', [0, 0]), ('MBON12', [3]),
('MBON12', [0]), ('nonMBONDownstream', [2, 3]), ('MBON', [0, 1, 99]),
('nonMBONDownstream', [3, 4]), ('legacyDownstream', [0, 1, 3, 4, 5])]:
populations = dict(POPS, **{key: np.array(value, np.int32)})
with self.subTest(key=key, value=value), self.assertRaises(ValueError):
m.validate_populations(NODES, populations)
def test_legacy_downstream_retains_mbon_and_new_setdiff_is_explicit(self):
result = m.validate_populations(NODES, POPS)
self.assertEqual(result['indices']['legacyDownstream'], [0, 1, 2, 3, 4, 5])
self.assertEqual(result['indices']['nonMBONDownstream'], [3, 4, 5])
self.assertIn('NEW hypothesis', result['population_semantics']['nonMBONDownstream'])
old_alias = dict(POPS)
old_alias['trueDownstream'] = old_alias.pop('nonMBONDownstream')
with self.assertRaisesRegex(ValueError, 'exactly'):
m.validate_populations(NODES, old_alias)
class ContrastTests(unittest.TestCase):
def test_fixed_types_not_mask_and_type_mean_not_population_sum(self):
p = dict(POPS, MBON=np.array([0, 1, 2], np.int32), MBON01=np.array([0], np.int32), MBON12=np.array([1, 2], np.int32))
pre = {c: state() for c in m.CUES}
post = {c: state() for c in m.CUES}; post[m.CUES[0]] = state((1, 3, 1, 2, 0, 0))
r = m.arm_metrics(pre, post, conditioned_cue=m.CUES[0], nodes=NODES, populations=p)
self.assertEqual(r['rawB']['post'][m.CUES[0]], 1.)
self.assertEqual(r['rawS'], 1.5)
self.assertFalse(r['B_is_decoder'])
def test_reciprocal_role_keeps_signed_orientation(self):
first, second = effect(), effect(conditioned=m.CUES[1])
self.assertEqual(first['rawS'], second['rawS'])
self.assertEqual(second['conditioned_cue'], m.CUES[1])
def test_raw_labels_not_aliases(self):
with self.assertRaises(ValueError):
m.arm_metrics({'A': state()}, {'B': state()}, conditioned_cue='A', nodes=NODES, populations=POPS)
def test_legacy_and_non_mbon_gains_are_separate_not_an_alias(self):
result = effect(gain=False)
self.assertEqual(result['separation']['MBON']['gain'], 1.)
self.assertEqual(result['separation']['legacyDownstream']['gain'], 1.)
self.assertEqual(result['separation']['nonMBONDownstream']['gain'], 0.)
pre = {cue: state((4, 0, 0, 0, 0, 0) if cue == m.CUES[0] else (0, 0, 0, 0, 0, 0)) for cue in m.CUES}
post = {cue: state((3, 0, 0, 1, 0, 0) if cue == m.CUES[0] else (0, 0, 0, 0, 0, 0)) for cue in m.CUES}
result = m.arm_metrics(pre, post, conditioned_cue=m.CUES[0], nodes=NODES, populations=POPS)
self.assertLess(result['separation']['legacyDownstream']['gain'], 0.)
self.assertGreater(result['separation']['nonMBONDownstream']['gain'], 0.)
class GateTests(unittest.TestCase):
def test_both_directions_and_suites_pass_only_as_draft_proposal(self):
for assay in ('normal', 'delayed'):
for dan in m.DAN_TYPES:
with self.subTest(assay=assay, dan=dan):
rows, registration = inventory(assay=assay, dan=dan)
r = evaluate(rows, registration, assay=assay, dan=dan)
self.assertTrue(r['proposal_gate_pass'])
self.assertEqual(r['actual_control_receipts'], 30)
self.assertEqual(r['successful_same_replicate_conjunctions'], 5)
self.assertFalse(r['scientific_learning_pass_claimed'])
self.assertFalse(r['candidate_generated'])
self.assertTrue(all(v['rawS'] < 0 for v in r['replicates']) if dan == 'PPL103' else
all(v['rawS'] > 0 for v in r['replicates']))
json.dumps(r, allow_nan=False)
def test_opposite_or_zero_signed_effect_is_failure_not_abs_success(self):
for direction in (-1, 0):
rows, registry = inventory(paired=[effect(direction=direction)] * 5)
r = evaluate(rows, registry)
self.assertFalse(r['proposal_gate_pass'])
self.assertEqual(r['successful_same_replicate_conjunctions'], 0)
def test_equal_control_is_failure(self):
rows, registry = inventory()
for row in rows:
row['arms']['CS-only']['metrics'] = copy.deepcopy(row['arms']['paired']['metrics'])
self.assertFalse(evaluate(rows, registry)['proposal_gate_pass'])
def test_same_replicate_conjunction_not_majority_of_each_predicate(self):
effects = [effect(), effect(), effect(gain=False), effect(g_effect=False), effect(direction=-1)]
rows, registry = inventory(paired=effects)
r = evaluate(rows, registry)
self.assertEqual(r['successful_same_replicate_conjunctions'], 2)
self.assertFalse(r['proposal_gate_pass'])
def test_exact_three_of_five_pass(self):
rows, registry = inventory(paired=[effect()] * 3 + [effect(direction=0)] * 2)
self.assertTrue(evaluate(rows, registry)['proposal_gate_pass'])
def test_positive_legacy_gain_cannot_rescue_new_non_mbon_gate(self):
rows, registry = inventory(paired=[effect(gain=False)] * 5)
result = evaluate(rows, registry)
self.assertFalse(result['proposal_gate_pass'])
for row in result['replicates']:
self.assertTrue(row['predicates']['legacyDownstream_AB_euclidean_gain_positive'])
self.assertFalse(row['predicates']['nonMBONDownstream_AB_euclidean_gain_positive'])
self.assertEqual(row['population_separation_gains']['legacyDownstream'], 1.)
self.assertEqual(row['population_separation_gains']['nonMBONDownstream'], 0.)
self.assertEqual(row['legacy_audit']['H_like_downstream_population'], 'legacyDownstream')
self.assertFalse(row['new_hypothesis_audit']['inherited_from_old_H'])
def test_population_metadata_or_overlapping_raw_vectors_cannot_be_forged(self):
for mutation in ('semantics', 'setdiff', 'subset_final'):
rows, registry = inventory()
summary = rows[0]['arms']['paired']['metrics']['cues'][m.CUES[0]]
if mutation == 'semantics':
summary['membership']['population_semantics']['nonMBONDownstream'] = 'old H population'
elif mutation == 'setdiff':
ids = summary['membership']['indices']
ids['nonMBONDownstream'] = [2, 3, 4]
summary['membership']['membership_sha256'] = m._json_hash(ids)
summary['populations']['nonMBONDownstream']['neuron_idx'] = [2, 3, 4]
else:
# Self-consistent terminal deltas can still contradict the full parent population.
child = summary['populations']['nonMBONDownstream']
child['pre_final_v'][0] += .001
child['post_final_v'][0] += .001
with self.subTest(mutation=mutation), self.assertRaises(ValueError):
evaluate(rows, registry)
def test_missing_control_incomplete_receipt_and_wrong_delayed_schedule_rejected(self):
for mutation in ('missing', 'incomplete', 'timing', 'epochs', 'duplicate_receipt', 'best_checkpoint'):
rows, registry = inventory()
if mutation == 'missing':
rows[0]['arms'].pop('unpaired')
else:
e = rows[0]['arms']['unpaired']['execution']
if mutation == 'incomplete': e['complete'] = False
if mutation == 'timing': e['us_tick'] = 1500
if mutation == 'epochs': e['completed_epochs'] = 100
if mutation == 'duplicate_receipt': e['receipt_sha256'] = rows[0]['arms']['paired']['execution']['receipt_sha256']
if mutation == 'best_checkpoint': e['terminal_pairings'] = 20
with self.subTest(mutation=mutation), self.assertRaises(ValueError): evaluate(rows, registry)
def test_evaluation_draws_cannot_be_training_replicates(self):
for mutation in ('unit', 'duplicate_id', 'training_overlap', 'eval_training_overlap', 'eval_overlap', 'extra_draw'):
rows, registry = inventory()
if mutation == 'unit': rows[0]['replicate_unit'] = 'evaluation'
if mutation == 'duplicate_id': rows[1]['replicate_id'] = rows[0]['replicate_id']
if mutation == 'training_overlap': registry['rep1'] = registry['rep0']
if mutation == 'eval_training_overlap': rows[0]['evaluation_stream_ids'] = registry['rep0']
if mutation == 'eval_overlap': rows[1]['evaluation_stream_ids'] = rows[0]['evaluation_stream_ids']
if mutation == 'extra_draw': rows.append(copy.deepcopy(rows[0]))
with self.subTest(mutation=mutation), self.assertRaises(ValueError): evaluate(rows, registry)
def test_raw_nan_forged_B_g_function_and_separation_rejected(self):
for mutation in ('nan', 'rawB', 'g', 'count', 'separation', 'latency'):
rows, registry = inventory()
metric = rows[0]['arms']['paired']['metrics']
group = metric['cues'][m.CUES[0]]['populations']['MBON']
if mutation == 'nan': metric['rawS'] = np.inf
if mutation == 'rawB': metric['rawB']['post'][m.CUES[0]] = 100.
if mutation == 'g': group['final_delta_g_max_abs'] = 100.
if mutation == 'count': group['spike_count_l1'] = 100
if mutation == 'separation': metric['separation']['MBON']['gain'] = 100.
if mutation == 'latency': group['defined_first_latency_changed'] = True
with self.subTest(mutation=mutation), self.assertRaises(ValueError): evaluate(rows, registry)
def test_control_baseline_and_cell_pooling_rejected(self):
for mutation in ('baseline', 'mode', 'DAN', 'assay', 'cue'):
rows, registry = inventory()
if mutation == 'baseline': rows[0]['arms']['frozen']['metrics']['cues'][m.CUES[0]]['pre_state_raw_sha256']['g'] = 'a' * 64
if mutation == 'mode': rows[0]['mode'] = 'matched_energy'
if mutation == 'DAN': rows[0]['dan_type'] = 'PAM-positive'
if mutation == 'assay': rows[0]['assay'] = 'delayed'
if mutation == 'cue': rows[0]['conditioned_cue'] = m.CUES[1]
with self.subTest(mutation=mutation), self.assertRaises(ValueError): evaluate(rows, registry)
def test_reversal_retention_remain_missing_not_fabricated_pass(self):
for assay in ('reversal', 'retention'):
r = m.unavailable_assay(assay)
self.assertEqual(r['status'], 'NOT-IMPLEMENTED')
self.assertIsNone(r['proposal_gate_pass'])
self.assertTrue(r['missing'])
def test_legacy_mean_control_audit_separate_and_strict_not_new_signed_gate(self):
raw = {'appetitive-conditioning': [2.] * 5, 'aversive-conditioning': [1.] * 5,
'd-zero-control': [0.] * 5, 'random-reward-control': [.5] * 5, 'frozen-brain-control': [0.] * 5}
r = m.legacy_h_mean_control_audit(raw)
self.assertTrue(r['predicate_pass'])
self.assertFalse(r['full_old_H_gate_computed'])
raw['random-reward-control'] = [1.] * 5
self.assertFalse(m.legacy_h_mean_control_audit(raw)['predicate_pass'])
raw.pop('d-zero-control')
with self.assertRaises(ValueError): m.legacy_h_mean_control_audit(raw)
def test_legacy_nan_or_evaluation_count_not_admitted(self):
raw = {'appetitive-conditioning': [2.] * 5, 'aversive-conditioning': [1.] * 5,
'd-zero-control': [0.] * 5, 'random-reward-control': [.5] * 5, 'frozen-brain-control': [0.] * 5}
for values in ([0.] * 25, [0., 0., np.nan, 0., 0.]):
raw['frozen-brain-control'] = values
with self.assertRaises(ValueError): m.legacy_h_mean_control_audit(raw)
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,399 @@
"""CPU synthetic checkpoint/clock tests; not a brain or GPU equivalence test."""
import ast
import copy
import json
from pathlib import Path
import tempfile
import types
import unittest
from unittest.mock import patch
import numpy as np
import runtime_continuation as c
R = c.R
AJ = Path(__file__).resolve().parent.parent / 'apl-timing-lab-v1/gpu_timing.py'
class RingTests(unittest.TestCase):
def execute(self, start, steps, raw, phase, apl_pending, events, *, admit_global=True):
ring = c.rotate_spike_ring(raw, phase)
apl = apl_pending.copy()
due, applied = [], []
for tick in range(steps):
absolute = start + tick
ring[tick % 19] = events[absolute]
admitted = absolute >= 18 if admit_global else tick >= 18
due.append(ring[(tick - 18) % 19].copy() if admitted else np.zeros(ring.shape[1], np.uint8))
slot = tick % 18
applied.append(apl[slot].copy())
apl[slot] = np.float32(absolute + 1)
head = steps % 18
return ring, steps, np.concatenate((apl[head:], apl[:head])), np.asarray(due), np.asarray(applied)
def test_multi_segments_preserve_all_future_arrivals_and_apl_slots(self):
rng = np.random.default_rng(193)
events = rng.integers(0, 2, (80, 4), dtype=np.uint8)
initial, pending = np.zeros((19, 4), np.uint8), np.zeros((18, 2), np.float32)
full = self.execute(0, 80, initial, 0, pending, events)
for lengths in ((5, 13, 1, 22, 39), (17, 1, 1, 18, 43), (19, 19, 19, 23), (1,) * 80):
ring, phase, tail, offset, due, applied = initial, 0, pending, 0, [], []
for steps in lengths:
ring, phase, tail, d, a = self.execute(offset, steps, ring, phase, tail, events)
due.append(d); applied.append(a); offset += steps
np.testing.assert_array_equal(np.concatenate(due), full[3])
np.testing.assert_array_equal(np.concatenate(applied), full[4])
np.testing.assert_array_equal(c.rotate_spike_ring(ring, phase), c.rotate_spike_ring(full[0], 80))
np.testing.assert_array_equal(tail, full[2])
def test_parent_shorter_than_delay_needs_global_admission(self):
events = np.zeros((24, 2), np.uint8); events[0, 1] = 1
parent = self.execute(0, 5, np.zeros((19, 2), np.uint8), 0, np.zeros((18, 2), np.float32), events)
child = self.execute(5, 19, *parent[:3], events)
wrong = self.execute(5, 19, *parent[:3], events, admit_global=False)
self.assertEqual(child[3][13, 1], 1) # absolute tick 18, local 13
self.assertEqual(wrong[3][13, 1], 0)
def test_local_tick_zero_request_uses_global_clock_and_ring_phase(self):
events = np.zeros((50, 3), np.uint8)
events[19, 2] = 1 # Synthetic US request at child's local tick zero.
first = self.execute(0, 19, np.zeros((19, 3), np.uint8), 0, np.zeros((18, 2), np.float32), events)
child = self.execute(19, 31, *first[:3], events)
self.assertEqual(child[3][18, 2], 1)
self.assertEqual(np.count_nonzero(child[3][:18, 2]), 0)
def test_rotation_is_copy_and_rejects_invalid_ring(self):
raw = np.zeros((19, 2), np.uint8); raw[18, 1] = 1
before = raw.tobytes()
result = c.rotate_spike_ring(raw, 19)
result[:] = 1
self.assertEqual(raw.tobytes(), before)
for value, phase in ((raw.astype(np.int32), 0), (raw[:18], 0), (raw, -1), (raw, True)):
with self.assertRaises(ValueError):
c.rotate_spike_ring(value, phase)
class ASTTests(unittest.TestCase):
def test_only_restore_and_global_admission_edits_in_both_modes(self):
for mode in ('light', 'reference'):
tree, audit = c.derive_continued(AJ.read_bytes(), mode)
self.assertTrue(audit['full_AST_rollback_identical'])
self.assertTrue(audit['kernels_sorting_and_local_ring_math_unchanged'])
self.assertEqual(sum(isinstance(node, ast.If) and ast.unparse(node.test) == c.CLOCK_NEW for node in ast.walk(tree)), 1)
self.assertEqual(sum(isinstance(node, ast.Expr) and ast.unparse(node) == c.RESTORE for node in ast.walk(tree)), 1)
compile(tree, '<pure continuation AST>', 'exec')
def test_reference_loop_diff_is_exactly_admission_condition(self):
base, _ = R.derive_ast(AJ.read_bytes(), 'reference')
actual, _ = c.derive_continued(AJ.read_bytes(), 'reference')
loops = lambda tree: next(node for node in ast.walk(tree) if isinstance(node, ast.For) and
isinstance(node.target, ast.Name) and node.target.id == 'tick')
modified = copy.deepcopy(loops(actual))
changed = [node for node in ast.walk(modified) if isinstance(node, ast.If) and ast.unparse(node.test) == c.CLOCK_NEW]
changed[0].test = ast.parse(c.CLOCK_OLD, mode='eval').body
self.assertEqual(R._dump(modified), R._dump(loops(base)))
def test_compile_never_mutates_sealed_module_namespace(self):
module = types.ModuleType('fake_aj'); module.__file__ = str(AJ)
original = dict(module.__dict__)
runtime = types.SimpleNamespace(write_json=lambda *_: None, audit_sequence=lambda *_: None)
function, _ = c.compile_continued(module, runtime, 'light')
self.assertEqual(module.__dict__, original)
self.assertEqual(function.__globals__['__file__'], str(Path(c.__file__).resolve()))
self.assertIs(function.__globals__['_continuation'], runtime)
class ArrayTarget:
def __init__(self, value):
self.value = np.array(value, copy=True)
def copy_(self, value):
self.value[...] = value
def any(self):
return np.asarray(self.value.any())
class RestoreTests(unittest.TestCase):
def test_reference_fifo_audit_preserves_carried_bits_for_short_segment(self):
initial = np.arange(36, dtype=np.float32).reshape(18, 2)
initial[0, 0] = np.float32(-0.)
generated = np.arange(10, dtype=np.float32).reshape(5, 2)
joined = np.concatenate((initial, generated))
applied, pending = joined[:5], joined[5:]
result = c.audit_carried_sequence(generated, applied, pending, initial)
self.assertTrue(result['sample_bits_preserved'])
bad = applied.copy(); bad[0, 0] = np.float32(0.)
with self.assertRaisesRegex(ValueError, 'bitwise'):
c.audit_carried_sequence(generated, bad, pending, initial)
def test_timing_counts_and_ranges_use_absolute_clock_not_segment_length(self):
obj = object.__new__(c.ContinuedRuntime)
obj.global_offset, obj.spec, obj.mode = 41, {'steps': 5}, 'reference'
obj.parent = {'terminal': {'apl_pending': np.zeros((18, 4064), np.float32)}}
result = obj._timing_metadata({'pending_generated_count': 5, 'pending_initial_zero_prefix': 13,
'initial_padding_positive_zero': True})
self.assertEqual(result['pending_generated_count'], 18)
self.assertEqual(result['pending_initial_zero_prefix'], 0)
self.assertEqual(result['pending_application_tick_start'], 46)
self.assertEqual(result['pending_generated_tick_start'], 28)
self.assertEqual(result['pending_segment_generated_rows'], 5)
self.assertEqual(result['pending_parent_generated_rows'], 13)
self.assertEqual(result['applied_parent_generated_rows'], 5)
self.assertNotIn('initial_padding_positive_zero', result)
def test_restore_carries_exact_state_and_queues_but_not_segment_counts(self):
parent = {'global_next_tick': 41, 'state': {'v': np.array([-.05, -.04], np.float64),
'g': np.array([-0., -.002], np.float64), 'ref': np.array([22, 3], np.int32)},
'terminal': {'spike_ring': np.zeros((19, 2), np.uint8), 'next_tick': np.asarray(5, np.int64),
'apl_pending': np.arange(36, dtype=np.float32).reshape(18, 2), 'apl_state': np.array([2., 3.], np.float32)}}
parent['terminal']['spike_ring'][4, 1] = 1
obj = object.__new__(c.ContinuedRuntime)
obj.parent, obj.installations, obj.restorations = parent, 1, 0
obj.torch = types.SimpleNamespace(float64=np.float64, float32=np.float32, int32=np.int32, uint8=np.uint8)
v, g, ref, counts = (ArrayTarget(np.zeros(2, dtype)) for dtype in (np.float64, np.float64, np.int32, np.int32))
ring, apl_ring = ArrayTarget(np.zeros((19, 2), np.uint8)), ArrayTarget(np.zeros((18, 2), np.float32))
apl = types.SimpleNamespace(state=ArrayTarget(np.zeros(2, np.float32)))
upload = lambda value, dtype: np.array(value, dtype=dtype, copy=True)
obj.restore(v, g, ref, counts, ring, apl_ring, apl, upload)
self.assertEqual(g.value.tobytes(), parent['state']['g'].tobytes())
np.testing.assert_array_equal(ref.value, parent['state']['ref'])
np.testing.assert_array_equal(counts.value, [0, 0])
np.testing.assert_array_equal(ring.value, c.rotate_spike_ring(parent['terminal']['spike_ring'], 5))
np.testing.assert_array_equal(apl_ring.value, parent['terminal']['apl_pending'])
np.testing.assert_array_equal(apl.state.value, parent['terminal']['apl_state'])
with self.assertRaises(ValueError):
obj.restore(v, g, ref, counts, ring, apl_ring, apl, upload)
class CheckpointTests(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.root = Path(self.tmp.name).resolve()
self.n = patch.object(c, 'N', 4); self.e = patch.object(c, 'E', 6)
self.n.start(); self.e.start()
def tearDown(self):
self.e.stop(); self.n.stop(); self.tmp.cleanup()
def save(self, path, value):
R._save(path, value)
def make(self, name, steps=5, parent=None, corrupt_carry=None):
directory = self.root / name; directory.mkdir()
inputs = self.root / (name + '-inputs'); inputs.mkdir()
spec = {'steps': steps}
if parent is not None:
spec['parent_checkpoint_manifest_sha256'] = parent['manifest_sha256']
self.save(inputs / 'spec.json', spec)
overlay_bytes = b'synthetic-bound-overlay-not-a-brain'
(inputs / 'overlay.npz').write_bytes(overlay_bytes)
meta = {'overlay_sha256': R.digest(inputs / 'overlay.npz'), 'mask_sha256': R.MASK_SHA,
'base_candidate_sha256': 'b' * 64}
self.save(inputs / 'overlay-manifest.json', meta)
(directory / 'overlay-input.npz').write_bytes(overlay_bytes)
self.save(directory / 'overlay-manifest.json', meta)
event_tick, event_neuron = np.array([0], np.int32), np.array([1], np.int32)
counts = np.array([0, 1, 0, 0], np.int32)
state = {'step': event_tick, 'neuron_idx': event_neuron, 'counts': counts,
'v': np.full(4, -.052, np.float64), 'g': np.array([-0., -.003, 0., 1.], np.float64),
'ref': np.array([0, 22, 3, 0], np.int32)}
ring = np.zeros((19, 4), np.uint8) if parent is None else c.rotate_spike_ring(parent['terminal']['spike_ring'], parent['steps'])
for tick in range(steps):
ring[tick % 19] = 0
if tick == 0:
ring[0, 1] = 1
pending = np.zeros((18, 4064), np.float32)
if corrupt_carry == 'spike':
ring[18, 0] ^= np.uint8(1)
elif corrupt_carry == 'apl':
pending[0, 0] = np.float32(1.)
terminal = {'spike_ring': ring, 'apl_pending': pending, 'apl_state': np.array([1., 2.], np.float32),
'next_tick': np.asarray(steps, np.int64)}
np.savez_compressed(directory / 'state.npz', **state)
np.savez_compressed(directory / 'terminal-queues.npz', **terminal)
np.save(directory / 'apl-pending-tail.float32.npy', pending)
self.save(directory / 'apl-timing-audit.json', {'apl_delay_steps': 18, 'brain_tail_advanced': False,
'pending_raw_sha256': R._raw(pending), 'final_apl_state_u32_bits': terminal['apl_state'].view(np.uint32).tolist()})
offset = 0 if parent is None else parent['global_next_tick']
summary = {'complete': True, 'pass_all': True, 'all_finite': True, 'steps': steps, 'completed_ticks': steps,
'nodes': 4, 'edges': 6, 'precision': 'float64', 'apl_precision': 'float32',
'within_trial_weight_updates': False, 'checkpoint_continuation_used': parent is not None,
'effective_weights_raw_sha256': 'e'*64, 'global_tick_offset': offset, 'global_next_tick': offset + steps,
'input_sha256': {'spec.json': R.digest(inputs / 'spec.json')}}
if parent is not None:
summary['parent_checkpoint_manifest_sha256'] = parent['manifest_sha256']
self.save(directory / 'summary.json', summary)
self.save(directory / 'finite-audit.json', {'all_finite': True, 'first_failure_tick': -1, 'finite_observed_ticks': steps})
self.save(directory / 'overlay-audit.json', {'complete': True, 'weights_updated_during_run': False,
'effective_weights_raw_sha256': 'e'*64, 'mask_sha256': R.MASK_SHA, 'am_helper_sha256': R.AM_OVERLAY_SHA,
'base_candidate_sha256': 'b'*64})
provenance = {'schema': 'phase4an-reporting-adapter-v1' if parent is None else 'phase4an-continuation-runtime-v1',
'complete': True, 'weights_updated_during_run': False, 'old_globals_patched': False,
'source_file': R.__file__ if parent is None else c.__file__,
'source_sha256': c.LIGHT_SHA if parent is None else R.digest(c.__file__),
'input_sha256': {str(path): R.digest(path) for path in inputs.iterdir()}}
if parent is not None:
provenance.update(parent_checkpoint_manifest_sha256=parent['manifest_sha256'],
global_tick_offset=offset, global_next_tick=offset + steps)
for key, file in (('summary_sha256', 'summary.json'), ('finite_audit_sha256', 'finite-audit.json'),
('terminal_queues_sha256', 'terminal-queues.npz'), ('overlay_audit_sha256', 'overlay-audit.json'),
('state_sha256', 'state.npz')):
provenance[key] = R.digest(directory / file)
self.save(directory / 'runtime-provenance.json', provenance)
if parent is not None:
self.save(directory / c.RECEIPT, {'schema': 'phase4an-continuation-receipt-v1', 'complete': True,
'parent_checkpoint_dir': str(parent['directory']), 'parent_checkpoint_manifest_sha256': parent['manifest_sha256'],
'segment_steps': steps, 'counts_scope': 'segment_only_not_lifetime', 'weights_updated_during_run': False,
'global_tick_offset': offset, 'global_next_tick': offset + steps, 'payload_sha256': c._tree_hashes(directory)})
return directory
def test_fresh_seal_is_exclusive_and_binds_state_bytes(self):
directory = self.make('fresh')
sealed = c.seal_checkpoint(directory)
loaded = c.load_checkpoint(directory, sealed['sha256'])
self.assertEqual(loaded['global_next_tick'], 5)
self.assertEqual(loaded['state']['g'].tobytes(), np.array([-0., -.003, 0., 1.], np.float64).tobytes())
self.assertIn('state.npz', sealed['files_sha256'])
self.assertFalse(sealed['pre_seal_state_integrity_independently_proven'])
with self.assertRaisesRegex(ValueError, 'already sealed'):
c.seal_checkpoint(directory)
with (directory / 'state.npz').open('ab') as stream:
stream.write(b'tampered')
with self.assertRaisesRegex(ValueError, 'inventory/hash'):
c.load_checkpoint(directory, sealed['sha256'])
def test_multisegment_receipts_bind_absolute_offset_and_local_phase(self):
parent = None
total = 0
for i, steps in enumerate((5, 13, 1, 19)):
directory = self.make('segment-' + str(i), steps, parent)
sealed = c.seal_checkpoint(directory)
parent = c.load_checkpoint(directory, sealed['sha256'])
total += steps
self.assertEqual(parent['global_next_tick'], total)
self.assertEqual(parent['terminal']['next_tick'].item(), steps)
self.assertEqual(parent['ancestry_directories'], [self.root / ('segment-' + str(j)) for j in range(i, -1, -1)])
self.assertEqual(total, 38)
def test_run_rejects_output_and_build_in_every_ancestor_before_overlay_or_gpu(self):
parent = None
for i in range(3):
directory = self.make('ancestor-' + str(i), 20, parent)
seal = c.seal_checkpoint(directory)
parent = c.load_checkpoint(directory, seal['sha256'])
case = self.root / 'next-case'; case.mkdir()
self.save(case / 'spec.json', {'max_steps': 5, 'steps': 5, 'dt_ms': .1, 'duration_ms': .5,
'apl_delay_steps': 18, 'record_history': False,
'parent_checkpoint_manifest_sha256': parent['manifest_sha256']})
before = {path: c._tree_hashes(path) for path in parent['ancestry_directories']}
with patch.object(R, 'load_overlay', side_effect=AssertionError('Overlay/GPU preparation must not begin')) as overlay:
for ancestor in parent['ancestry_directories']:
for forbidden in (ancestor, ancestor / 'new-child'):
for output, build in ((forbidden, self.root / 'safe-build'), (self.root / 'safe-output', forbidden)):
with self.subTest(ancestor=ancestor.name, output=output, build=build), self.assertRaises(ValueError):
c.run_continued(case, output, build, self.root / 'unused-overlay.npz',
self.root / 'unused-manifest.json', parent['directory'])
overlay.assert_not_called()
self.assertEqual(before, {path: c._tree_hashes(path) for path in parent['ancestry_directories']})
self.assertFalse((self.root / 'safe-output').exists())
self.assertFalse((self.root / 'safe-build').exists())
def test_reject_bad_external_pin_extra_file_and_error_marker(self):
directory = self.make('fresh')
sealed = c.seal_checkpoint(directory)
with self.assertRaisesRegex(ValueError, 'external pin'):
c.load_checkpoint(directory, '0' * 64)
(directory / 'unexpected.bin').write_bytes(b'extra')
with self.assertRaisesRegex(ValueError, 'inventory/hash'):
c.load_checkpoint(directory, sealed['sha256'])
failure = self.make('failure')
self.save(failure / 'runtime-error.json', {'complete': False})
with self.assertRaisesRegex(ValueError, 'failure/error'):
c.seal_checkpoint(failure)
def test_self_consistent_receipts_do_not_hide_unwritten_carry_slot_corruption(self):
directory = self.make('parent', 20)
sealed = c.seal_checkpoint(directory)
parent = c.load_checkpoint(directory, sealed['sha256'])
for kind, message in (('spike', 'carried spike'), ('apl', 'carried APL')):
child = self.make('bad-' + kind, 5, parent, corrupt_carry=kind)
# The synthetic writer hashes its incorrect output consistently;
# the independent parent-to-child carry invariant must still reject.
with self.subTest(kind=kind), self.assertRaisesRegex(ValueError, message):
c.seal_checkpoint(child)
def test_reject_partial_receipt_and_modified_original_input(self):
directory = self.make('bad-receipt')
(directory / 'summary.json').write_text('{}', encoding='utf-8')
with self.assertRaises(ValueError):
c.seal_checkpoint(directory)
directory = self.make('modified-input')
(self.root / 'modified-input-inputs' / 'spec.json').write_text('{}', encoding='utf-8')
with self.assertRaisesRegex(ValueError, 'input/source'):
c.seal_checkpoint(directory)
def test_descendant_rejects_mutated_ancestor_even_if_own_files_are_unchanged(self):
directory = self.make('ancestor', 20)
sealed = c.seal_checkpoint(directory)
parent = c.load_checkpoint(directory, sealed['sha256'])
child = self.make('descendant', 5, parent)
child_seal = c.seal_checkpoint(child)
with (directory / 'state.npz').open('ab') as stream:
stream.write(b'ancestor changed after child seal')
with self.assertRaisesRegex(ValueError, 'inventory/hash'):
c.load_checkpoint(child, child_seal['sha256'])
def test_self_consistent_hashes_cannot_hide_parent_pin_or_weight_disagreement(self):
directory = self.make('bound-parent', 20)
sealed = c.seal_checkpoint(directory)
parent = c.load_checkpoint(directory, sealed['sha256'])
def rewrite(path, value):
path.write_text(json.dumps(value, sort_keys=True), encoding='utf-8')
for kind in ('case', 'summary', 'provenance', 'weights'):
child = self.make('bound-child-' + kind, 5, parent)
spec_path = self.root / ('bound-child-' + kind + '-inputs') / 'spec.json'
summary = R._read_json(child / 'summary.json')
provenance = R._read_json(child / 'runtime-provenance.json')
if kind == 'case':
spec = R._read_json(spec_path); spec['parent_checkpoint_manifest_sha256'] = '0' * 64
rewrite(spec_path, spec)
summary['input_sha256']['spec.json'] = R.digest(spec_path)
provenance['input_sha256'][str(spec_path)] = R.digest(spec_path)
elif kind in ('summary', 'provenance'):
target = summary if kind == 'summary' else provenance
target['parent_checkpoint_manifest_sha256'] = '0' * 64
else:
summary['effective_weights_raw_sha256'] = 'f' * 64
audit = R._read_json(child / 'overlay-audit.json'); audit['effective_weights_raw_sha256'] = 'f' * 64
rewrite(child / 'overlay-audit.json', audit)
provenance['overlay_audit_sha256'] = R.digest(child / 'overlay-audit.json')
rewrite(child / 'summary.json', summary)
provenance['summary_sha256'] = R.digest(child / 'summary.json')
rewrite(child / 'runtime-provenance.json', provenance)
receipt = R._read_json(child / c.RECEIPT)
receipt['payload_sha256'] = c._tree_hashes(child); receipt['payload_sha256'].pop(c.RECEIPT)
rewrite(child / c.RECEIPT, receipt)
# All individual byte hashes are now consistent. Only semantic
# cross-receipt binding or parent/child weight equality rejects it.
with self.subTest(kind=kind), self.assertRaisesRegex(ValueError, 'weights changed|pin disagrees'):
c.seal_checkpoint(child)
def test_reject_invalid_state_dtype_events_ref_and_ring(self):
directory = self.make('arrays')
state = c._npz(directory / 'state.npz', ('step', 'neuron_idx', 'counts', 'v', 'g', 'ref'))
terminal = c._npz(directory / 'terminal-queues.npz', ('spike_ring', 'apl_state', 'apl_pending', 'next_tick'))
for key, value in (('v', state['v'].astype(np.float32)), ('g', np.full(4, np.nan)),
('ref', np.full(4, 23, np.int32)), ('step', np.array([5], np.int32)),
('counts', np.zeros(4, np.int32))):
with self.subTest(key=key), self.assertRaises(ValueError):
c.validate_state(dict(state, **{key: value}), terminal, 5, fresh=True)
bad = dict(terminal, spike_ring=terminal['spike_ring'].copy()); bad['spike_ring'][0, 1] = 0
with self.assertRaisesRegex(ValueError, 'emitted'):
c.validate_state(state, bad, 5, fresh=True)
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,298 @@
"""Synthetic AST/array diagnostics only. No real brain/GPU or old-file writes."""
import ast
import copy
import json
from pathlib import Path
import tempfile
import types
import unittest
from unittest.mock import patch
import numpy as np
import runtime_light as r
AJ = Path(__file__).absolute().parent.parent / 'apl-timing-lab-v1/gpu_timing.py'
AM = Path(__file__).absolute().parent.parent / 'wholebrain-plastic-interface-v1/overlay.py'
class ScalarTensor:
"""Tiny NumPy scalar stub for diagnostic algebra, not a neuron simulator."""
def __init__(self, value):
self.value = np.asarray(value)
def logical_and_(self, other):
self.value = self.value & other.value
return self
def __lt__(self, other):
return ScalarTensor(self.value < other)
def __and__(self, other):
return ScalarTensor(self.value & other.value)
def __invert__(self):
return ScalarTensor(~self.value)
def all(self):
return ScalarTensor(self.value.all())
def item(self):
return self.value.item()
class TorchScalars:
bool, int64 = np.bool_, np.int64
ones = staticmethod(lambda shape, device, dtype: ScalarTensor(np.ones(shape, dtype)))
zeros = staticmethod(lambda shape, device, dtype: ScalarTensor(np.zeros(shape, dtype)))
full = staticmethod(lambda shape, value, device, dtype: ScalarTensor(np.full(shape, value, dtype)))
ones_like = staticmethod(lambda value: ScalarTensor(np.ones_like(value.value)))
full_like = staticmethod(lambda value, fill: ScalarTensor(np.full_like(value.value, fill)))
isfinite = staticmethod(lambda value: ScalarTensor(np.isfinite(value)))
where = staticmethod(lambda condition, first, other: ScalarTensor(np.where(condition.value, first.value, other.value)))
class AstTests(unittest.TestCase):
def source(self):
if not AJ.is_file():
self.skipTest('Original pinned AJ source is not installed')
return AJ.read_bytes()
def test_math_projection_equal_light_and_reference_and_guard_explicit(self):
for mode in ('light', 'reference'):
tree, audit = r.derive_ast(self.source(), mode)
self.assertTrue(audit['mathematical_ast_identical'])
self.assertEqual(audit['math_ast_sha256'], audit['derived_math_ast_sha256'])
self.assertIn("spec['max_steps']", ast.unparse(tree.body[0]))
compile(tree, '<synthetic-AST-check>', 'exec')
def test_reference_entire_tick_loop_remains_identical(self):
source = ast.parse(self.source())
original = next(node for node in source.body if isinstance(node, ast.FunctionDef) and node.name == 'run_gpu')
tree, _ = r.derive_ast(self.source(), 'reference')
tick = lambda f: next(n for n in ast.walk(f) if isinstance(n, ast.For) and isinstance(n.target, ast.Name) and n.target.id == 'tick')
self.assertEqual(r._dump(tick(original)), r._dump(tick(tree.body[-1])))
def test_light_retains_events_requests_kernels_and_no_full_state_cpu_copy(self):
tree, _ = r.derive_ast(self.source(), 'light')
function = tree.body[-1]
tick = next(n for n in ast.walk(function) if isinstance(n, ast.For) and isinstance(n.target, ast.Name) and n.target.id == 'tick')
text = ast.unparse(tick)
for expected in ('ext.state_step', 'ext.fill_edges', 'ext.apply_arrivals', 'apl.advance',
'event_steps.append', 'event_nodes.append', 'writer.writerows', '_runtime.after_tick',
'apl_ring[slot].clone()', 'apl_ring[slot].copy_(h)'):
self.assertIn(expected, text)
for forbidden in ('v.cpu()', 'g.cpu()', 'ref.cpu()', 'drive.cpu()', 'h.cpu()', 'queue_summary(',
'queue_output', 'hash_output', 'np.isfinite'):
self.assertNotIn(forbidden, text)
def test_altered_math_is_detected_and_source_pin_rejects_comments(self):
tree, audit = r.derive_ast(self.source(), 'light')
function = copy.deepcopy(tree.body[-1])
node = next(n for n in ast.walk(function) if isinstance(n, ast.Assign) and
ast.unparse(n) == 'g[kc_local_gpu] = g[kc_local_gpu] - delta_si')
node.value.op = ast.Add()
self.assertNotEqual(r.math_projection(function), r.math_projection(tree.body[-1]))
with self.assertRaises(ValueError):
r.derive_ast(self.source() + b'\n# altered\n', 'light')
def test_fresh_namespace_no_fake_file_no_old_globals_changed(self):
self.source()
module = types.ModuleType('synthetic_AJ')
module.__file__ = str(AJ)
module.marker = object()
before = dict(module.__dict__)
runtime = types.SimpleNamespace(write_json=lambda *_: None)
function, audit = r.compile_runtime(module, runtime, 'light')
self.assertEqual(before, module.__dict__)
self.assertEqual(function.__globals__['__file__'], str(Path(r.__file__).absolute()))
self.assertNotEqual(function.__globals__['__file__'], str(AJ))
self.assertFalse(audit['old_globals_patched'])
class ContractTests(unittest.TestCase):
def spec(self, steps=1200):
return dict(max_steps=steps, steps=steps, dt_ms=.1, duration_ms=steps*.1,
apl_delay_steps=18, record_history=False)
def test_directory_parent_traversal_cannot_bypass_input_or_source_boundaries(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()
case, models, graph, bundle = (root / name for name in ('case', 'models', 'graph', 'bundle'))
for path in (case, models, graph, bundle, root / 'new'):
path.mkdir()
safe = root / 'new'
for protected in (case, models, graph, bundle):
alias = safe / '..' / protected.name / 'cache'
for output, build in ((alias, safe / 'build'), (safe / 'out', alias)):
with self.subTest(protected=protected.name, output=output), self.assertRaisesRegex(ValueError, 'cannot write'):
r.execution_paths(case, output, build, (models, graph, bundle))
actual = r.execution_paths(case, safe / '..' / 'safe-output', safe / '..' / 'safe-build', (models, graph, bundle))
self.assertEqual(actual, (case, root / 'safe-output', root / 'safe-build'))
self.assertFalse((root / 'safe-output').exists())
self.assertFalse((root / 'safe-build').exists())
def test_directory_link_alias_cannot_bypass_protected_boundary(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()
case, models = root / 'case', root / 'models'
case.mkdir(); models.mkdir()
alias = root / 'alias-to-models'
try:
alias.symlink_to(models, target_is_directory=True)
except (OSError, NotImplementedError) as error:
self.skipTest('Directory symlink unavailable on this host: ' + str(error))
for output, build in ((alias / 'out', root / 'build'), (root / 'out', alias / 'build')):
with self.subTest(output=output), self.assertRaisesRegex(ValueError, 'cannot write'):
r.execution_paths(case, output, build, (models,))
# A protected root supplied through an alias is also canonicalized.
with self.assertRaisesRegex(ValueError, 'cannot write'):
r.execution_paths(case, root / 'out', models / 'build', (alias,))
def test_explicit_bounds_1000_1200_retention_no_hidden_duration(self):
with tempfile.TemporaryDirectory() as directory:
for steps in (1000, 1200, 100000):
self.assertEqual(r.validate_spec(self.spec(steps), directory)['steps'], steps)
for update in ({'max_steps': 999}, {'max_steps': True}, {'steps': 1200.},
{'duration_ms': 100.}, {'dt_ms': .2}, {'apl_delay_steps': 0},
{'within_trial_weight_updates': True}, {'checkpoint': None}):
with self.subTest(update=update), self.assertRaises(ValueError):
r.validate_spec(dict(self.spec(), **update), directory)
Path(directory, 'arrays.npz').write_bytes(b'unsupported checkpoint')
with self.assertRaises(ValueError):
r.validate_spec(self.spec(), directory)
def test_finite_observer_remembers_nan_that_later_disappears(self):
observer = r.FiniteObserver(TorchScalars, 'CPU-test-stub')
initial = np.array([1., -2.], np.float64)
saved = initial.tobytes()
observer.observe(0, initial)
observer.observe(1, np.array([np.nan], np.float32))
observer.observe(2, np.array([0.], np.float64))
result = observer.result()
self.assertFalse(result['all_finite'])
self.assertEqual(result['first_failure_tick'], 1)
self.assertEqual(result['finite_observed_ticks'], 3)
self.assertEqual(saved, initial.tobytes())
with self.assertRaises(ValueError):
observer.observe(4, initial)
def test_finite_observer_all_valid_reports_minus_one(self):
observer = r.FiniteObserver(TorchScalars, 'CPU-test-stub')
for tick in range(4):
observer.observe(tick, np.array([1, 0], np.int32), np.array([-0., 1.], np.float64))
self.assertTrue(observer.result()['all_finite'])
self.assertEqual(observer.result()['first_failure_tick'], -1)
def test_summary_has_no_fake_outer_learning_claim_and_light_audit_is_not_full(self):
obj = object.__new__(r._Runtime)
obj.mode, obj.spec = 'light', dict(max_steps=1200, steps=1200, outer_protocol='conditioning-owned-by-parent')
obj.result = dict(first_failure_tick=-1, arriving_edges=11, accepted_arriving_edges=7)
inherited = dict(learning=False, arriving_edges=0, accepted_arriving_edges=0, complete=True,
apl_timing={'generated_trace': 'absent.npy', 'applied_trace': 'absent.npy'})
final = obj.summary(inherited)
self.assertNotIn('learning', final)
self.assertFalse(final['within_trial_weight_updates'])
self.assertFalse(final['learning_pass_claimed'])
self.assertFalse(final['full_sequence_bitwise_audit_performed'])
self.assertEqual(final['arriving_edges'], 11)
self.assertIsNone(final['apl_timing']['generated_trace'])
self.assertIsNone(final['apl_timing']['applied_trace'])
self.assertFalse(obj.timing(1200, np.zeros((18, 4064), np.float32))['full_sequence_bitwise_audit_performed'])
self.assertEqual(inherited['arriving_edges'], 0)
def test_terminal_queue_schema_chronological_pending_and_fresh_tick(self):
class CPUArray:
def __init__(self, array):
self.value = array
def cpu(self):
return self
def numpy(self):
return self.value
ring = np.zeros((19, 166700), np.uint8)
ring[2, 3] = 1
apl_ring = np.repeat(np.arange(18, dtype=np.float32)[:, None], 4064, axis=1)
apl = types.SimpleNamespace(state=CPUArray(np.array([1., 2.], np.float32)))
obj = object.__new__(r._Runtime)
obj.mode, obj.installations, obj.finished = 'light', 1, False
obj.observer = types.SimpleNamespace(result=lambda: dict(all_finite=True, first_failure_tick=-1,
finite_observed_ticks=1200, arriving_edges=11, accepted_arriving_edges=7))
with tempfile.TemporaryDirectory() as directory:
obj.finish(CPUArray(ring), CPUArray(apl_ring), apl, 1200, Path(directory))
with np.load(Path(directory)/'terminal-queues.npz', allow_pickle=False) as output:
self.assertEqual(set(output.files), {'spike_ring', 'apl_state', 'apl_pending', 'next_tick'})
self.assertEqual(output['spike_ring'].dtype, np.uint8)
self.assertEqual(output['spike_ring'].tobytes(), ring.tobytes())
self.assertEqual(output['next_tick'].dtype, np.int64)
self.assertEqual(output['next_tick'].item(), 1200)
expected = np.concatenate((apl_ring[1200 % 18:], apl_ring[:1200 % 18]))
np.testing.assert_array_equal(output['apl_pending'], expected)
self.assertTrue(obj.finished)
with self.assertRaises(ValueError):
obj.finish(CPUArray(ring), CPUArray(apl_ring), apl, 1200, Path(directory))
def test_pinned_overlay_helpers_toy_units_and_missing_nonzero_fail_closed(self):
if not AM.is_file():
self.skipTest('Original frozen AM overlay helper not installed')
helper = r._load('phase4an_test_overlay_helper', AM, r.AM_OVERLAY_SHA)
ids, base32 = np.array([3, 9], np.int64), np.array([.275, 1.1], np.float32)
delta = np.float32(-.25) * base32
plan = helper.validate_arrays(ids, delta, ids, base32)
base = np.array([.000275, .0011], np.float64)
result, _ = helper.materialize_numpy(base, ids, np.zeros(2, bool), plan)
self.assertTrue(helper.bit_equal(result, base + delta.astype(np.float64) * .001))
with self.assertRaises(ValueError):
helper.map_installation(base[:1], ids[:1], np.zeros(1, bool), plan)
with self.assertRaises(ValueError):
helper.validate_arrays(ids, np.array([.1, 0], np.float32), ids, base32)
def test_manifest_cannot_self_authorize_replacement_mask(self):
with tempfile.TemporaryDirectory() as directory:
path = Path(directory)
manifest = dict(schema='phase4an-overlay-v1', overlay_name='learned', overlay_sha256='0'*64,
mask_path=str(path/'mask.npy'), mask_sha256='0'*64, base_candidate_path=str(path/'base.npy'),
base_candidate_sha256='0'*64, aj_model_dir=str(AJ.parent), am_overlay_path=str(AM),
am_overlay_sha256=r.AM_OVERLAY_SHA)
(path/'manifest.json').write_text(json.dumps(manifest), encoding='utf-8')
with self.assertRaisesRegex(ValueError, 'authority'):
r.load_overlay(path/'delta.npz', path/'manifest.json')
def test_learned_manifest_uses_external_base_and_requires_parent_v2(self):
if not AM.is_file() or not AJ.is_file():
self.skipTest('Pinned static source dependencies are not installed')
with tempfile.TemporaryDirectory() as directory:
path = Path(directory)
ids = np.arange(14232, dtype=np.int64)
base = np.full(14232, .275, np.float32)
delta = np.zeros(14232, np.float32); delta[0] = np.float32(-.00275)
np.save(path/'mask.npy', ids); np.save(path/'base.npy', base)
np.savez(path/'delta.npz', edge_id=ids, plastic_delta_mv=delta)
mask_sha = r.digest(path/'mask.npy')
manifest = dict(schema='phase4an-overlay-v1', overlay_name='learned',
overlay_sha256=r.digest(path/'delta.npz'), mask_path=str(path/'mask.npy'), mask_sha256=mask_sha,
base_candidate_path=str(path/'base.npy'), base_candidate_sha256=r.digest(path/'base.npy'),
aj_model_dir=str(AJ.parent), am_overlay_path=str(AM), am_overlay_sha256=r.AM_OVERLAY_SHA,
parent_snapshot_sha256='a'*64, eligibility_v2_source_sha256=r.V2_SHA)
(path/'manifest.json').write_text(json.dumps(manifest), encoding='utf-8')
# Only the authority fixture is patched; production pin remains fixed.
with patch.object(r, 'MASK_SHA', mask_sha):
helper, plan, meta, inputs = r.load_overlay(path/'delta.npz', path/'manifest.json')
self.assertEqual(plan['plastic_delta_mv'].tobytes(), delta.tobytes())
self.assertFalse(plan['plastic_delta_mv'].flags.writeable)
self.assertEqual(meta['parent_snapshot_sha256'], 'a'*64)
unexpected = dict(manifest, within_trial_weight_updates=True)
(path/'manifest.json').write_text(json.dumps(unexpected), encoding='utf-8')
with self.assertRaisesRegex(ValueError, 'Unknown'):
r.load_overlay(path/'delta.npz', path/'manifest.json')
manifest.pop('parent_snapshot_sha256')
(path/'manifest.json').write_text(json.dumps(manifest), encoding='utf-8')
with self.assertRaisesRegex(ValueError, 'parent'):
r.load_overlay(path/'delta.npz', path/'manifest.json')
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,123 @@
import copy
import hashlib
import json
import unittest
from unittest.mock import patch
import seed_registry as m
class Seeds(unittest.TestCase):
def test_order_independent(self):
a, b = m.SeedRegistry(461), m.SeedRegistry(461)
keys = [("mode", "PAM15", str(replicate), str(epoch)) for replicate in range(5) for epoch in range(50)]
for key in keys:
a.add("training", key)
for key in reversed(keys):
b.add("training", key)
self.assertEqual(a.manifest(), b.manifest())
self.assertEqual(a.manifest()["independent_count"], 250)
self.assertTrue(m.validate_manifest(a.manifest()))
def test_splits_disjoint_and_no_sequential_overlap(self):
r = m.SeedRegistry(461)
for split in ("training", "evaluation", "design"):
for replicate in range(5):
for epoch in range(100):
r.add(split, ("shared-name", str(replicate), str(epoch)))
out = r.manifest()
seeds = [x["seed"] for x in out["independent_streams"]]
self.assertEqual(len(set(seeds)), 1500)
self.assertEqual(out["training_evaluation_seed_overlap"], 0)
self.assertTrue(all(0 <= s < 2**63 for s in seeds))
def test_alias_explicit_not_independent(self):
r = m.SeedRegistry(461)
seed = r.add("training", ("paired", "0"))
self.assertEqual(seed, r.alias("training", ("frozen", "0"), "training", ("paired", "0"), reason="Same prescheduled CS inputs, different learning policy"))
out = r.manifest()
self.assertEqual(out["independent_count"], 1)
self.assertEqual(out["alias_count"], 1)
self.assertFalse(out["aliases_count_as_independent_replicates"])
self.assertTrue(m.validate_manifest(out))
def test_no_cross_split_alias(self):
r = m.SeedRegistry(1)
r.add("training", ("a",))
with self.assertRaisesRegex(ValueError, "Cross-split"):
r.alias("evaluation", ("a",), "training", ("a",), reason="Not permitted")
def test_no_alias_chain(self):
r = m.SeedRegistry(1)
r.add("training", ("a",))
r.alias("training", ("b",), "training", ("a",), reason="Control")
with self.assertRaisesRegex(ValueError, "not another alias"):
r.alias("training", ("c",), "training", ("b",), reason="Invalid chain")
def test_no_silent_duplicate(self):
r = m.SeedRegistry(1)
r.add("training", ("a",))
with self.assertRaisesRegex(ValueError, "Duplicate"):
r.add("training", ("a",))
with self.assertRaisesRegex(ValueError, "Duplicate"):
r.alias("training", ("a",), "training", ("a",), reason="Invalid")
r.alias("training", ("b",), "training", ("a",), reason="Control")
with self.assertRaisesRegex(ValueError, "Duplicate"):
r.add("training", ("b",))
def test_collision_fails_without_resampling(self):
r = m.SeedRegistry(1)
with patch.object(m, "seed_for", return_value=(123, "a" * 64)):
r.add("training", ("a",))
with self.assertRaisesRegex(ValueError, "Seed collision"):
r.add("evaluation", ("b",))
self.assertEqual(len(r._streams), 1)
def test_mutations_detected(self):
r = m.SeedRegistry(1)
r.add("training", ("a",))
original = r.manifest()
for mutate in (lambda x: x["independent_streams"][0].update(seed=7),
lambda x: x.update(training_evaluation_seed_overlap=1),
lambda x: x.update(independent_count=2),
lambda x: x.update(payload_sha256="a" * 64),
lambda x: x.update(training_executed=True)):
corrupt = copy.deepcopy(original)
mutate(corrupt)
with self.assertRaises(ValueError):
m.validate_manifest(corrupt)
def test_manifest_copy_not_mutable_alias(self):
r = m.SeedRegistry(1)
r.add("training", ("a",))
out = r.manifest()
out["independent_streams"][0]["key"][0] = "changed"
self.assertEqual(r.manifest()["independent_streams"][0]["key"], ["a"])
self.assertTrue(m.validate_manifest(json.loads(json.dumps(r.manifest()))))
def test_json_numeric_boolean_type_changes_rejected_even_when_rehashed(self):
r = m.SeedRegistry(1)
r.add("training", ("a",))
for field, replacement in (("training_executed", 0), ("independent_count", 1.0)):
for rehash in (False, True):
corrupt = r.manifest()
corrupt[field] = replacement
if rehash:
payload = {key: value for key, value in corrupt.items() if key != "payload_sha256"}
corrupt["payload_sha256"] = hashlib.sha256(m._canonical(payload).encode("utf-8")).hexdigest()
with self.subTest(field=field, rehash=rehash), self.assertRaises(ValueError):
m.validate_manifest(corrupt)
def test_reject_invalid_values(self):
for master in (True, -1, 2**63, 1.0, "1"):
with self.assertRaises(ValueError):
m.SeedRegistry(master)
for key in ([], "a", [""], [" a"], ["a "]):
with self.assertRaises(ValueError):
m.seed_for(1, "training", key)
with self.assertRaises(ValueError):
m.seed_for(1, "unknown", ["a"])
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,171 @@
import copy
from collections import Counter
import unittest
from unittest.mock import patch
import seed_registry
import trial_plan as m
class TrialPlan(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.original = m.build_trial_plan(461)
def test_complete_reconstruction_and_budget(self):
self.assertTrue(m.validate_trial_plan(self.original))
budget = self.original["budget"]
self.assertEqual(budget["training_control_reversal_epochs"], 68000)
self.assertEqual(budget["training_control_reversal_steps"], 99600000)
self.assertEqual(budget["minimum_biological_seconds_excluding_evaluation"], 10760)
self.assertEqual(budget["minimum_steps_excluding_evaluation"], 107600000)
self.assertFalse(budget["evaluation_probe_inventory_complete"])
self.assertIsNone(budget["evaluation_epochs"])
self.assertFalse(self.original["plan"]["training_executed"])
def test_exact_cells_five_training_replicates_not_evaluation_streams(self):
cells = {}
for block in self.original["plan"]["blocks"]:
key = block["mode"], block["dan_type"], block["initial_reinforced_cue"]
cells.setdefault(key, []).append(block["replicate"])
self.assertEqual(len(block["evaluation_streams"]), 2)
self.assertEqual(len(cells), 8)
self.assertTrue(all(replicates == [1, 2, 3, 4, 5] for replicates in cells.values()))
def test_all_epoch_control_roles_and_matched_delayed_slots(self):
for block in self.original["plan"]["blocks"]:
for suite in block["suites"]:
assay = suite["assay"]
arms = {a["arm"]: a for a in suite["arms"]}
self.assertEqual(tuple(arms), m.REVERSAL_ARMS if assay == "reversal" else m.ARMS)
expected_steps, expected_us = (1800, 1500) if assay == "delayed" else (1200, 900)
for arm, spec in arms.items():
epochs = spec["epochs"]
self.assertEqual(len(epochs), 150 if arm == "unpaired" else 100)
cues = Counter(e["cs"]["cue"] for e in epochs if e["cs"] is not None)
self.assertEqual(cues, Counter() if arm == "dan_only" else Counter(A=50, B=50))
us = [e for e in epochs if e["us"]["requested"]]
self.assertEqual(len(us), 0 if arm == "cs_only" else 50)
self.assertEqual(len({e["us"]["inventory_id"] for e in us}), len(us))
for i, e in enumerate(epochs):
self.assertEqual(e["epoch_index"], i)
self.assertEqual(e["steps"], expected_steps)
self.assertEqual(e["us"]["nominal_slot_tick"], expected_us)
self.assertEqual(e["us"]["tick"], expected_us if e["us"]["requested"] else None)
self.assertIsNone(e["us"]["rng_seed"])
self.assertEqual(e["us"]["rng_draws"], 0)
self.assertEqual(e["update_enabled"], arm != "frozen")
self.assertEqual(e["D_command"], 1.0 if e["us"]["requested"] and arm != "d_zero" else 0.0)
if arm == "random_reward":
self.assertEqual(Counter(e["cs"]["cue"] for e in us), Counter(A=25, B=25))
if arm == "unpaired":
self.assertTrue(all(e["cs"] is None for e in us))
if arm == "dan_only":
self.assertEqual(sum(not e["us"]["requested"] for e in epochs), 50)
def test_alias_binding_same_occurrence_even_unpaired_insertion(self):
for block in self.original["plan"]["blocks"]:
for suite in block["suites"]:
base = suite["arms"][0]
lookup = {(e["cs"]["cue"], e["cs"]["occurrence"]): e["cs"]
for e in base["epochs"]}
base_order = [(e["cs"]["cue"], e["cs"]["occurrence"]) for e in base["epochs"]]
for arm in suite["arms"][1:]:
actual_order = []
for e in arm["epochs"]:
if e["cs"] is None:
continue
cs = e["cs"]
source = lookup[(cs["cue"], cs["occurrence"])]
self.assertEqual(cs["seed"], source["seed"])
self.assertEqual(cs["source_stream_id"], source["stream_id"])
self.assertNotEqual(cs["stream_id"], source["stream_id"])
actual_order.append((cs["cue"], cs["occurrence"]))
if arm["arm"] != "dan_only":
self.assertEqual(actual_order, base_order)
def test_reversal_not_extra_acquisition_or_new_compartment(self):
for b in self.original["plan"]["blocks"]:
reversal, continuation = b["suites"][2]["arms"]
original = b["initial_reinforced_cue"]
other = b["initial_neutral_cue"]
for spec, cue in ((reversal, other), (continuation, original)):
self.assertEqual(spec["delta_start"], {"kind": "fork", "source": b["block_id"] + "/normal/paired/terminal-delta"})
self.assertEqual({e["cs"]["cue"] for e in spec["epochs"] if e["us"]["requested"]}, {cue})
def test_split_and_replicate_seeds_are_independent_fullwidth(self):
manifest = self.original["seed_manifest"]
all_seeds = [r["seed"] for r in manifest["independent_streams"]]
self.assertEqual(len(all_seeds), len(set(all_seeds)))
by_block = {}
by_split = {}
for row in manifest["independent_streams"]:
by_block.setdefault(row["key"][0], set()).add(row["seed"])
by_split.setdefault(row["split"], set()).add(row["seed"])
self.assertEqual(len(by_block), 40)
blocks = list(by_block)
for i, block in enumerate(blocks):
for other in blocks[i + 1:]:
self.assertFalse(by_block[block] & by_block[other])
self.assertFalse(by_split["training"] & by_split["evaluation"])
self.assertTrue(any(seed > 2**32 for seed in all_seeds))
for block in self.original["plan"]["blocks"]:
for probe in block["evaluation_streams"]:
self.assertIn(probe["seed"], by_split["evaluation"])
self.assertEqual(probe["seed_bits"], 63)
def test_generic_valid_alias_can_still_be_protocol_illegal(self):
for source, destination in (
(("b-rep1", "normal", "CS", "A", "00", "paired"), ("b-rep2", "normal", "CS", "A", "00", "frozen")),
(("b", "normal", "US", "A", "00", "paired"), ("b", "normal", "CS", "A", "00", "frozen")),
(("b", "normal", "CS", "A", "00", "paired"), ("b", "delayed", "CS", "A", "00", "frozen")),
(("b", "normal", "CS", "A", "00", "paired"), ("b", "normal", "CS", "B", "00", "frozen")),
(("b", "normal", "CS", "A", "00", "paired"), ("b", "normal", "CS", "A", "01", "frozen")),
(("b", "normal", "CS", "A", "00", "paired"), ("b", "normal", "CS", "A", "00", "undeclared-control")),
):
r = seed_registry.SeedRegistry(1)
r.add("training", source)
r.alias("training", destination, "training", source, reason="Synthetic prohibited crossing")
self.assertTrue(seed_registry.validate_manifest(r.manifest()))
with self.assertRaises(ValueError):
m._validate_alias_permissions(r.manifest())
def test_mutations_fail_even_after_forging_payload_sha(self):
def first(p):
return p["plan"]["blocks"][0]["suites"][0]["arms"][0]["epochs"][0]
mutations = (
lambda p: p["plan"]["blocks"].pop(),
lambda p: p["plan"]["blocks"].__setitem__(1, copy.deepcopy(p["plan"]["blocks"][0])),
lambda p: p["plan"]["blocks"][0]["suites"][0]["arms"][0]["epochs"].pop(),
lambda p: first(p)["cs"].update(seed=first(p)["cs"]["seed"] & 0xffffffff),
lambda p: first(p)["cs"].update(seed_bits=32),
lambda p: first(p)["cs"].update(odor="post-outcome-selected odor"),
lambda p: p["plan"].update(execution_authorized=0),
lambda p: p["budget"].update(evaluation_epochs=0),
)
for index, mutate in enumerate(mutations):
payload = copy.deepcopy(self.original)
mutate(payload)
payload["payload_sha256"] = m._sha({k: v for k, v in payload.items() if k != "payload_sha256"})
with self.subTest(mutation=index), self.assertRaises(ValueError):
m.validate_trial_plan(payload)
def test_deterministic_small_shuffle_and_fullwidth_seed(self):
a = m._Shuffle(23).shuffled(range(30))
self.assertEqual(a, m._Shuffle(23).shuffled(range(30)))
self.assertNotEqual(a, m._Shuffle(23 + 2**32).shuffled(range(30)))
self.assertEqual(sorted(a), list(range(30)))
with self.assertRaises(ValueError):
m._Shuffle(True)
def test_integer_rejection_advances_counter_not_seed(self):
shuffle = m._Shuffle(23 + 2**32)
with patch.object(m, "_sha", side_effect=["f" * 64, "0" * 64]) as hashed:
self.assertEqual(shuffle.below(3), 0)
self.assertEqual(shuffle.counter, 2)
self.assertEqual(hashed.call_args_list[0].args[0], [m.SHUFFLE_VERSION, 23 + 2**32, 0])
self.assertEqual(hashed.call_args_list[1].args[0], [m.SHUFFLE_VERSION, 23 + 2**32, 1])
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,325 @@
"""Pure, unregistered conditioning inventory; no files, brain inputs or runs.
build_trial_plan(master) returns plan/seed_manifest/budget. validate_trial_plan
checks seed authority, alias permissions, and exact deterministic reconstruction
of the complete inventory, not just totals. A consumer must still bind the exact
63-bit seed and generator to its generated input receipt; this module cannot
certify a future consumer that silently truncates an integer.
"""
from __future__ import annotations
from collections import Counter
import hashlib
import json
import seed_registry
SCHEMA = "phase4an-trial-plan-v1-draft"
SHUFFLE_VERSION = "sha256-counter-rejection-fisher-yates-v1"
MODES = ("physiological", "matched_energy")
DAN_BRANCHES = (("PAM15", "y5B'2a"), ("PPL103", "y2a'1"))
CUES = ("A", "B")
ODORS = {"A": "4-ethylphenol", "B": "3-hexanone"}
ARMS = ("paired", "d_zero", "frozen", "random_reward", "unpaired", "cs_only", "dan_only")
REVERSAL_ARMS = ("reversal", "no_reversal")
CONTROL_ALIASES = frozenset(("d_zero", "frozen", "random_reward", "unpaired", "cs_only"))
PROBE_CHECKPOINTS = (0, 1, 2, 5, 10, 20, 50)
def _require(value, message):
if not value:
raise ValueError(message)
def _canonical(value):
return json.dumps(value, sort_keys=True, separators=(",", ":"),
ensure_ascii=False, allow_nan=False)
def _sha(value):
return hashlib.sha256(_canonical(value).encode("utf-8")).hexdigest()
class _Shuffle:
"""Explicit portable integer algorithm, unrelated to neural input RNG.
Each draw hashes canonical UTF-8 JSON [version, full63bitseed, counter].
Interpret all 256 digest bits as a big-endian unsigned integer; increment
counter even for a rejection. Reject x >= 2**256 - (2**256 % bound), then
return x % bound. Fisher-Yates descends i=n-1..1 and swaps with draw(i+1).
No Python random/NumPy algorithm, modulo bias, or seed search is used.
"""
def __init__(self, seed):
_require(type(seed) is int and 0 <= seed < 2**63, "Full 63-bit shuffle seed required")
self.seed = seed
self.counter = 0
def below(self, bound):
_require(type(bound) is int and 0 < bound <= 2**256, "Invalid shuffle bound")
limit = 2**256 - (2**256 % bound)
while True:
raw = bytes.fromhex(_sha([SHUFFLE_VERSION, self.seed, self.counter]))
self.counter += 1
value = int.from_bytes(raw, "big")
if value < limit:
return value % bound
def shuffled(self, values):
result = list(values)
for i in range(len(result) - 1, 0, -1):
j = self.below(i + 1)
result[i], result[j] = result[j], result[i]
return result
def _stream_id(master, split, key):
return split + ":" + _sha([SCHEMA, "stream-id", master, split, list(key)])
def _block_id(mode, dan, reinforced, replicate):
return f"{mode}__{dan}__reinforce_{reinforced}__rep{replicate:02d}"
def _input_key(block, assay, cue, occurrence, arm):
return (block, assay, "CS", cue, f"{occurrence:02d}", arm)
def _cs_binding(registry, block, assay, cue, occurrence, arm, base_arm):
key = _input_key(block, assay, cue, occurrence, arm)
source_key = _input_key(block, assay, cue, occurrence, base_arm)
if arm == base_arm:
seed = registry.add("training", key)
else:
seed = registry.alias("training", key, "training", source_key,
reason="Same block/assay/CS cue/occurrence; named counterfactual control")
return {"cue": cue, "odor": ODORS[cue], "occurrence": occurrence,
"stream_id": _stream_id(registry.master, "training", key),
"source_stream_id": _stream_id(registry.master, "training", source_key),
"seed": seed, "seed_bits": 63, "seed_conversion": "exact integer; no truncation or addition",
"role": "CS", "start_tick": 0, "stop_tick_exclusive": 1000}
def _ordered_cues(seed):
counts = {"A": 0, "B": 0}
result = []
for cue in _Shuffle(seed).shuffled(["A"] * 50 + ["B"] * 50):
result.append((cue, counts[cue]))
counts[cue] += 1
return result
def _epoch(registry, block, assay, arm, ordinal, cue, occurrence, base_arm,
us_ordinal, yoked_cue=None, yoked_occurrence=None):
delayed = assay == "delayed"
steps, us_tick = (1800, 1500) if delayed else (1200, 900)
requested = us_ordinal is not None
cs = None if cue is None else _cs_binding(registry, block, assay, cue, occurrence, arm, base_arm)
return {
"epoch_id": f"{block}/{assay}/{arm}/{ordinal:03d}",
"epoch_index": ordinal, "steps": steps, "duration_ms": steps // 10,
"initial_neural_state": "frozen_reset_including_KC_trace_and_both_delay_queues",
"cs": cs,
"us": {"requested": requested,
"inventory_id": None if not requested else f"{block}/{assay}/US/{us_ordinal:02d}",
"target_binding": None if not requested else block + "/confirmed-DAN-targets",
"tick": us_tick if requested else None,
"nominal_slot_tick": us_tick,
"request_rule": "fixed_single_pulse_all_bound_DAN_targets" if requested else "none",
"rng_seed": None, "rng_draws": 0},
"D_command": 1.0 if requested and arm != "d_zero" else 0.0,
"D_definition": "scheduled artificial scalar, not multiplied by acceptance",
"update_enabled": arm != "frozen",
"update_schedule": "epoch_boundary_only",
"yoked_cue_removed": yoked_cue,
"yoked_occurrence_removed": yoked_occurrence,
}
def _suite(registry, block, assay, reinforced):
base_arm = "reversal" if assay == "reversal" else "paired"
order_key = (block, assay, "cue-order")
order_seed = registry.add("design", order_key)
ordered = _ordered_cues(order_seed)
designs = [{"purpose": "cue-order", "stream_id": _stream_id(registry.master, "design", order_key),
"seed": order_seed, "algorithm": SHUFFLE_VERSION}]
random_sets = None
placements = None
if assay != "reversal":
key = (block, assay, "random-reward-placement")
seed = registry.add("design", key)
shuffle = _Shuffle(seed)
random_sets = {cue: sorted(shuffle.shuffled(range(50))[:25]) for cue in CUES}
designs.append({"purpose": "random-reward-placement", "stream_id": _stream_id(registry.master, "design", key),
"seed": seed, "algorithm": SHUFFLE_VERSION})
key = (block, assay, "unpaired-US-placement")
seed = registry.add("design", key)
placements = _Shuffle(seed).shuffled(["CS"] * 100 + ["US"] * 50)
designs.append({"purpose": "unpaired-US-placement", "stream_id": _stream_id(registry.master, "design", key),
"seed": seed, "algorithm": SHUFFLE_VERSION})
arms = []
for arm in REVERSAL_ARMS if assay == "reversal" else ARMS:
events = []
arm_reinforced = ("B" if reinforced == "A" else "A") if arm == "reversal" else reinforced
if arm == "unpaired":
cursor, us_cursor = 0, 0
for token in placements:
if token == "CS":
cue, occurrence = ordered[cursor]
cursor += 1
events.append(_epoch(registry, block, assay, arm, len(events), cue, occurrence, base_arm, None))
else:
events.append(_epoch(registry, block, assay, arm, len(events), None, None, base_arm, us_cursor))
us_cursor += 1
else:
reward_us_ids = None
if arm == "random_reward":
reward_us_ids = {(cue, occurrence): offset + i
for cue, offset in (("A", 0), ("B", 25))
for i, occurrence in enumerate(random_sets[cue])}
for cue, occurrence in ordered:
if arm == "random_reward":
us_ordinal = reward_us_ids.get((cue, occurrence))
elif arm == "cs_only":
us_ordinal = None
else:
us_ordinal = occurrence if cue == arm_reinforced else None
if arm == "dan_only":
events.append(_epoch(registry, block, assay, arm, len(events), None, None, base_arm,
us_ordinal, yoked_cue=cue, yoked_occurrence=occurrence))
else:
events.append(_epoch(registry, block, assay, arm, len(events), cue, occurrence, base_arm, us_ordinal))
arms.append({
"arm": arm,
"delta_start": {"kind": "fork", "source": f"{block}/normal/paired/terminal-delta"}
if assay == "reversal" else {"kind": "zero", "source": None},
"carry_between_epochs": ["plastic_delta_only"],
"reinforced_cue": arm_reinforced if arm not in ("random_reward", "unpaired", "cs_only", "dan_only") else None,
"random_reward_occurrences": random_sets if arm == "random_reward" else None,
"epochs": events,
})
return {"assay": assay, "design_streams": designs, "arms": arms}
def _assemble(master):
registry = seed_registry.SeedRegistry(master)
blocks = []
for mode in MODES:
for dan, compartment in DAN_BRANCHES:
for reinforced in CUES:
for replicate in range(1, 6):
block = _block_id(mode, dan, reinforced, replicate)
evaluations = []
for cue in CUES:
key = _input_key(block, "evaluation", cue, 0, "base-probe")
seed = registry.add("evaluation", key)
evaluations.append({"cue": cue, "odor": ODORS[cue], "role": "CS", "seed": seed,
"seed_bits": 63, "stream_id": _stream_id(master, "evaluation", key),
"use": "same saved input across pre/post/controls; never extra training replicate"})
blocks.append({
"block_id": block, "mode": mode, "dan_type": dan,
"compartment_verbatim": compartment, "replicate": replicate,
"initial_reinforced_cue": reinforced,
"initial_neutral_cue": "B" if reinforced == "A" else "A",
"DAN_target_binding": {"binding_id": block + "/confirmed-DAN-targets",
"source": "sealed H exact confirmed mapping; not materialized by this planner",
"required_before_execution": True, "body_ids": None},
"suites": [_suite(registry, block, assay, reinforced)
for assay in ("normal", "delayed", "reversal")],
"evaluation_streams": evaluations,
"evaluation_checkpoints": list(PROBE_CHECKPOINTS),
"evaluation_policy": "separate future probe inventory required; evaluation exposure not included in epoch budget",
"retention": {"branches": ["normal/paired", "normal/frozen"],
"checkpoint_seconds": [0, 1, 10], "idle_seconds_each": 10,
"state_continuous": True, "probes_fork_and_do_not_advance_idle": True},
})
plan = {
"schema": SCHEMA, "status": "DRAFT-NOT-REGISTERED", "master": master,
"execution_authorized": False, "brain_inputs_generated": False,
"training_executed": False, "shuffle_algorithm": SHUFFLE_VERSION,
"shuffle_specification": _Shuffle.__doc__.strip(),
"input_seed_contract": "full63bit integer; no uint32 truncation, offset, modulo or hidden global RNG",
"consumer_generator_implementation_verified": False,
"fixed_odors": dict(ODORS), "dt_ms": 0.1,
"training_replicates_per_cell": 5, "blocks": blocks,
"alias_policy": "training CS only; same block/assay/cue/occurrence; named controls only",
"scheduled_US_rng": "none; deterministic pulse; design shuffle randomness is separately registered",
}
counts, steps = Counter(), 0
cs_epochs, us_epochs, effective_updates = 0, 0, 0
for block in blocks:
for suite in block["suites"]:
for arm in suite["arms"]:
counts[(suite["assay"], arm["arm"])] += len(arm["epochs"])
for epoch in arm["epochs"]:
steps += epoch["steps"]
cs_epochs += epoch["cs"] is not None
us_epochs += epoch["us"]["requested"]
effective_updates += epoch["update_enabled"] and epoch["D_command"] != 0
budget = {
"blocks": 40, "training_replicates_per_cell": 5,
"epoch_counts_by_assay_arm": [{"assay": a, "arm": b, "epochs": count}
for (a, b), count in sorted(counts.items())],
"training_control_reversal_epochs": sum(counts.values()),
"CS_epochs": cs_epochs, "US_request_epochs": us_epochs,
"enabled_nonzero_D_update_opportunities": effective_updates,
"training_control_reversal_steps": steps,
"training_control_reversal_biological_seconds": steps // 10000,
"retention_idle_seconds": 40 * 2 * 10,
"retention_idle_steps": 40 * 2 * 10 * 10000,
"minimum_biological_seconds_excluding_evaluation": steps // 10000 + 800,
"minimum_steps_excluding_evaluation": steps + 8000000,
"evaluation_epochs": None, "evaluation_biological_seconds": None,
"evaluation_probe_inventory_complete": False,
"excludes": ["evaluation probes", "implementation validation", "frozen regressions", "I/O and encoding wall time"],
"reason_not_total_runtime": "Only training/control/reversal and continuous retention idle are budgeted; never label this complete total exposure",
}
_require(budget["training_control_reversal_epochs"] == 68000 and steps == 99600000,
"Internal protocol inventory arithmetic changed")
result = {"plan": plan, "seed_manifest": registry.manifest(), "budget": budget}
result["payload_sha256"] = _sha(result)
return result
def _validate_alias_permissions(manifest):
for row in manifest["aliases"]:
key, source = row["key"], row["source_key"]
_require(row["split"] == row["source_split"] == "training",
"Only named training CS controls may alias streams")
_require(len(key) == len(source) == 6 and key[2] == source[2] == "CS",
"CS/US/design/evaluation role aliasing is forbidden")
_require(key[:5] == source[:5], "Alias crosses block/replicate/assay/cue/occurrence")
if key[1] in ("normal", "delayed"):
_require(source[5] == "paired" and key[5] in CONTROL_ALIASES,
"Unknown normal/delayed control alias")
else:
_require(key[1] == "reversal" and source[5] == "reversal" and key[5] == "no_reversal",
"Unknown reversal control alias")
def validate_trial_plan(payload):
"""Fail closed on incomplete coverage, illegal aliasing, mutation or types.
Rebuild the whole fixed 40-block schedule and every RNG derivation. This
does not trust a count, a caller's declared sha, or a row labeled replicate.
Canonical comparison distinguishes Boolean/integer/floating-point JSON.
"""
_require(type(payload) is dict and set(payload) == {"plan", "seed_manifest", "budget", "payload_sha256"},
"Invalid complete trial-plan envelope")
actual = {key: value for key, value in payload.items() if key != "payload_sha256"}
_require(type(payload["payload_sha256"]) is str and payload["payload_sha256"] == _sha(actual),
"Actual trial-plan payload SHA mismatch")
seed_registry.validate_manifest(payload["seed_manifest"])
_validate_alias_permissions(payload["seed_manifest"])
_require(payload["plan"]["master"] == payload["seed_manifest"]["master"], "Master identities differ")
expected = _assemble(payload["plan"]["master"])
_require(_canonical(payload) == _canonical(expected),
"Full protocol coverage/input binding/order/type reconstruction mismatch")
return True
def build_trial_plan(master):
"""Return a deterministic JSON-serializable DRAFT inventory without I/O."""
return _assemble(master)