Seal Phase 4AN exact continuation engineering evidence

This commit is contained in:
Digital Drosophila Archive
2026-09-30 16:46:21 +08:00
parent de61cf14b1
commit aeccb7d0cc
628 changed files with 127078 additions and 1 deletions
@@ -0,0 +1,116 @@
# Phase 4AN GPU 工程验证门草案
状态:**未注册 DRAFT;不是执行授权,尚未运行本门。** 日期:2026-09-30。
本门只验证新的轻记录执行器和连续状态接口是否保存冻结 D18 动力学。它不是 conditioning、retention 学习结论,也不产生 learned candidate。AM 是否已正式完成、其最终状态和 seal SHA,必须在本门预注册时从真实证据核验,本文不代填成功状态或哈希。
## 1. 固定范围与最小库存
只使用一个 AM 已准备的 100 ms 全脑输入,分别安装其原有 `zero` 与 `static25` 固定 overlay。选择规则先于本门任何新 GPU 输出:从 AM `protocol/preregistered.json` 的原 `planned` 顺序取第一条 `repeat == 0` 的记录,保存其完整行、序号、case 路径及 SHA;不得按活动强弱、旧 static25 效果或运行快慢挑 case。其原始 case 实际位于 AK `cases/<cohort>/<name>`;AM `run_suite.py` 没有另造一套 case。若预注册者希望改变此规则,应先修订草案并锁定,不得在看新输出后换输入。
每个 overlay 固定以下库存,所有行均使用同一原始 1000 tick 请求序列或其精确前缀/分段:
| 行组 | 固定执行内容 | 新执行调用数/overlay |
|---|---|---:|
| Fresh 对照 | 完整 1000 tick reference r0、light r0、light r1 | 3 |
| 连续链 | reference、light 各一条,段长 `[5, 13, 1, 481, 500]` | 10 |
| 切点前缀 | 从初态独立运行 reference 至绝对 tick 数 18、19、500 | 3 |
合计 32 次短执行,11,074 个全脑 tick,即累计 1.1074 生物秒;不是 68,000 个训练 epoch。两条链的第一段均为 fresh 5 tick,其中 reference 第一段兼作第一个切点的前缀参照,不重复运行。切点为累计 tick 数 5、18、19、500,终点为 1000:覆盖短于 18 tick 的父/子段、在绝对 tick 18 的首次合法延迟到达、19-slot 环的相位,以及较晚的活动边界。不得把部分库存完成称作完整门通过。
固定参数:166,700 neurons、25,582,938 edges、`dt_ms=0.1`、18 tick synaptic delay、19 行 spike ring、18 行 graded-APL ring、LIF float64 SI 状态与旧 APL float32 操作。保留 14,232 mask、5845 fast-APL 出边置零规则及原 APL 全形矩阵。`static25` 是 AM 原 float32-mV 候选基值的固定 `-0.25 * base` overlay,经原 SI bridge 安装;不是声称 float64 原权重每项恰减 25%。两条 overlay 分别与自己的 AM 参照比较,不能要求 zero 与 static25 相等。
禁止:weight updates、eligibility 推进/应用、eta 改动、DAN/US 新请求、odor/seed 新抽样、动力学调参、训练、额外 tail ticks。这里的连续链只检验工程续接,不检验记忆保留。
## 2. 当前真实接口与输入派生
以下是底层 Python API;本门的独立 coordinator 使用它们,不改旧模块 globals:
```python
runtime_light.run_light(case_dir, out_dir, build_root, overlay_file, overlay_manifest)
runtime_light.run_reference(case_dir, out_dir, build_root, overlay_file, overlay_manifest)
runtime_continuation.seal_checkpoint(checkpoint_dir)
runtime_continuation.load_checkpoint(checkpoint_dir, expected_sha)
runtime_continuation.run_continued(
case_dir, out_dir, build_root, overlay_file, overlay_manifest,
checkpoint_dir, mode='light' # 或 'reference'
)
```
本门的 comparator、有限库存 coordinator、预注册/执行 receipt 已在独立 engineering 模块实现;必须在服务器通过纯合成测试、完成只读代码审查并锁定全部源码后,才能注册执行。现有 AM comparator 对 AM adapter/provenance 和完整 1000 tick 有硬 schema 约束,不应直接冒充 AN comparator 或修改其旧 globals。
在新的独占输入目录派生 spec,不改 AK/AM 原文件。完整 fresh case 保留原 `name`、model、APL、图、系数和请求;增加显式 `max_steps=1000`,仅调整新的输出预算根及已列明的非科学标签。要求 `record_history=False`。新旧 spec 的逐字段差异必须列入 receipt,不能用“大致相同”或整类忽略代替。
前缀保留原请求中 `tick < end` 的行。分段 `[start, stop)` 只保留该区间原请求,局部 tick 精确减 `start`;全局 neuron_idx 不变。每段 `steps=stop-start`、`duration_ms=steps*0.1`、`max_steps=1000`。将各段请求恢复绝对 tick 后拼接,必须与原请求两个 int32 数组逐位相同、顺序相同、无丢失/重复;不能重抽随机输入。NPZ 容器 SHA 可以因新文件而不同,但数组语义必须 exact,并同时保留文件 SHA。
第一段完成后立即 `seal_checkpoint`;外部 coordinator 把返回的实际 manifest SHA 绑定到下一段新 spec 的 `parent_checkpoint_manifest_sha256`,可同时写准确 `global_tick_offset`。此 SHA 是运行后产出的数据依赖,预注册应锁定派生规则而不是预造值。后续 `run_continued` 自身会产生并封存 checkpoint;父 SHA 仍须外部绑定。fresh seal 是 `state.npz` 的首次新增封存绑定,不能声称它追溯证明 seal 前文件未改:应在同一受控流程中立即比较、封存,保存前后 inventory,期间不允许第三方写入。
AN overlay manifest 使用现有 `phase4an-overlay-v1` 精确键:`schema, overlay_name, overlay_sha256, mask_path, mask_sha256, base_candidate_path, base_candidate_sha256, aj_model_dir, am_overlay_path, am_overlay_sha256`。复用 AM 原 overlay 数组字节及实际 SHA,仅新建 manifest;不能把 AM 原 manifest 的额外字段直接塞给 AN,也不使用 `learned` 模式。
## 3. 预注册必须完整绑定的证据
在任何本门 GPU 执行前,保存不可覆盖的 input/source inventory 与其外部 SHA。至少包括:
- AM 最终状态、model/experiment seal、`protocol/preregistered.json`、`prepared-hashes.json`、所选原 case spec/requests,以及 `gpu/zero/<name>-r0`、`gpu/static25/<name>-r0` 的完整 `output-hashes.json`、comparison、adapter provenance 与全部待比较 artifact。核验完整 AM seal,而非只相信单个 `pass_all`。
- 图的 `nodes/edge_id/src/dst/ptr/signed_raw/weights.npy` 全部文件;APL `input_matrix/output_matrix/kc_globals/apl_globals.npy`;AF 原 `coefficients.json`;AM `protocol/base-candidate.float32-mV.npy`、两个 overlay NPZ/manifest、原 H mask。图和 bundle 的所有 SHA 来自外部锚定的 AM prereg inputs,不是仅由 AN 本次读入后自报。
- AN `runtime_light.py`、`runtime_continuation.py`、对应 tests、即将实现的本门 coordinator/comparator/tests,以及本草案最终注册版。固定其实际 SHA 和源 AST 审计;不得在跑到一半后修改源代码继续使用同一 prereg。
- `runtime_light.PINS` 的整个依赖链:AJ GPU/timing queue、AG-r2 GPU/legacy APL worker、AE backend/C++/CUDA、AF queue helper,以及 `AM_OVERLAY_SHA` 对应原 `overlay.py`。记录 `runtime_continuation.LIGHT_SHA` 与实际 light 文件一致。运行前后都重验旧来源与静态输入,不依赖时间戳缓存来替代字节哈希。
- 新输入每份 spec/requests/manifest 的 SHA,原始记录到前缀/分段的确定性映射,以及库版本、科学 Python 原 venv 入口、GPU UUID/driver/CUDA/Torch/NumPy、编译标志和 extension 来源。Python 用 `-B`;不可通过 resolve venv executable 改变包环境。Torch import 前完成 GPU/CUDA 环境设置;仅记录并核对原 TF32/CuBLAS/确定性配置,不为追求一致或速度偷偷改默认值。
AM 已完成封存并取得小证据下载包;该下载包不含全部大型轨迹,因此预注册仍必须在服务器完整校验 AM 两个 seal。实际 case 名与 seal SHA 依照上述确定规则写入预注册,不以本文代填。缺失 authority、旧输入变化或 AM 不完整时停止,不进入 GPU 门。
## 4. Exact 比较,不把摘要布尔当证明
所有浮点比较按原 dtype 的连续 little-endian raw bits,包括正负零;禁止 tolerance、round、只比均值或把压缩 NPZ 文件字节相等误作数组相等。所有状态要求 finite,事件、计数和请求库存必须完整。
### Fresh light/reference ↔ 同 overlay AM
逐数组比 `state.npz` 六键:`step/neuron_idx/counts/ref` 为 int32,`v/g` 为 float64;全部 N 个末态和全部稀疏事件相同,events 自身排序唯一且 bincount 等于 counts。light 与 reference 都保留 `requests.csv`,其五字段 `tick, neuron_idx, attempted, accepted, rejected` 必须与 AM 原输出逐行 exact;不能只比较请求总数。reference 另比每 tick `v_sha256/g_sha256/ref_sha256`、全部 APL drive/state/generated inhibition/applied h float32、pending tail 和每 tick queue。来源标签、新 spec SHA、记录范围和计时等确有差异的字段须显式白名单,不能整份忽略 provenance。
AN 新 `terminal-queues.npz` 没有对应 AM 原文件时,只从 AM 已记录事实独立重建:1000 tick 最后 19 tick 的全局 sparse events 决定 raw spike ring;末行 `apl-state.float32.npy` 决定 APL state;原 `apl-pending-tail.float32.npy` 决定 chronological pending。比较全部数组及 `next_tick=1000`,不补跑 AM。
light 没有全 APL trace、逐 tick V/g/ref hash 或 queue.jsonl;不能把这些缺文件当空记录、不能称 light 已做这些逐 tick 对照。但 `ReportingOnly.visit_With` 只移除 queue/hash 的两个 context item,留下 `request_output`,也保留 `writer.writerows` 和请求接受判定的 GPU→CPU 拷贝。因此 light 的门包括完整 events/counts/终态、逐行五字段请求接受记录、APL state、完整两种 pending queue、有限值覆盖、总到达/接受计数及固定权重身份与 reference exact;不能把它描述为完全无逐请求记录或完全无逐 tick CPU 同步。reference 承担完整逐 tick 状态/队列观测。light r1 要与 r0 的科学数组、请求 CSV、终态队列及有限值/总量一致,计时不是 exact 字段。
### Monolithic ↔ segmented
每个 overlay、每种模式分别比自己的完整 1000 tick fresh 基线,不跨 overlay 借结果。拼接所有段的 sparse events,并按 receipt offset 恢复绝对 tick;逐段 counts 用 int64 求和、验证范围后与单体 counts exact。末态 V/g/ref、最终 APL state/pending、effective weights raw SHA 必须一致;每个中间切点还与相同绝对长度的 fresh reference 前缀末态对照。切点 5 使用已记录的首段 reference,切点 18/19/500 使用预定前缀,终点使用完整基线。
spike ring 不能直接比较物理行号:各自按 `terminal.next_tick % 19` 转成“下一写 slot 起”的 19 行顺序后逐位比较;continued 的 `next_tick` 是本段局部相位,绝对时间来自 receipt。APL pending 已按下一次应用顺序保存,直接逐位比较;不得把它当未来脑活动或继续衰减的完整尾迹。短段未覆盖的 spike slots 和未消费 APL prefix 还须精确等于父 carry。
reference 的全部逐 tick 状态 hashes 和 APL 四种 trace 按全局时间拼接后与 monolithic 对照;不重复/跳过边界 tick。`queue.jsonl` 的 `joint_sha256` 编码了局部 tick,不能直接跨段相等比较:按全局 tick 对齐,精确比较 `count, accepted_count, ordered_edge_sha256, accepted_mask_sha256`;用完整事件、原 CSR、原 ref 规则独立重建原 source-order queue,并分别验证每份文件自己的 local-header joint hash。不能对 queue 重新按 target 排序后才声称原队列相同。**light 和 reference 两条链的请求 CSV 均须**按 offset 还原 tick、保持其余四字段和行序,拼接后与各自 monolithic CSV 逐行 exact;同时与预定请求库存对应,不能漏掉拒收请求。
上述门可证明 reference 完整记录和 light 已记录状态的一致;**不能据此声称不可观测的 light 每 tick 内部 V/g bits 已被直接逐项比较。** 若正式注册要求这一更强结论,须先单独批准新只读观测器和相应 source pin,而不是运行后扩大结论。
## 5. 防真空和故障门
每个 overlay 都必须从当前 reference 实际记录证明:事件非空;generated 与 applied APL h 至少有非零样本;至少一个预定内部切点同时出现非空 spike carry 和非零 APL pending。还须保存至少一个真实跨边界 synaptic witness:源 spike 在边界之前、`arrival_tick=source_spike_tick+18` 位于边界之后且小于 1000;其真实 canonical edge 权重非零、到达被接受,并包含在原队列哈希重建中。记录 boundary、source/target、edge_id、spike/arrival tick、接受标志及权重,不把这个工程 witness 解释为单一生物学原因。
这些仅是“确实测试过 live carry”的覆盖条件,不是调参阈值。不得强迫所有边界都有活动,也不得用空队列 exact 当作 live-carry PASS。固定 case/切点不满足时,结果为 coverage INCOMPLETE,保存全部输出;不可临时加 pulse、选另一个效果强的 case 或后移边界救场。任何扩展需新草案修订及新 prereg,明确看过既有结果。
故障检查以纯临时合成测试完成:错误外部 SHA、祖先 state 篡改、局部/绝对时钟错配、未写 ring slot 或未消费 APL prefix 改值、输入数组 dtype/NaN/Inf、重复/遗漏事件、拒收被错算接受、输出/编译目录越入任一祖先或冻结输入、已有目录重用。不要故意破坏 AM 或 GPU 正式输出来造负例。
## 6. 计时、压缩与磁盘预算
建议注册一个独立新实验输出根,新增 artifact/build 合计硬上限 **8 GiB**,启动前另保留 **10 GiB** 空闲余量;这是待批准的工程资源上限,不是神经科学门。所有图/APL/旧输出仅只读引用,不复制大型图或全脑 dense history。每次运行前后核实已写字节和剩余空间;不足则停止并保留 partial,不删证据续跑,也不缩短注册 tick 数。
32 次调用共享新的 extension build cache,但不共享可变神经/APL状态,只有声明的父 checkpoint 传状态。记录每次 wall time、已有 `diagnostic_simulation_seconds`、build/cache 时间、完整校验/压缩在 wall 中的费用及峰值 VRAM。reference r0 先行、light r0/r1 随后;明确首次编译/文件缓存顺序造成的影响,不能把一次差值称为普适性能 benchmark。可报告实测 100 ms trial 的耗时/压缩字节,不以此承诺 conditioning 大库存的总成本。
压缩格式按实际来源逐项核对:当前固定 SHA 的 AJ `gpu_timing.py::run_gpu` 写 `state.npz` 的语句是 `np.savez_compressed(target, **state)`,不是 `np.savez`;AN 的 light/reference AST 派生没有替换该语句。`terminal-queues.npz` 则由 AN `_Runtime.finish` 明确使用 `np.savez_compressed`。记录各数组 uncompressed `nbytes`、这两种 NPZ 的实际压缩 bytes、全部 JSONL/CSV/NPY 及 build bytes,并在执行 receipt 核验各 NPZ entry 的实际压缩方式;不得仅凭 `.npz` 扩展名判断。不能只按 NPZ 小体积漏掉 reference APL 原始 `.npy`。reference 四种 APL trace 的主体约为每 tick 32,528 bytes;完整 1000 tick 约 32.528 MB,尚未含事件/末态/队列/头部。所有估计与最终实际 inventory 同报,不预称已节省多少磁盘;不改原输出格式或移除证据以追求预算。
## 7. 执行 receipt 与停止结论
新 coordinator 须 exclusive 写每行执行 receipt 和全门 inventory:注册 SHA、行 ID/overlay/mode/repeat/区间、case/requests/代码输入 SHA、进程退出码、完整 stdout/stderr、开始/结束记录、实际 completed ticks、原始 output inventory SHA、独立 comparator 结果/输入 SHA、finite coverage、weight identity、父 checkpoint 路径/外部 SHA、全局时钟、计时及磁盘字节。单独的 `summary.complete/pass_all` 不足:它可能先于 wrapper 后置核验失败写出;必须有完整 runtime provenance、无 error/failure marker,并重新验证 input/old integrity。
checkpoint manifest、continuation receipt 和外部执行 inventory 各自按现有排除自身的规则封存,不能自哈希循环。新比较报告和 coordinator receipt 放在被封存 checkpoint 目录之外,避免为加报告改坏父链。恢复只可跳过经完整库存/输入/独立比较重验的已完成行;存在 partial、故障 marker 或不完整祖先则停止,不覆盖、不自动科学重试。
全部库存、exact 比较、live-carry 覆盖和前后 integrity 均通过时,最高结论仅为“固定 zero/static25 的 GPU 记录模式与连续状态工程等价门通过”;`learning_pass_claimed=False`、`candidate_generated=False`、`weights_updated_during_run=False`。数值不一致为 FAIL;缺证据、预算中断或 live-carry 未覆盖为 INCOMPLETE;不把未执行项记 PASS。任何结果都不自动授权 conditioning,后续训练协议、独立输入/种子/控制与科学门仍需另行审定。
## 8. 未来可另行注册的补充,不属于当前库存
以下来自独立审查者的工程提案,**不自动追加到本门、不授权执行,也不是当前 32 次调用的 PASS 条件**。需要分别固定新输入、代码/比较器、父 checkpoint 来源和资源预算后重新审定;不可把本草案悄然扩为 90 次调用。
第一项是独立全图 synthetic live-carry 短矩阵。可从冻结 mask 的最低 canonical KC source 及其最低 global edge ID 的真实 MBON target 定义 K、M;从冻结 APL input matrix 的非零列按预定 canonical 规则选 Q,不按新输出选人。拟定 80 tick 的两种请求表:K 在 0/1/22/44/66;必要时 Q 在 0;另一表额外 M 在 18,以覆盖到达接收/不应期拒收、child local tick 0 请求及 tick 84 尚未到期的 spike tail。Q 的实际身份和非零列须静态核验,不能仅凭前缀猜 PN/KC;预期 live/accept/reject 条件也须由实际记录证明。zero/static25 × 两表,各比 reference monolithic 与 reference/light 分段 `[5,12,1,1,18,43]`,提案合计 52 次短调用、960 tick(0.096 生物秒)。它测试人工工程请求,不是新增 odor conditioning;本门固定 AM case 的覆盖不足不能未经新注册就用该矩阵救成 PASS。
第二项是在上述另注册短矩阵通过并得到明确绑定的 active checkpoint 后,再注册较长的连续 idle 工程门:zero/static25 各比较 light 一段 100,000 tick 与连续 10,000 + 90,000 tick,均零新输入、零权重更新,提案合计 6 次 continuation 调用。比较全部 sparse events、分段 counts 总和、终态 V/g/ref/APL state 及规范化 queues;不记录 10 秒逐 tick CPU hash 时,不能宣称逐 tick 浮点状态已独立对照。1 秒 checkpoint 沿同一 idle 主链续到 10 秒;任何测试 probe 必须另 fork,不能把 probe 插入主轨迹,其调用数和预算另列。所有祖先 manifest 在 fork 前后不变。即使通过,也仅证明较长时钟/状态接口,不是记忆保持或 learning PASS。
源码依据:本目录 `runtime_light.py` 的 `validate_spec/load_overlay/derive_ast/_Runtime.finish/_run`;`runtime_continuation.py` 的 `rotate_spike_ring/validate_state/load_checkpoint/run_continued/audit_carried_sequence`;封存 AM `prepare.py/run_suite.py/adapter.py/comparison.py`;封存 AJ `gpu_timing.py` 与 AF `oracle_full.py::queue_summary`。本文只读这些来源,未运行 GPU、未注册或修改其内容。
@@ -0,0 +1,373 @@
"""Deterministic, read-only CPU analysis of the registered 32-call AN gate.
analyze(reg, reg_sha, verified_records) accepts the actual runtime receipts
already verified by engineering_workflow.verify_execution. It does not run
that workflow, publish, seal, save, retry or import a GPU runtime. All external
AM seals/checkpoint chains/process receipts remain the caller's prior duty.
This module rehashes the files it reads, executes every canonical descriptor,
and independently reconstructs the reference queues and effective SI weights.
"""
from __future__ import annotations
import hashlib
import json
import math
from pathlib import Path
import types
import numpy as np
import engineering_compare as compare
import engineering_plan as plan_module
import engineering_queue_audit as queue_audit
import engineering_scopes as scopes
N, E, K = 166700, 25582938, 4064
CANDIDATES, FAST_APL_EDGES = 14232, 5845
AM_OVERLAY_SHA = '7f891d0ba79b393c96e75fd2dee2eddc51e57d8213eaac2779703bc2c18712b4'
MASK_SHA = '8b0fb6eb2e6c31da4d17dc20d890aa3f367e514f05642de5bad6b42a1611c05b'
SUCCESS = 'PASS-EXACT-LIGHT-CONTINUATION-ENGINEERING-NOT-LEARNING'
def require(value, message):
if not value:
raise ValueError(message)
def _canonical(value):
return json.dumps(value, sort_keys=True, separators=(',', ':'), ensure_ascii=False, allow_nan=False)
def _sha_file(path):
h = hashlib.sha256()
with Path(path).open('rb') as stream:
for data in iter(lambda: stream.read(8 << 20), b''):
h.update(data)
return h.hexdigest()
class _Reader:
"""Fresh byte hashes, not stat caching; deterministic read manifest."""
def __init__(self, registered):
self.registered = registered
self.reads = {}
self.arrays = []
def bind(self, path, *, registered=True, expected=None):
path = Path(path)
require(path.is_absolute() and path.is_file() and not path.is_symlink(), 'Missing/linked/relative evidence: ' + str(path))
path = path.resolve(strict=True)
if registered:
require(str(path) in self.registered, 'Unregistered dependency: ' + str(path))
require(expected is None or expected == self.registered[str(path)], 'Conflicting external hash authority')
expected = self.registered[str(path)]
actual = _sha_file(path)
require(expected is None or (compare._sha(expected) and actual == expected), 'Evidence bytes changed: ' + str(path))
value = dict(path=str(path), bytes=path.stat().st_size, sha256=actual)
require(str(path) not in self.reads or self.reads[str(path)] == value, 'File changed during analysis')
self.reads[str(path)] = value
return path
def json(self, path, **kwargs):
return compare._json(self.bind(path, **kwargs))
def npy(self, path):
value = np.load(self.bind(path), mmap_mode='r', allow_pickle=False)
self.arrays.append(value)
return value
def npz(self, path, keys):
with np.load(self.bind(path), allow_pickle=False) as archive:
require(len(archive.files) == len(keys) and set(archive.files) == set(keys), 'Unexpected input NPZ keys')
return {name: archive[name].copy() for name in keys}
def run(self, directory, *, legacy):
directory = Path(directory)
names = ['summary.json', 'state.npz', 'apl-timing-audit.json', 'apl-pending-tail.float32.npy', 'requests.csv']
if legacy:
names.append('adapter-provenance.json')
else:
names.append('terminal-queues.npz')
names += [name for name in [*compare.TRACE_FILES.values(), 'tick-state-hashes.jsonl', 'queue.jsonl']
if (directory / name).is_file()]
for name in names:
self.bind(directory / name, registered=legacy)
return compare.load_run(directory, allow_legacy_am=legacy)
def finish(self):
for row in self.reads.values():
path = Path(row['path'])
require(path.is_file() and path.stat().st_size == row['bytes'] and _sha_file(path) == row['sha256'],
'Read evidence changed during analysis: ' + str(path))
return [self.reads[key] for key in sorted(self.reads)]
def close(self):
for value in self.arrays:
mmap = getattr(value, '_mmap', None)
if mmap is not None:
mmap.close()
def _registration(reg, reg_sha, verified):
require(type(reg) is dict and reg.get('schema') == 'phase4an-engineering-registration-v1' and
reg.get('training') is False and compare._sha(reg_sha), 'Invalid engineering registration')
plan = plan_module.validate_plan(reg['plan'])
ids = [row['row_id'] for row in plan['rows']]
require(type(verified) is dict and set(verified) == set(ids) and set(reg['cases']) == set(ids),
'Exactly all 32 registered runtime receipts/cases are required')
exp = None
for row in plan['rows']:
identity = row['row_id']; path = Path(reg['cases'][identity]['case_dir'])
require(path.is_absolute() and path.name == identity and path.parent.name == 'cases', 'Invalid registered case layout')
if exp is None:
exp = path.parent.parent
require(path == exp / 'cases' / identity, 'Cases span inconsistent experiment roots')
receipt = verified[identity]
require(receipt.get('schema') == 'phase4an-engineering-runtime-record-v1' and receipt.get('complete') is True and
receipt.get('training') is False and receipt.get('registration_sha256') == reg_sha and
_canonical(receipt.get('row')) == _canonical(row) and compare._sha(receipt.get('checkpoint_manifest_sha256')),
'Runtime receipt is not the exact verified registered row: ' + identity)
require(exp.name == 'phase4an-engineering-validation' and exp.parent.name == 'experiments', 'Unexpected fixed engineering output root')
require(set(reg['overlays']) == set(reg['selected_case']['AM_outputs']) == {'zero', 'static25'}, 'Incomplete overlay/AM authority')
return plan, exp
def _materialize(reader, reg, graph, stage):
"""Use the exact pinned pure AM helper; no original globals are patched."""
declaration = reg['overlays'][stage]
meta = reader.json(declaration['overlay_manifest'])
require(meta.get('schema') == 'phase4an-overlay-v1' and meta.get('overlay_name') == stage and
meta.get('am_overlay_sha256') == AM_OVERLAY_SHA and meta.get('mask_sha256') == MASK_SHA,
'Original fixed overlay authority changed')
helper_path = reader.bind(meta['am_overlay_path'], expected=AM_OVERLAY_SHA)
module = types.ModuleType('phase4an_analysis_pinned_overlay')
module.__file__ = str(helper_path)
exec(compile(helper_path.read_bytes(), str(helper_path), 'exec'), module.__dict__)
mask = reader.npy(meta['mask_path']); base_candidate = reader.npy(meta['base_candidate_path'])
require(reader.reads[str(Path(meta['mask_path']).resolve())]['sha256'] == meta['mask_sha256'] and
reader.reads[str(Path(meta['base_candidate_path']).resolve())]['sha256'] == meta['base_candidate_sha256'],
'Overlay external mask/base hash mismatch')
overlay_file = reader.bind(declaration['overlay_file'], expected=meta['overlay_sha256'])
arrays = reader.npz(overlay_file, {'edge_id', 'plastic_delta_mv'})
require(mask.dtype == np.int64 and mask.shape == (CANDIDATES,), 'Complete frozen plastic mask required')
values = module.validate_arrays(arrays['edge_id'], arrays['plastic_delta_mv'], mask, base_candidate)
if stage == 'zero':
require(not np.any(values['plastic_delta_mv'] != 0), 'Zero overlay is nonzero')
else:
require(module.bit_equal(values['plastic_delta_mv'], np.multiply(np.float32(-.25), base_candidate)),
'static25 differs from the original float32-mV floor')
effective, mapping = module.materialize_numpy(graph['weights'], graph['edge_id'], graph['apl_out_mask'], values)
require(mapping['mapped_candidate_count'] == CANDIDATES and mapping['missing_candidate_count'] == 0 and
mapping['apl_overlap_count'] == 0, 'Incomplete/overlapping candidate mapping')
base_sha, effective_sha = module.expected_hashes(graph['weights'], graph['apl_out_mask'], mapping)
require(queue_audit._raw_sha(effective) == effective_sha, 'CPU materialization/hash disagreement')
return effective, dict(effective_weights_raw_sha256=effective_sha, base_weights_raw_sha256=base_sha,
candidate_count=CANDIDATES, installed_nonzero_count=mapping['installed_nonzero_count'],
apl_fast_edges_zeroed=FAST_APL_EDGES, bridge=module.BRIDGE,
helper_source_sha256=AM_OVERLAY_SHA, weights_updated_during_run=False)
def _load_context(reg, reg_sha, plan, exp):
reader = _Reader(reg['input_sha256'])
records, legacy = {}, {}
try:
actual_reg = reader.json(exp / 'protocol/preregistered.json', registered=False, expected=reg_sha)
require(_canonical(actual_reg) == _canonical(reg), 'Supplied registration differs from actual registered bytes')
for path, expected in sorted(reg['source_sha256'].items()):
reader.bind(path, registered=False, expected=expected)
original = Path(reg['selected_case']['original_dir'])
spec = reader.json(original / 'spec.json')
require(spec.get('kind') == 'fullgraph' and spec.get('steps') == 1000 and spec.get('dt_ms') == .1 and
spec.get('apl_delay_steps') == 18, 'Original fixed fullgraph D18 spec changed')
requests = reader.npz(original / 'requests.npz', {'tick', 'neuron_idx'})
graph_dir, bundle = Path(spec['graph_dir']), Path(spec['apl_bundle_dir'])
graph = {key: reader.npy(graph_dir / (key + '.npy')) for key in ('nodes', 'ptr', 'dst', 'edge_id', 'weights')}
require(graph['nodes'].dtype == np.int32 and np.array_equal(graph['nodes'], np.arange(N, dtype=np.int32)),
'Graph is not the complete canonical neuron set')
n, e = queue_audit._graph(graph['ptr'], graph['dst'], graph['edge_id'], graph['weights'])
require((n, e) == (N, E), 'Original graph dimensions changed')
apl = reader.npy(bundle / 'apl_globals.npy')
require(apl.dtype == np.int32 and apl.shape == (2,) and np.all((apl >= 0) & (apl < N)) and
apl[0] < apl[1], 'Invalid canonical APL source identity')
mask = np.zeros(E, np.bool_)
for source in apl:
mask[graph['ptr'][source]:graph['ptr'][source + 1]] = True
require(int(mask.sum()) == FAST_APL_EDGES, 'Frozen fast-APL source-edge mask changed')
graph['apl_out_mask'] = mask
storage = {}
for row in plan['rows']:
identity = row['row_id']; folder = exp / 'runs' / identity
require(folder.is_dir() and folder.resolve() == folder and not any(p.is_symlink() for p in folder.rglob('*')),
'Missing/linked runtime evidence folder')
records[identity] = reader.run(folder, legacy=False)
storage[identity] = sum(p.stat().st_size for p in folder.rglob('*') if p.is_file())
for stage in ('zero', 'static25'):
legacy[stage] = reader.run(reg['selected_case']['AM_outputs'][stage], legacy=True)
return dict(reader=reader, records=records, legacy=legacy, graph=graph, requests=requests,
storage=storage, input_scope='read-only authenticated selected original inputs and AM outputs')
except BaseException:
for record in [*records.values(), *legacy.values()]:
compare.close_run(record)
reader.close()
raise
def _pair(candidate, reference):
checks, observed = compare._pair_checks(candidate, reference)
return dict(numerical_exact=all(checks.values()), checks=checks,
failed_checks=[key for key, value in checks.items() if not value],
reference_comparisons_available=observed, learning_pass_claimed=False)
def _dispatch(descriptor, records, legacy, overlay):
candidates = [records[name] for name in descriptor['candidate_row_ids']]
reference = records[descriptor['reference_row_id']] if descriptor['reference_row_id'] is not None else None
kind = descriptor['check']
if kind == 'am_exact':
return compare.compare_fresh(candidates[0], legacy[overlay])
if kind == 'fresh_exact':
return compare.compare_fresh(candidates[0], reference)
if kind in ('prefix_of_reference', 'segment_of_reference'):
return scopes.compare_slice(candidates[0], reference)
if kind == 'segment_exact':
return _pair(candidates[0], reference)
if kind == 'checkpoint_exact':
return scopes.compare_checkpoint(candidates[0], reference)
if kind == 'parent_carry':
return scopes.compare_parent_carry(candidates[0], reference)
if kind == 'chain_exact':
return compare.compare_segments(reference, candidates)
raise ValueError('Unimplemented registered comparison: ' + str(kind))
def _number(value, name, integer=False):
if value is None:
return None
require((type(value) is int if integer else type(value) in (int, float)) and math.isfinite(value) and value >= 0,
'Malformed observed metric: ' + name)
return value
def _run_checks(row, record, expected_hash, reference):
compare.validate_run(record)
summary = record['summary']; identity = row['row_id']
require(len(record['state']['v']) == N and record['terminal']['apl_pending'].shape == (18, K), 'Incomplete full brain state: ' + identity)
checks = {key: summary.get(key) is True for key in ('complete', 'pass_all', 'all_finite', 'counts_match_sparse_events')}
fixed = dict(nodes=N, edges=E, steps=row['steps'], completed_ticks=row['steps'], apl_delay_steps=18,
requested=len(record['requests']), total_spikes=len(record['state']['step']),
accepted_requests=int(record['requests'][:, 3].sum(dtype=np.int64)),
rejected_requests=int(record['requests'][:, 4].sum(dtype=np.int64)),
apl_fast_edges_zeroed=FAST_APL_EDGES, kc_targets_present=K)
for key, value in fixed.items():
checks[key] = type(summary.get(key)) is int and summary[key] == value
checks['registered_clock'] = record['steps'] == row['steps'] and record['global_tick_offset'] == row['start']
checks['recording_mode'] = summary.get('diagnostic_mode') == row['mode']
checks['effective_weight_bytes'] = summary.get('effective_weights_raw_sha256') == expected_hash
checks['within_trial_weights_fixed'] = summary.get('within_trial_weight_updates') is False
checks['learning_not_claimed'] = summary.get('learning_pass_claimed') is False
expected_queue = reference['queues'][row['start']:row['stop']]
for field, queue_key in [('arriving_edges', 'count'), ('accepted_arriving_edges', 'accepted_count')]:
expected = sum(q[queue_key] for q in expected_queue)
checks[field] = type(summary.get(field)) is int and summary[field] == expected
return {name: bool(value) for name, value in checks.items()}
def analyze(reg, reg_sha, verified_records):
"""No writes/runs. The caller rechecks registration and receipts afterward."""
plan, exp = _registration(reg, reg_sha, verified_records)
context = _load_context(reg, reg_sha, plan, exp)
records, legacy = context['records'], context['legacy']
try:
require(set(records) == {r['row_id'] for r in plan['rows']} and set(legacy) == {'zero', 'static25'},
'Loaded evidence inventory is incomplete')
comparisons = []
for row in plan['rows']:
for number, descriptor in enumerate(row['expected_comparisons']):
try:
result = _dispatch(descriptor, records, legacy, row['overlay'])
require(type(result.get('numerical_exact')) is bool, 'Comparator lacks an explicit Boolean numerical gate')
except (ValueError, KeyError, TypeError, IndexError, OverflowError) as error:
result = dict(numerical_exact=False, error_type=type(error).__name__, error=str(error),
incomplete_observation_not_fabricated=True)
comparisons.append(dict(descriptor_id=row['row_id'] + '/' + str(number),
descriptor=descriptor, result=result))
expected_descriptors = sum(len(r['expected_comparisons']) for r in plan['rows'])
require(len(comparisons) == expected_descriptors, 'Not every registered descriptor was evaluated')
overlay_reports, metric_rows, all_row_checks = {}, [], {}
for stage in ('zero', 'static25'):
effective, installation = _materialize(context['reader'], reg, context['graph'], stage)
stage_rows = [r for r in plan['rows'] if r['overlay'] == stage]
ref_row = next(r for r in stage_rows if r['kind'] == 'fresh' and r['mode'] == 'reference')
reference = records[ref_row['row_id']]
reference_chain = [records[r['row_id']] for r in stage_rows if r['kind'] == 'chain' and r['mode'] == 'reference']
cuts = [r['start'] for r in stage_rows if r['kind'] == 'chain' and r['mode'] == 'reference' and r['start'] > 0]
independent = queue_audit.audit_queues(reference, reference_chain, ptr=context['graph']['ptr'], dst=context['graph']['dst'],
edge_id=context['graph']['edge_id'], effective_weights=effective, requests=context['requests'], cuts=cuts)
del effective
reference_result = next(item['result'] for item in comparisons if item['descriptor']['check'] == 'chain_exact' and
item['descriptor']['reference_row_id'] == ref_row['row_id'])
light_result = next(item['result'] for item in comparisons if item['descriptor']['check'] == 'chain_exact' and
item['descriptor']['reference_row_id'] == plan_module._fresh_id(stage, 'light'))
coverage = reference_result.get('coverage', {})
apl_values = coverage.get('carried_APL_nonzero_applied_values')
simultaneous_carry = any(type(row.get('pending_source_spike_events')) is int and row['pending_source_spike_events'] > 0 and
type(row.get('carried_APL_nonzero_pending_values')) is int and row['carried_APL_nonzero_pending_values'] > 0
for row in coverage.get('boundaries', []))
coverage_pass = (reference_result.get('nonvacuous_temporal_coverage') is True and
type(apl_values) is int and apl_values > 0 and simultaneous_carry and
independent.get('accepted_nonzero_crossing_witness_verified') is True)
legacy_checks = dict(full_dimensions=type(legacy[stage]['summary'].get('nodes')) is int and legacy[stage]['summary']['nodes'] == N and
type(legacy[stage]['summary'].get('edges')) is int and legacy[stage]['summary']['edges'] == E,
effective_weight_bytes=legacy[stage]['summary'].get('effective_weights_raw_sha256') == installation['effective_weights_raw_sha256'])
overlay_reports[stage] = dict(installation=installation, independent_queue_audit=independent,
reference_numerical_exact=reference_result.get('numerical_exact') is True,
light_numerical_exact=light_result.get('numerical_exact') is True,
live_coverage_pass=coverage_pass, legacy_checks=legacy_checks,
same_internal_boundary_has_spike_and_APL_carry=simultaneous_carry,
carried_APL_nonzero_applied_values=apl_values,
light_standalone_coverage=light_result.get('coverage'),
light_per_tick_float_state_directly_observed=False,
reference_coverage_does_not_fabricate_light_observations=True)
for row in stage_rows:
identity = row['row_id']; record = records[identity]; summary = record['summary']; receipt = verified_records[identity]
checks = _run_checks(row, record, installation['effective_weights_raw_sha256'], reference)
all_row_checks[identity] = checks
metric_rows.append(dict(row_id=identity, overlay=stage, mode=row['mode'], start=row['start'], stop=row['stop'],
biological_time_ms=row['steps'] * .1, runtime_and_immediate_seal_wall_seconds=_number(receipt.get('runtime_and_immediate_seal_wall_seconds'), 'runtime wall'),
runtime_reported_wall_seconds=_number(summary.get('wall_seconds'), 'runtime reported wall'),
diagnostic_simulation_seconds=_number(summary.get('diagnostic_simulation_seconds'), 'simulation'),
build_or_cache_seconds=_number(summary.get('build_or_cache_seconds'), 'build/cache'),
peak_allocated_vram_bytes=_number(summary.get('peak_allocated_vram_bytes'), 'allocated VRAM', True),
peak_reserved_vram_bytes=_number(summary.get('peak_reserved_vram_bytes'), 'reserved VRAM', True),
sealed_output_file_bytes=context['storage'][identity], compressed_storage=receipt.get('storage'),
gpu_utilization=None, cpu_utilization=None, unobserved_metrics=['gpu_utilization', 'cpu_utilization'],
checkpoint_manifest_sha256=receipt['checkpoint_manifest_sha256'],
verified_runtime_receipt_canonical_sha256=hashlib.sha256(_canonical(receipt).encode('utf-8')).hexdigest()))
numerical = (all(item['result']['numerical_exact'] for item in comparisons) and
all(all(checks.values()) for checks in all_row_checks.values()) and
all(all(r['legacy_checks'].values()) and r['independent_queue_audit']['queue_ref_request_exact'] for r in overlay_reports.values()))
covered = all(row['live_coverage_pass'] for row in overlay_reports.values())
read_manifest = context['reader'].finish()
report = dict(schema='phase4an-engineering-analysis-v1', registration_sha256=reg_sha,
status='FAIL-NUMERICAL' if not numerical else (SUCCESS if covered else 'INCOMPLETE-COVERAGE'),
pass_all=bool(numerical and covered), numerical_exact=bool(numerical), nonvacuous_temporal_coverage=bool(covered),
expected_calls=32, analyzed_calls=len(records), executed_ticks=plan['total_ticks'],
expected_comparisons=expected_descriptors, evaluated_comparisons=len(comparisons), comparisons=comparisons,
run_checks=all_row_checks, failed_run_checks={identity: [k for k, ok in checks.items() if not ok] for identity, checks in all_row_checks.items() if not all(checks.values())},
overlay_reports=overlay_reports, metrics=metric_rows, read_manifest=read_manifest,
source_sha256=reg['source_sha256'], process_receipts_and_checkpoint_chains_verified_by_caller=True,
external_AM_full_seals_reverified_here=False, no_new_brain_runs_during_analysis=True,
GPU_used=False, training=False, learning_pass_claimed=False, candidate_generated=False,
physiology_or_memory_success_claimed=False,
limitations=['Fixed zero/static25 engineering trajectories only; not conditioning or learned memory.',
'Light intermediate V/g/ref, APL histories and per-tick queue arrays were not directly recorded.',
'Independent queue/ref reconstruction is conditional on recorded spikes; V/g dynamics are not independently re-simulated.',
'Missing performance metrics are null, not zero. No universal performance claim is made.',
'Caller verifies all process receipts, parent chains, registration authority and full AM/frozen seals.'])
_canonical(report) # Fail rather than return non-JSON/nonfinite output.
return report
finally:
for record in [*records.values(), *legacy.values()]:
compare.close_run(record)
context['reader'].close()
@@ -0,0 +1,413 @@
"""Read-only CPU comparisons for the UNREGISTERED AN engineering gate.
load_run(path, allow_legacy_am=False) reads actual AJ/AM/AN artifact schemas.
Call close_run(record) after using explicitly loaded memory-mapped traces.
compare_fresh(left, right, legacy_am=None) and compare_segments(monolithic,
segments) accept loaded dictionaries or paths and return JSON-safe reports.
No report is written, no GPU is imported, no training/biological PASS is claimed.
The caller owns external source/input/root SHA pins and case selection. These
comparisons do not replace seal verification or prove unseen inputs unchanged.
AM did not save its raw spike ring: its fresh terminal ring is explicitly
reconstructed from sparse events, never described as independently recorded.
Reference queue joint hashes include LOCAL tick headers; cross-segment checks
compare the separate ordered-edge/accepted-mask hashes and counts instead.
"""
from __future__ import annotations
import csv
import hashlib
import json
from pathlib import Path
import re
import numpy as np
STATE_KEYS = ('step', 'neuron_idx', 'counts', 'v', 'g', 'ref')
REQUEST_FIELDS = ('tick', 'neuron_idx', 'attempted', 'accepted', 'rejected')
TRACE_FILES = {'drive': 'apl-drive.float32.npy', 'state': 'apl-state.float32.npy',
'generated': 'apl-inhibition.float32.npy', 'applied': 'apl-applied.float32.npy'}
HASH_FIELDS = ('v_sha256', 'g_sha256', 'ref_sha256')
QUEUE_FIELDS = ('count', 'accepted_count', 'ordered_edge_sha256', 'accepted_mask_sha256')
def require(ok, message):
if not ok:
raise ValueError(message)
def bit_equal(a, b):
return (isinstance(a, np.ndarray) and isinstance(b, np.ndarray) and a.dtype == b.dtype and a.shape == b.shape and
np.array_equal(np.ascontiguousarray(a).view(np.uint8), np.ascontiguousarray(b).view(np.uint8)))
def _sha(value):
return isinstance(value, str) and re.fullmatch('[0-9a-f]{64}', value) is not None
def _json(path):
def pairs(values):
result = {}
for key, value in values:
require(key not in result, 'Duplicate JSON key: ' + key)
result[key] = value
return result
return json.loads(Path(path).read_text(encoding='utf-8'), object_pairs_hook=pairs,
parse_constant=lambda value: (_ for _ in ()).throw(ValueError('Nonfinite JSON: ' + value)))
def _jsonl(path):
with Path(path).open(encoding='utf-8') as stream:
return [json.loads(line) for line in stream if line.strip()]
def _npz(path, keys):
with np.load(path, allow_pickle=False) as archive:
require(len(archive.files) == len(keys) and set(archive.files) == set(keys), 'Unexpected array keys: ' + str(path))
return {key: archive[key].copy() for key in keys}
def reconstructed_fresh_ring(state, steps, neurons):
ring = np.zeros((19, neurons), np.uint8)
for tick in range(max(0, steps - 19), steps):
first, last = np.searchsorted(state['step'], (tick, tick + 1))
ring[tick % 19, state['neuron_idx'][first:last]] = 1
return ring
def normalized_ring(terminal):
ring, phase = terminal['spike_ring'], int(terminal['next_tick'])
return ring[(np.arange(19) + phase % 19) % 19]
def _requests(path):
with Path(path).open(encoding='utf-8', newline='') as stream:
reader = csv.DictReader(stream)
require(tuple(reader.fieldnames or ()) == REQUEST_FIELDS, 'requests.csv must use the exact five AJ fields in order')
rows = []
for row in reader:
require(set(row) == set(REQUEST_FIELDS), 'Extra/missing request CSV field')
rows.append([int(row[key]) for key in REQUEST_FIELDS])
return np.asarray(rows, dtype=np.int64).reshape(-1, 5)
def load_run(directory, *, allow_legacy_am=False):
directory = Path(directory).resolve(strict=True)
require(directory.is_dir(), 'Output is not a directory')
require(not any('error' in p.name.lower() or 'failure' in p.name.lower() for p in directory.iterdir()),
'Run contains error/failure evidence')
summary = _json(directory / 'summary.json')
require(summary.get('complete') is True and summary.get('pass_all') is True and summary.get('all_finite') is True and
summary.get('steps') == summary.get('completed_ticks'), 'Run is incomplete or failed')
state = _npz(directory / 'state.npz', STATE_KEYS)
steps, neurons = summary['steps'], len(state['v'])
require(summary.get('nodes') == neurons and summary.get('precision') == 'float64' and
summary.get('apl_precision') == 'float32' and summary.get('apl_delay_steps') == 18, 'Unexpected AN/AM numerical schema')
timing = _json(directory / 'apl-timing-audit.json')
pending = np.load(directory / 'apl-pending-tail.float32.npy', mmap_mode='r', allow_pickle=False)
terminal_path = directory / 'terminal-queues.npz'
if terminal_path.is_file():
terminal = _npz(terminal_path, ('spike_ring', 'apl_state', 'apl_pending', 'next_tick'))
origin = 'recorded_terminal_queues'
else:
require(allow_legacy_am and (directory / 'adapter-provenance.json').is_file(), 'Recorded terminal queues are missing')
bits = timing.get('final_apl_state_u32_bits')
require(isinstance(bits, list) and len(bits) == 2 and all(type(v) is int and 0 <= v < 2**32 for v in bits),
'AM has no exact terminal APL bits')
terminal = {'spike_ring': reconstructed_fresh_ring(state, steps, neurons),
'apl_state': np.asarray(bits, dtype=np.uint32).view(np.float32), 'apl_pending': np.asarray(pending),
'next_tick': np.asarray(steps, np.int64)}
origin = 'AM_spike_ring_derived_from_fresh_events_APL_bits_from_timing'
require(bit_equal(terminal['apl_pending'], pending), 'Terminal/pending file mismatch')
require(timing.get('final_apl_state_u32_bits') == terminal['apl_state'].view(np.uint32).tolist(), 'Terminal APL bit metadata mismatch')
continued = summary.get('checkpoint_continuation_used', False)
require(type(continued) is bool, 'Invalid continuation scope flag')
offset = summary.get('global_tick_offset', 0)
require(not continued or (type(offset) is int and offset > 0 and summary.get('global_next_tick') == offset + steps),
'Continued output lacks an explicit global clock')
require(continued or offset == 0, 'Fresh output has a nonzero global offset')
traces = {name: np.load(directory / file, mmap_mode='r', allow_pickle=False)
for name, file in TRACE_FILES.items() if (directory / file).is_file()}
reference = summary.get('diagnostic_mode') == 'reference' or origin.startswith('AM_')
if reference or summary.get('full_APL_history_recorded') is True:
require(len(traces) == 4, 'Declared reference APL history is incomplete')
hashes = _jsonl(directory / 'tick-state-hashes.jsonl') if (directory / 'tick-state-hashes.jsonl').is_file() else None
queues = _jsonl(directory / 'queue.jsonl') if (directory / 'queue.jsonl').is_file() else None
if reference or summary.get('full_tick_state_hashes_recorded') is True:
require(hashes is not None, 'Declared state hashes are missing')
if reference or summary.get('full_queue_hashes_recorded') is True:
require(queues is not None, 'Declared queue hashes are missing')
result = {'source': str(directory), 'state': state, 'terminal': terminal, 'requests': _requests(directory / 'requests.csv'),
'steps': steps, 'global_tick_offset': offset, 'summary': summary, 'traces': traces,
'state_hashes': hashes, 'queues': queues, 'terminal_origin': origin}
validate_run(result)
return result
def close_run(run):
"""Release read-only mappings opened by load_run; no files are modified."""
arrays = [*run.get('traces', {}).values(), *run.get('terminal', {}).values()]
closed = set()
for array in arrays:
current = array
while current is not None:
mapping = getattr(current, '_mmap', None)
if mapping is not None and id(mapping) not in closed:
mapping.close(); closed.add(id(mapping))
current = getattr(current, 'base', None)
def validate_run(run):
"""Array-level contract, also usable by synthetic tests without files."""
require(isinstance(run, dict), 'Run must be a dictionary')
steps, offset = run['steps'], run['global_tick_offset']
require(type(steps) is int and steps > 0 and type(offset) is int and offset >= 0, 'Invalid segment clock')
state, terminal = run['state'], run['terminal']
require(set(state) == set(STATE_KEYS), 'Expected exactly the six state keys')
n = len(state['v'])
require(n > 0, 'Empty neuron population')
for name, dtype in (('v', np.float64), ('g', np.float64), ('ref', np.int32), ('counts', np.int32)):
value = state[name]
require(value.dtype == dtype and value.shape == (n,) and np.isfinite(value).all(), 'Invalid state ' + name)
require(np.all((state['ref'] >= 0) & (state['ref'] <= 22)) and np.all(state['counts'] >= 0), 'Invalid ref/count values')
tick, node = state['step'], state['neuron_idx']
require(tick.dtype == node.dtype == np.int32 and tick.ndim == node.ndim == 1 and tick.shape == node.shape and
np.all((tick >= 0) & (tick < steps)) and np.all((node >= 0) & (node < n)), 'Invalid events')
keys = tick.astype(np.int64) * n + node
require(len(keys) < 2 or np.all(keys[1:] > keys[:-1]), 'Events must be sorted unique')
require(np.array_equal(np.bincount(node, minlength=n), state['counts']), 'Counts disagree with segment events')
request = run['requests']
require(isinstance(request, np.ndarray) and request.dtype == np.int64 and request.ndim == 2 and request.shape[1] == 5,
'Requests must be int64 rows in the five AJ fields')
require(np.all((request[:, 0] >= 0) & (request[:, 0] < steps)) and np.all((request[:, 1] >= 0) & (request[:, 1] < n)) and
np.all(request[:, 2] == 1) and np.all((request[:, 3] == 0) | (request[:, 3] == 1)) and
np.all(request[:, 4] == 1 - request[:, 3]), 'Invalid request values')
request_keys = request[:, 0] * n + request[:, 1]
require(len(request_keys) < 2 or np.all(request_keys[1:] > request_keys[:-1]), 'Requests must be sorted unique')
require(set(terminal) == {'spike_ring', 'apl_state', 'apl_pending', 'next_tick'}, 'Invalid terminal keys')
ring = terminal['spike_ring']
require(ring.dtype == np.uint8 and ring.shape == (19, n) and np.all(ring <= 1), 'Invalid spike ring')
phase = terminal['next_tick']
require(isinstance(phase, np.ndarray) and phase.dtype == np.int64 and phase.shape == () and int(phase) == steps,
'Terminal phase must be local segment length')
pending, apl_state = terminal['apl_pending'], terminal['apl_state']
require(pending.dtype == np.float32 and pending.ndim == 2 and pending.shape[0] == 18 and pending.shape[1] > 0 and
apl_state.dtype == np.float32 and apl_state.shape == (2,) and np.isfinite(pending).all() and np.isfinite(apl_state).all(),
'Invalid APL terminal state')
for t in range(max(0, steps - 19), steps):
expected = np.zeros(n, np.uint8)
first, last = np.searchsorted(tick, (t, t + 1)); expected[node[first:last]] = 1
require(bit_equal(ring[t % 19], expected), 'Written ring slot differs from sparse events')
if offset == 0 and steps < 19:
require(not np.any(ring[steps:]), 'Fresh uninitialized ring slots are not zero')
traces = run.get('traces', {})
require(not traces or set(traces) == set(TRACE_FILES), 'Partial APL trace set')
for key, trace in traces.items():
width = 2 if key in ('drive', 'state') else pending.shape[1]
require(trace.dtype == np.float32 and trace.shape == (steps, width) and np.isfinite(trace).all(), 'Invalid APL history: ' + key)
if traces:
require(bit_equal(traces['state'][-1], apl_state), 'Final APL history/state mismatch')
kept = min(steps, 18)
require(bit_equal(pending[-kept:], traces['generated'][-kept:]), 'Terminal APL queue does not retain generated suffix')
if steps > 18:
require(bit_equal(traces['applied'][18:], traces['generated'][:-18]), 'Within-segment APL delay differs from 18')
if offset == 0:
require(np.all(traces['applied'][:min(18, steps)].view(np.uint32) == 0), 'Fresh APL application padding is not positive zero')
hashes, queues = run.get('state_hashes'), run.get('queues')
if hashes is not None:
require(len(hashes) == steps, 'State hash coverage incomplete')
for t, row in enumerate(hashes):
require(set(row) == {'tick', *HASH_FIELDS} and type(row['tick']) is int and row['tick'] == t and
all(_sha(row[key]) for key in HASH_FIELDS), 'Invalid state hash row')
for name in ('v', 'g', 'ref'):
raw = np.ascontiguousarray(state[name], dtype=state[name].dtype.newbyteorder('<')).tobytes()
require(hashes[-1][name + '_sha256'] == hashlib.sha256(raw).hexdigest(), 'Last state hash differs from terminal ' + name)
if queues is not None:
require(len(queues) == steps, 'Queue coverage incomplete')
for t, row in enumerate(queues):
require(set(row) == {'tick', *QUEUE_FIELDS, 'joint_sha256'} and type(row['tick']) is int and row['tick'] == t and
type(row['count']) is int and type(row['accepted_count']) is int and 0 <= row['accepted_count'] <= row['count'] and
all(_sha(row[key]) for key in ('ordered_edge_sha256', 'accepted_mask_sha256', 'joint_sha256')), 'Invalid queue row')
return run
def _run(value, legacy=False):
return load_run(value, allow_legacy_am=legacy) if isinstance(value, (str, Path)) else validate_run(value)
def _activity(run):
terminal = run['terminal']
result = {'spikes': len(run['state']['step']), 'requested': len(run['requests']),
'accepted_requests': int(run['requests'][:, 3].sum(dtype=np.int64)),
'terminal_apl_state_nonzero_values': int(np.count_nonzero(terminal['apl_state'])),
'terminal_pending_nonzero_values': int(np.count_nonzero(terminal['apl_pending']))}
traces = run.get('traces', {})
result['generated_h_nonzero_values'] = int(np.count_nonzero(traces['generated'])) if traces else None
result['applied_h_nonzero_values'] = int(np.count_nonzero(traces['applied'])) if traces else None
result['apl_nonzero_observed'] = bool(result['terminal_apl_state_nonzero_values'] or result['terminal_pending_nonzero_values'] or
(result['generated_h_nonzero_values'] or 0) or (result['applied_h_nonzero_values'] or 0))
return result
def _pair_checks(left, right, prefix='pair'):
checks = {prefix + '.steps': left['steps'] == right['steps'], prefix + '.offset': left['global_tick_offset'] == right['global_tick_offset']}
for name in STATE_KEYS:
checks[prefix + '.state.' + name] = bit_equal(left['state'][name], right['state'][name])
checks[prefix + '.requests_five_fields'] = bit_equal(left['requests'], right['requests'])
for name in ('spike_ring', 'apl_state', 'apl_pending', 'next_tick'):
checks[prefix + '.terminal.' + name] = bit_equal(left['terminal'][name], right['terminal'][name])
for field in ('effective_weights_raw_sha256',):
a, b = left.get('summary', {}).get(field), right.get('summary', {}).get(field)
if a is not None or b is not None:
checks[prefix + '.' + field] = _sha(a) and a == b
availability = {'APL_traces': bool(left.get('traces')) and bool(right.get('traces')),
'state_hashes': left.get('state_hashes') is not None and right.get('state_hashes') is not None,
'queue_hashes': left.get('queues') is not None and right.get('queues') is not None}
if availability['APL_traces']:
for name in TRACE_FILES:
checks[prefix + '.trace.' + name] = bit_equal(left['traces'][name], right['traces'][name])
if availability['state_hashes']:
checks[prefix + '.state_hashes'] = left['state_hashes'] == right['state_hashes']
if availability['queue_hashes']:
checks[prefix + '.queues'] = left['queues'] == right['queues']
return checks, availability
def compare_fresh(left, right, legacy_am=None):
left, right = _run(left), _run(right)
require(left['global_tick_offset'] == right['global_tick_offset'] == 0, 'Fresh comparison requires fresh runs')
checks, available = _pair_checks(left, right)
origins = [left.get('terminal_origin', 'caller_arrays'), right.get('terminal_origin', 'caller_arrays')]
if legacy_am is not None:
legacy = _run(legacy_am, legacy=True)
require(legacy['global_tick_offset'] == 0, 'AM control must be fresh')
extra, legacy_available = _pair_checks(left, legacy, 'AM')
checks.update(extra); available['AM'] = legacy_available
# The left run may be light. Do not thereby skip the independent full
# diagnostic comparison between the right reference and sealed AM.
extra, reference_available = _pair_checks(right, legacy, 'reference_vs_AM')
checks.update(extra); available['reference_vs_AM'] = reference_available
origins.append(legacy.get('terminal_origin', 'caller_arrays'))
activity = _activity(left)
exact = bool(all(checks.values()))
nonvacuous = activity['spikes'] > 0 and activity['apl_nonzero_observed']
return {'schema': 'phase4an-fresh-engineering-comparison-v1', 'numerical_exact': exact,
'nonvacuous_activity': nonvacuous, 'pass_all': exact and nonvacuous,
'status': 'FAIL-NUMERICAL' if not exact else ('PASS-FRESH-SCOPE' if nonvacuous else 'INCOMPLETE-COVERAGE'),
'checks': checks, 'failed_checks': [key for key, value in checks.items() if not value], 'activity': activity,
'reference_comparisons_available': available, 'terminal_origins': origins,
'cross_segment_validation_performed': False, 'temporal_validation_complete': False,
'scope': 'Fresh event/state/request/terminal identity only; no continuation, learning, physiology or performance PASS',
'independent_CSR_queue_reconstruction_performed': False,
'accepted_nonzero_weight_edge_witness_verified': False,
'external_seals_and_input_authority_verified_here': False}
def compare_segments(monolithic, segments):
mono = _run(monolithic)
parts = [_run(value) for value in segments]
require(len(parts) >= 2, 'Need a monolithic run and at least two segments')
origin = mono['global_tick_offset']
if origin:
parent = mono.get('summary', {}).get('parent_checkpoint_manifest_sha256')
require(_sha(parent) and parts[0].get('summary', {}).get('parent_checkpoint_manifest_sha256') == parent,
'Nonzero-origin comparisons must bind the same starting parent checkpoint')
cursor = origin
for part in parts:
require(part['global_tick_offset'] == cursor, 'Segment clock gap/overlap/order error')
require(len(part['state']['v']) == len(mono['state']['v']), 'Segment population changed')
cursor += part['steps']
require(cursor == origin + mono['steps'], 'Segments do not cover the monolithic duration')
checks = {}
step = np.concatenate([part['state']['step'].astype(np.int64) + part['global_tick_offset'] for part in parts])
neuron = np.concatenate([part['state']['neuron_idx'] for part in parts])
checks['concatenated_global_events'] = np.array_equal(step, mono['state']['step'].astype(np.int64) + origin) and bit_equal(neuron, mono['state']['neuron_idx'])
counts = np.zeros_like(mono['state']['counts'], dtype=np.int64)
for part in parts:
counts += part['state']['counts'].astype(np.int64)
checks['sum_segment_counts'] = np.array_equal(counts, mono['state']['counts'].astype(np.int64))
requests = []
for part in parts:
rows = part['requests'].copy(); rows[:, 0] += part['global_tick_offset']; requests.append(rows)
mono_requests = mono['requests'].copy(); mono_requests[:, 0] += origin
checks['concatenated_requests_five_fields'] = bit_equal(np.concatenate(requests), mono_requests)
last = parts[-1]
for name in ('v', 'g', 'ref'):
checks['terminal.' + name] = bit_equal(last['state'][name], mono['state'][name])
checks['terminal.normalized_spike_ring'] = bit_equal(normalized_ring(last['terminal']), normalized_ring(mono['terminal']))
for name in ('apl_state', 'apl_pending'):
checks['terminal.' + name] = bit_equal(last['terminal'][name], mono['terminal'][name])
expected_weight = mono.get('summary', {}).get('effective_weights_raw_sha256')
if expected_weight is not None:
checks['unchanged_effective_weights'] = _sha(expected_weight) and all(part.get('summary', {}).get('effective_weights_raw_sha256') == expected_weight for part in parts)
trace_available = bool(mono.get('traces')) and all(part.get('traces') for part in parts)
hash_available = mono.get('state_hashes') is not None and all(part.get('state_hashes') is not None for part in parts)
queue_available = mono.get('queues') is not None and all(part.get('queues') is not None for part in parts)
if trace_available:
for name in TRACE_FILES:
checks['concatenated_APL.' + name] = all(bit_equal(part['traces'][name], mono['traces'][name][part['global_tick_offset'] - origin:part['global_tick_offset'] - origin + part['steps']]) for part in parts)
if hash_available:
checks['concatenated_per_tick_state_hashes'] = all(all(row[name] == mono['state_hashes'][part['global_tick_offset'] - origin + tick][name] for name in HASH_FIELDS)
for part in parts for tick, row in enumerate(part['state_hashes']))
if queue_available:
checks['concatenated_queue_order_and_acceptance'] = all(all(row[name] == mono['queues'][part['global_tick_offset'] - origin + tick][name] for name in QUEUE_FIELDS)
for part in parts for tick, row in enumerate(part['queues']))
boundary_rows = []
events = mono['state']['step'].astype(np.int64) + origin
for before, part in zip(parts, parts[1:]):
cut, stop = part['global_tick_offset'], part['global_tick_offset'] + part['steps']
emitted_before = (events < cut) & (events + 18 >= cut)
arriving_this_segment = emitted_before & (events + 18 < stop)
pending = before['terminal']['apl_pending']
consume = min(part['steps'], 18)
spike_carry = normalized_ring(before['terminal'])
# Future local arrival tick j consumes normalized slot j+1. This also
# covers spikes inherited from before a nonzero monolithic origin,
# which are legitimately absent from that run's local event log.
pending_spikes = int(spike_carry[1:].sum(dtype=np.int64))
arriving_spikes = int(spike_carry[1:consume + 1].sum(dtype=np.int64))
if cut - origin >= 18:
checks['boundary.' + str(cut) + '.ring_pending_event_count'] = pending_spikes == int(emitted_before.sum())
# Slots not overwritten by this short segment must remain bit-identical.
if part['steps'] < 19:
checks['boundary.' + str(cut) + '.unwritten_spike_slots'] = bit_equal(part['terminal']['spike_ring'][part['steps']:], spike_carry[part['steps']:])
if part['steps'] < 18:
checks['boundary.' + str(cut) + '.unconsumed_APL'] = bit_equal(part['terminal']['apl_pending'][:18 - part['steps']], pending[part['steps']:])
if part.get('traces'):
checks['boundary.' + str(cut) + '.applied_parent_APL'] = bit_equal(part['traces']['applied'][:consume], pending[:consume])
queue_rows = (part['queues'][:consume] if part.get('queues') is not None else
mono['queues'][cut - origin:cut - origin + consume] if mono.get('queues') is not None else None)
boundary_rows.append({'global_tick': cut, 'next_segment_steps': part['steps'],
'pending_source_spike_events': pending_spikes, 'source_spikes_arriving_in_next_segment': arriving_spikes,
'visible_local_log_source_spikes_arriving_in_next_segment': int(arriving_this_segment.sum()),
'pending_source_spike_origin': 'recorded normalized ring slots; includes pre-origin carry',
'observed_carried_arriving_edges': sum(row['count'] for row in queue_rows) if queue_rows is not None else None,
'observed_carried_accepted_edges': sum(row['accepted_count'] for row in queue_rows) if queue_rows is not None else None,
'carried_APL_nonzero_pending_values': int(np.count_nonzero(pending)),
'carried_APL_nonzero_values_applied_in_next_segment': int(np.count_nonzero(pending[:consume])),
'carried_APL_nonzero_rows_applied_in_next_segment': int(np.count_nonzero(np.any(pending[:consume] != 0, axis=1)))})
source_arrivals = sum(row['source_spikes_arriving_in_next_segment'] for row in boundary_rows)
edge_counts = [row['observed_carried_arriving_edges'] for row in boundary_rows]
observed_edges = sum(value for value in edge_counts if value is not None) if any(value is not None for value in edge_counts) else None
applied_values = sum(row['carried_APL_nonzero_values_applied_in_next_segment'] for row in boundary_rows)
activity = _activity(mono)
coverage = {'boundaries': boundary_rows, 'source_spikes_crossing_boundaries_and_due_in_receiver': source_arrivals,
'observed_carried_arriving_edges': observed_edges, 'carried_APL_nonzero_applied_values': applied_values,
'boundary_pending_event_counts_may_overlap_across_short_segments': True,
'actual_edge_arrival_coverage_known': observed_edges is not None}
nonvacuous = bool(activity['spikes'] > 0 and activity['apl_nonzero_observed'] and source_arrivals > 0 and
observed_edges is not None and observed_edges > 0 and applied_values > 0)
exact = bool(all(checks.values()))
return {'schema': 'phase4an-segment-engineering-comparison-v1', 'numerical_exact': exact,
'nonvacuous_temporal_coverage': nonvacuous, 'temporal_validation_complete': exact and nonvacuous,
'pass_all': exact and nonvacuous,
'status': 'FAIL-NUMERICAL' if not exact else ('PASS-SEGMENT-SCOPE' if nonvacuous else 'INCOMPLETE-COVERAGE'),
'checks': checks, 'failed_checks': [key for key, value in checks.items() if not value],
'activity': activity, 'coverage': coverage, 'segments': len(parts), 'steps': mono['steps'], 'global_tick_offset': origin,
'reference_comparisons_available': {'APL_traces': trace_available, 'state_hashes': hash_available, 'queue_hashes': queue_available},
'queue_joint_hash_compared': False, 'queue_joint_hash_exclusion_reason': 'local tick header differs; ordered-edge/mask hashes and counts compared when available',
'scope': 'Explicit finite engineering trajectory/terminal/carry checks only; no learning, physiology, or performance PASS',
'independent_CSR_queue_reconstruction_performed': False,
'accepted_nonzero_weight_edge_witness_verified': False,
'external_seals_and_input_authority_verified_here': False}
@@ -0,0 +1,152 @@
"""Pure exact input derivations for a future preregistered engineering gate.
No I/O, random draws, scientific execution or registration. The coordinator
must bind the original case/overlay and every static dependency to the sealed
AM manifest before using these helpers. A self-consistent dictionary is not
independent input authority. Derivation outputs are copies; validate_requests
only checks and returns its original, unmodified input.
"""
from __future__ import annotations
import copy
import re
from pathlib import PurePosixPath
import numpy as np
N = 166700
AM_HELPER_SHA = '7f891d0ba79b393c96e75fd2dee2eddc51e57d8213eaac2779703bc2c18712b4'
MASK_SHA = '8b0fb6eb2e6c31da4d17dc20d890aa3f367e514f05642de5bad6b42a1611c05b'
def require(ok, message):
if not ok:
raise ValueError(message)
def _positive_int(value, name):
require(type(value) is int and 0 < value <= np.iinfo(np.int32).max, 'Invalid ' + name)
def _sha(value):
return type(value) is str and re.fullmatch('[0-9a-f]{64}', value) is not None
def _absolute(value):
return (type(value) is str and value.startswith('/') and not value.startswith('//') and
PurePosixPath(value).is_absolute() and
'..' not in PurePosixPath(value).parts)
def validate_requests(requests, steps):
_positive_int(steps, 'steps')
require(type(requests) is dict and set(requests) == {'tick', 'neuron_idx'}, 'Exact sparse request schema required')
for key, value in requests.items():
require(isinstance(value, np.ndarray) and value.dtype == np.int32 and value.ndim == 1,
'Request requires int32 vector: ' + key)
tick, neuron = requests['tick'], requests['neuron_idx']
require(tick.shape == neuron.shape, 'Request shape mismatch')
require(np.all((tick >= 0) & (tick < steps)) and np.all((neuron >= 0) & (neuron < N)), 'Request outside fixed trial/canonical set')
key = tick.astype(np.int64) * N + neuron
require(len(key) < 2 or np.all(key[1:] > key[:-1]), 'Requests must be sorted and unique; no implicit deduplication')
return requests
def slice_requests(requests, steps, start, stop):
"""Half-open exact slice; translate ticks only, never redraw or reorder."""
validate_requests(requests, steps)
require(type(start) is int and type(stop) is int and 0 <= start < stop <= steps, 'Invalid half-open input interval')
lo, hi = np.searchsorted(requests['tick'], (start, stop))
result = {'tick': (requests['tick'][lo:hi] - np.int32(start)).copy(),
'neuron_idx': requests['neuron_idx'][lo:hi].copy()}
validate_requests(result, stop - start)
return result
def verify_partition(original, steps, intervals, parts):
"""Verify complete coverage plus exact request reconstruction, even blanks."""
validate_requests(original, steps)
require(type(intervals) in (list, tuple) and type(parts) in (list, tuple) and
len(intervals) == len(parts) > 0, 'Partition needs all intervals and parts')
end, reconstructed = 0, {'tick': [], 'neuron_idx': []}
for interval, part in zip(intervals, parts):
require(type(interval) in (list, tuple) and len(interval) == 2, 'Invalid interval schema')
start, stop = interval
require(type(start) is int and type(stop) is int and start == end and start < stop <= steps,
'Partition gap/overlap/nonpositive interval')
validate_requests(part, stop - start)
expected = slice_requests(original, steps, start, stop)
require(all(part[k].tobytes() == expected[k].tobytes() for k in part), 'Partition changes actual requests')
reconstructed['tick'].append(part['tick'].astype(np.int64) + start)
reconstructed['neuron_idx'].append(part['neuron_idx'])
end = stop
require(end == steps, 'Partition omits the terminal interval')
require(np.array_equal(np.concatenate(reconstructed['tick']), original['tick']) and
np.array_equal(np.concatenate(reconstructed['neuron_idx']), original['neuron_idx']), 'Reconstructed input mismatch')
return {'complete': True, 'intervals': [list(x) for x in intervals], 'steps': steps,
'requests': len(original['tick']), 'random_draws': 0, 'input_reselected': False}
def derive_spec(original, *, start, stop, budget_root, output_budget_bytes, parent_sha=None):
"""Explicit new case admission/budget labels; frozen model fields unchanged.
New budget_root must be outside the OLD budget tree. Full source/data/
ancestor containment checks remain the runtime/coordinator's responsibility.
A nonzero interval requires the actual parent manifest's external SHA;
future parent hashes must not be fabricated during preregistration.
"""
require(type(original) is dict and original.get('kind') == 'fullgraph', 'Original fullgraph spec required')
steps = original.get('steps')
_positive_int(steps, 'original steps')
require(original.get('dt_ms') == .1 and original.get('duration_ms') == steps * .1 and
type(original.get('apl_delay_steps')) is int and original['apl_delay_steps'] == 18 and
original.get('record_history') is False and original.get('training') is False and
original.get('within_trial_weight_updates', False) is False, 'Original fixed D18 trial contract required')
require(not any('checkpoint' in k.lower() or k in ('global_tick_offset', 'resume', 'continuation', 'initial_state')
for k in original), 'Original must be a fresh case')
require(type(start) is int and type(stop) is int and 0 <= start < stop <= steps, 'Invalid case interval')
require(_absolute(budget_root) and _absolute(original.get('budget_root')), 'Explicit absolute budget roots required')
new, old = PurePosixPath(budget_root), PurePosixPath(original['budget_root'])
require(not new.is_relative_to(old) and not old.is_relative_to(new), 'New output budget must not overlap old experiment tree')
require(type(output_budget_bytes) is int and output_budget_bytes > 0, 'Explicit positive byte budget required')
require((start == 0 and parent_sha is None) or (start > 0 and _sha(parent_sha)), 'Only a continued interval needs a real parent SHA')
result = copy.deepcopy(original)
result.update(steps=stop - start, duration_ms=(stop - start) * .1, max_steps=steps,
budget_root=budget_root, output_budget_bytes=output_budget_bytes,
within_trial_weight_updates=False,
outer_protocol='phase4an-engineering-validation-no-training')
if start:
result.update(global_tick_offset=start, parent_checkpoint_manifest_sha256=parent_sha)
changes = {key: {'present_before': key in original, 'before': copy.deepcopy(original.get(key)),
'after': copy.deepcopy(value)} for key, value in result.items()
if key not in original or value != original[key]}
allowed = {'steps', 'duration_ms', 'max_steps', 'budget_root', 'output_budget_bytes',
'within_trial_weight_updates', 'outer_protocol', 'global_tick_offset',
'parent_checkpoint_manifest_sha256'}
require(set(changes) <= allowed and all(result[k] == v for k, v in original.items() if k not in allowed),
'Unexpected scientific spec edit')
return result, {'input_interval': [start, stop], 'explicit_field_changes': changes,
'all_other_fields_unchanged': True, 'random_draws': 0, 'training': False}
def adapt_overlay_manifest(original, am_helper_path):
"""Translate observed AM manifest fields; preserve original overlay bytes.
This function validates declarations only. Callers independently rehash
every referenced file against external AM/source authority before use.
"""
require(type(original) is dict and original.get('schema') == 'phase4am-overlay-v1' and
original.get('overlay_name') in ('zero', 'static25') and
original.get('training') is False and original.get('fixed_for_whole_trial') is True,
'Only the fixed AM zero/static25 overlays are in this engineering gate')
require(original.get('mask_sha256') == MASK_SHA and _absolute(am_helper_path), 'Original mask/helper identity required')
names = ('overlay_name', 'overlay_sha256', 'mask_path', 'mask_sha256',
'base_candidate_path', 'base_candidate_sha256', 'aj_model_dir')
require(all(k in original for k in names), 'Missing observed AM manifest field')
require(all(_sha(original[k]) for k in ('overlay_sha256', 'mask_sha256', 'base_candidate_sha256')),
'Invalid external overlay/base SHA')
require(all(_absolute(original[k]) for k in ('mask_path', 'base_candidate_path', 'aj_model_dir')),
'Invalid external overlay/base path')
return dict(schema='phase4an-overlay-v1', **{k: copy.deepcopy(original[k]) for k in names},
am_overlay_path=am_helper_path, am_overlay_sha256=AM_HELPER_SHA)
@@ -0,0 +1,153 @@
"""Pure fixed 32-call engineering inventory; no I/O, GPU or input preparation.
Rows describe required executions and comparisons, not evidence they occurred.
The external AM reference is a symbolic role: a coordinator must bind its actual
case, overlay, sealed outputs and hashes before execution. A validated plan does
not prove input authority, comparison success, conditioning or learning.
All public results are independently constructed mutable copies.
"""
from __future__ import annotations
SCHEMA = 'phase4an-engineering-plan-draft-v1'
OVERLAYS = ('zero', 'static25')
MODES = ('reference', 'light')
CHAIN_LENGTHS = (5, 13, 1, 481, 500)
PREFIX_ENDS = (18, 19, 500)
def _fresh_id(overlay, mode, repeat=0):
return f'{overlay}--fresh--{mode}-r{repeat}'
def _prefix_id(overlay, end):
return f'{overlay}--prefix--reference-t{end:04d}'
def _chain_id(overlay, mode, index):
return f'{overlay}--chain--{mode}-s{index:02d}'
def _comparison(check, row, *, reference=None, external=None, scope, candidates=None):
"""Exact descriptors; start/stop are always absolute, half-open ticks."""
return dict(check=check, reference_row_id=reference, external_reference=external,
candidate_row_ids=[row['row_id']] if candidates is None else list(candidates),
start=row['start'], stop=row['stop'], scope=scope)
def _row(identity, overlay, mode, kind, start, stop, parent=None):
return dict(row_id=identity, overlay=overlay, mode=mode, kind=kind,
start=start, stop=stop, steps=stop - start, parent_row_id=parent,
expected_comparisons=[])
def _terminal_reference(overlay, stop):
if stop == 5:
return _chain_id(overlay, 'reference', 0)
if stop == 1000:
return _fresh_id(overlay, 'reference')
return _prefix_id(overlay, stop)
def build_plan():
"""Return the sole current proposal: 32 calls / 11,074 full-graph ticks.
kind='chain' with parent_row_id=None is its fresh first segment, NOT a
continuation from an assumed zero checkpoint. Later segments explicitly
reference their predecessor. A chain's local request/event coordinates
are translated using start; terminal.next_tick remains its local phase.
All fresh/prefix rows globally precede chains. Reference chains precede
light chains, allowing the reference first segment to supply prefix 5.
No extra prefix-5 execution, synthetic 52-call matrix, long idle run,
input selection, training, parameter update or executor is implemented.
Comparison scopes:
reference_full/reference_prefix/reference_slice: include reference
tick-state hashes, APL traces, queues, requests, events and states;
light_recorded: requests.csv, events/counts, final state, APL state,
normalized pending queues and finite/weight/arrival accounting;
terminal_state_and_normalized_queues: no comparison of segment counts
against cumulative prefix counts; counts are handled by chain_exact;
parent_state_and_pending_queues: bound parent restoration/carry checks.
A comparator must implement these scopes from artifacts, not this plan's
existence. Shift local queue headers before comparing absolute timelines.
"""
rows = []
for overlay in OVERLAYS:
for mode, repeat in (('reference', 0), ('light', 0), ('light', 1)):
row = _row(_fresh_id(overlay, mode, repeat), overlay, mode, 'fresh', 0, 1000)
scope = 'reference_full' if mode == 'reference' else 'light_recorded'
row['expected_comparisons'].append(_comparison(
'am_exact', row, external='sealed_AM_selected_case_same_overlay_r0', scope=scope))
if mode == 'light':
reference = _fresh_id(overlay, 'reference' if repeat == 0 else 'light')
row['expected_comparisons'].append(_comparison(
'fresh_exact', row, reference=reference, scope='light_recorded'))
rows.append(row)
for end in PREFIX_ENDS:
row = _row(_prefix_id(overlay, end), overlay, 'reference', 'prefix', 0, end)
row['expected_comparisons'].append(_comparison(
'prefix_of_reference', row, reference=_fresh_id(overlay, 'reference'), scope='reference_prefix'))
rows.append(row)
for overlay in OVERLAYS:
for mode in MODES:
start, parent, members = 0, None, []
for index, length in enumerate(CHAIN_LENGTHS):
row = _row(_chain_id(overlay, mode, index), overlay, mode, 'chain', start, start + length, parent)
members.append(row['row_id'])
if mode == 'reference':
row['expected_comparisons'].append(_comparison(
'segment_of_reference', row, reference=_fresh_id(overlay, 'reference'), scope='reference_slice'))
else:
row['expected_comparisons'].append(_comparison(
'segment_exact', row, reference=_chain_id(overlay, 'reference', index), scope='light_recorded'))
terminal = _terminal_reference(overlay, row['stop'])
if terminal != row['row_id']:
row['expected_comparisons'].append(_comparison(
'checkpoint_exact', row, reference=terminal, scope='terminal_state_and_normalized_queues'))
if parent is not None:
row['expected_comparisons'].append(_comparison(
'parent_carry', row, reference=parent, scope='parent_state_and_pending_queues'))
if index == len(CHAIN_LENGTHS) - 1:
comparison = _comparison('chain_exact', row, reference=_fresh_id(overlay, mode),
scope='reference_full' if mode == 'reference' else 'light_recorded', candidates=members)
comparison['start'] = 0
row['expected_comparisons'].append(comparison)
rows.append(row)
start, parent = row['stop'], row['row_id']
return dict(schema=SCHEMA, overlays=list(OVERLAYS), chain_lengths=list(CHAIN_LENGTHS),
rows=rows, total_calls=32, total_ticks=11074,
training=False, candidate_generated=False)
def _exact(actual, expected, path):
"""JSON-shaped values with exact Python types; True is not integer 1."""
if type(actual) is not type(expected):
raise ValueError(f'Plan type mismatch at {path}')
if type(expected) is dict:
if any(type(key) is not str for key in actual) or set(actual) != set(expected):
raise ValueError(f'Plan fields mismatch at {path}')
for key in expected:
_exact(actual[key], expected[key], path + '.' + key)
elif type(expected) is list:
if len(actual) != len(expected):
raise ValueError(f'Plan inventory length mismatch at {path}')
for index, (left, right) in enumerate(zip(actual, expected)):
_exact(left, right, f'{path}[{index}]')
elif actual != expected:
raise ValueError(f'Plan value mismatch at {path}')
def validate_plan(plan):
"""Reject any noncanonical plan; return a new copy, never a PASS receipt.
Exact canonical reconstruction rejects unknown fields, order changes,
omitted/extra overlays or calls, interval gaps, false parent links and
weakening/removal of required comparisons. No input files are read.
"""
canonical = build_plan()
_exact(plan, canonical, 'plan')
return canonical
@@ -0,0 +1,532 @@
"""Read-only AM authority verification and exclusive AN engineering preparation.
Public prepare(model_manifest_sha, experiment_manifest_sha, final_sha) requires
three externally recorded, actual AM evidence hashes. It verifies both complete
AM seals once, checks their declared gates, then fixes the first planned r0 case.
It creates only templates/requests/overlay descriptors and a preregistration;
never a GPU trajectory or fictitious future checkpoint hash. The caller records
the returned preregistration SHA outside this experiment before execution.
read_registration(expected_sha) rehashes the exact used source/input inventory
and control anchors without rerunning either scientific gate or full AM seal.
Future spec.json/checkpoint receipts are runtime artifacts, not preregistered
bytes. No frozen model module globals are modified; no Torch is imported.
"""
from __future__ import annotations
import hashlib
import importlib.util
import json
from pathlib import Path, PurePosixPath
import re
import shutil
import sys
import numpy as np
ROOT = Path('/data/digital-drosophila')
MODEL = Path(__file__).resolve().parent
EXP = ROOT / 'experiments/phase4an-engineering-validation'
AM_MODEL_REL = Path('models/male-cns-lif/wholebrain-plastic-interface-v1')
AM_EXP_REL = Path('experiments/phase4am-wholebrain-plastic-interface')
AK_REL = Path('experiments/phase4ak-apl-timing-validation')
AI_HELPER_REL = Path('models/male-cns-lif/pn-recruitment-audit-v1/publication.py')
AI_HELPER_SHA = '7a79c499098b74163e7bb16adabcbc54e7069dafa204d0b99c7f4c03d79f8016'
AM_PREREG_SHA = '465886b78ada08092c2c83de278844707499718977fb6dc6a7608391da0e5304'
AM_OVERLAY_SHA = '7f891d0ba79b393c96e75fd2dee2eddc51e57d8213eaac2779703bc2c18712b4'
FINAL_STATUS = 'PASS-EXACT-WHOLEBRAIN-PLASTIC-INTERFACE-NOT-LEARNING'
REG_SCHEMA = 'phase4an-engineering-registration-v1'
BUDGET = {'output_budget_bytes': 8 * 1024**3, 'minimum_free_reserve_bytes': 10 * 1024**3}
MODULES = ('runtime_light', 'runtime_continuation', 'engineering_inputs', 'engineering_plan',
'engineering_compare', 'engineering_queue_audit', 'engineering_prepare', 'engineering_workflow',
'engineering_scopes', 'engineering_analysis')
SOURCE_FILES = tuple(sorted([name + '.py' for name in MODULES] + ['test_' + name + '.py' for name in MODULES] + ['ENGINEERING_GATE_DRAFT.md']))
REFERENCES = ('auditory-10hz', 'auditory-100hz', 'olfactory-50hz', 'mechanosensory-50hz', 'gustatory-50hz')
GRAPH_FILES = ('nodes.npy', 'edge_id.npy', 'src.npy', 'dst.npy', 'signed_raw.npy', 'ptr.npy', 'weights.npy')
APL_FILES = ('input_matrix.npy', 'output_matrix.npy', 'kc_globals.npy', 'apl_globals.npy')
_MODULE_CACHE = {}
def require(ok, message):
if not ok:
raise ValueError(message)
def digest(path):
value = hashlib.sha256()
with Path(path).open('rb') as stream:
for block in iter(lambda: stream.read(8 << 20), b''):
value.update(block)
return value.hexdigest()
def sha(value):
return type(value) is str and re.fullmatch('[0-9a-f]{64}', value) is not None
def read(path):
def unique(pairs):
result = {}
for key, value in pairs:
require(key not in result, 'Duplicate JSON key: ' + key)
result[key] = value
return result
return json.loads(Path(path).read_text(encoding='utf-8'), object_pairs_hook=unique,
parse_constant=lambda value: (_ for _ in ()).throw(ValueError('Nonfinite JSON: ' + value)))
def save(path, value):
path = Path(path)
path.parent.mkdir(parents=True, exist_ok=True)
with path.open('x', encoding='utf-8') as stream:
json.dump(value, stream, sort_keys=True, indent=2, ensure_ascii=False, allow_nan=False)
stream.write('\n')
def load_module(name, path, expected):
path = Path(path).resolve(strict=True)
raw = path.read_bytes()
require(hashlib.sha256(raw).hexdigest() == expected, 'Source pin mismatch: ' + str(path))
identity = str(path), expected
if name in sys.modules:
require(_MODULE_CACHE.get(name) == (identity, sys.modules[name]), 'Unverified module namespace collision')
return sys.modules[name]
spec = importlib.util.spec_from_file_location(name, path)
module = importlib.util.module_from_spec(spec)
sys.modules[name] = module
try:
exec(compile(raw, str(path), 'exec'), module.__dict__)
except BaseException:
sys.modules.pop(name, None)
raise
_MODULE_CACHE[name] = identity, module
return module
def source_inventory(model):
model = Path(model).resolve(strict=True)
result = {}
for name in SOURCE_FILES:
path = model / name
require(path.is_file() and not path.is_symlink() and path.resolve().parent == model, 'Required engineering source missing/linked: ' + name)
result[str(path)] = digest(path)
return result
def _relative(value):
require(type(value) is str and value and '\\' not in value, 'Invalid relative inventory path')
path = PurePosixPath(value)
require(not path.is_absolute() and str(path) == value and all(part not in ('', '.', '..') and ':' not in part for part in path.parts),
'Noncanonical relative inventory path')
return path
def _safe_file(root, relative):
relative = _relative(relative)
root = Path(root).resolve(strict=True)
path = root
for part in relative.parts:
path = path / part
require(not path.is_symlink(), 'Linked authority/input file')
require(path.is_file() and path.resolve().is_relative_to(root), 'Missing or escaping input file')
return path
def _index(rows):
require(type(rows) is list and rows, 'Missing sealed inventory rows')
result = {}
for row in rows:
require(type(row) is dict and set(row) == {'path', 'bytes', 'sha256'} and
type(row['bytes']) is int and row['bytes'] >= 0 and sha(row['sha256']), 'Invalid sealed inventory row')
_relative(row['path'])
require(row['path'] not in result, 'Duplicate sealed inventory row')
result[row['path']] = row
return result
def _bind(inputs, path, expected=None):
path = Path(path).resolve(strict=True)
actual = digest(path)
require(expected is None or actual == expected, 'Authority/input SHA mismatch: ' + str(path))
require(str(path) not in inputs or inputs[str(path)] == actual, 'Conflicting authority for input')
inputs[str(path)] = actual
return path
def _sealed_json(experiment, index, relative):
require(relative in index, 'Required authority absent from AM seal: ' + relative)
path = _safe_file(experiment, relative)
require(digest(path) == index[relative]['sha256'], 'Sealed JSON changed: ' + relative)
return read(path)
def validate_final_status(final, scientific, before, after, refs):
"""Check actual AM final/summary/report schemas, never infer scientific PASS."""
require(final.get('status') == scientific.get('status') == FINAL_STATUS, 'AM final status is not the required completed interface gate')
for key in ('zero_overlay_exact', 'static25_repeat_exact', 'old_integrity_pass', 'frozen_five_reference_pass',
'frozen_eligibility_and_mask_unchanged', 'base_weights_unchanged'):
require(final.get(key) is True, 'AM final gate failed/missing: ' + key)
require(final.get('primary_trials_per_stage') == 120 and final.get('repeats_per_stage') == 24 and
scientific.get('primary_trials_per_stage') == 120 and scientific.get('technical_repeats_per_stage') == 24,
'AM complete coverage counts changed')
for value in (final, scientific):
require(value.get('training') is False and value.get('learned_candidate') is False, 'AM interface must not claim training/learning')
require(scientific.get('zero_overlay_exact') is True and scientific.get('static25_repeat_exact') is True, 'Scientific summary gate mismatch')
require(type(before) is dict and type(after) is dict and len(before) == len(after) == 28 and set(before) == set(after),
'AM must retain the same complete 28-root old-integrity inventory')
for name in before:
for report in (before[name], after[name]):
require(type(report) is dict and report.get('pass') is True, 'Old-integrity gate failed: ' + name)
if 'mismatches' in report:
require(report['mismatches'] == [], 'Old-integrity mismatch list is nonempty')
for key in ('manifest_sha256', 'checksums_sha256'):
if key in before[name] or key in after[name]:
require(sha(before[name].get(key)) and before[name][key] == after[name].get(key), 'Old-root control changed: ' + name)
require(refs.get('pass_all') is True and type(refs.get('trials')) is list and
[row.get('trial') for row in refs['trials']] == list(REFERENCES), 'Five old deterministic references are incomplete or reordered')
for row in refs['trials']:
require(type(row.get('exact')) is dict and set(row['exact']) == {'step', 'neuron_idx', 'counts', 'v', 'g'} and
all(value is True for value in row['exact'].values()), 'Old reference arrays are not exact')
def validate_planned(planned):
require(type(planned) is list and len(planned) == 144, 'AM planned coverage must be 144')
keys = []
for row in planned:
require(type(row) is dict and type(row.get('name')) is str and re.fullmatch('[A-Za-z0-9_-]+', row['name']) and
row.get('cohort') in ('seed_validation', 'novel_to_aj') and type(row.get('repeat')) is int and row['repeat'] in (0, 1),
'Invalid AM planned row identity')
keys.append((row['name'], row['repeat']))
require(len(set(keys)) == 144 and sum(row['repeat'] == 0 for row in planned) == 120 and sum(row['repeat'] == 1 for row in planned) == 24,
'AM primary/repeat set incomplete')
require(all((name, 0) in set(keys) for name, repeat in keys if repeat == 1), 'AM repeat lacks primary')
return next(dict(row) for row in planned if row['repeat'] == 0)
def validate_stage(experiment, index, stage, planned, root):
"""Validate all 144 declared results against the already verified AM seal."""
receipt = _sealed_json(experiment, index, 'audit/' + stage + '-complete.json')
require(receipt.get('stage') == stage and receipt.get('pass_all') is True and receipt.get('trials') == 144 and
receipt.get('primary_trials') == 120 and receipt.get('technical_repeats') == 24 and receipt.get('training') is False,
'AM stage is incomplete: ' + stage)
completed = receipt.get('completed')
require(type(completed) is list and len(completed) == 144, 'AM stage completion inventory truncated')
for row, done in zip(planned, completed):
require(done.get('name') == row['name'] and done.get('repeat') == row['repeat'], 'AM stage completion order/identity mismatch')
prefix = 'gpu/' + stage + '/' + row['name'] + '-r' + str(row['repeat']) + '/'
expected = [dict(path=key[len(prefix):], bytes=value['bytes'], sha256=value['sha256']) for key, value in sorted(index.items())
if key.startswith(prefix) and key != prefix + 'output-hashes.json']
inventory = _sealed_json(experiment, index, prefix + 'output-hashes.json')
require(inventory == expected and inventory, 'AM completed output inventory differs from sealed payload')
require(done.get('output_hashes_sha256') == index[prefix + 'output-hashes.json']['sha256'] and
done.get('comparison_sha256') == index[prefix + 'comparison.json']['sha256'], 'AM stage receipt SHA mismatch')
summary = _sealed_json(experiment, index, prefix + 'summary.json')
compare = _sealed_json(experiment, index, prefix + 'comparison.json')
folder, case = experiment / prefix.rstrip('/'), root / AK_REL / 'cases' / row['cohort'] / row['name']
require(summary.get('case') == row['name'] and summary.get('complete') is True and summary.get('pass_all') is True and
summary.get('all_finite') is True and summary.get('steps') == summary.get('completed_ticks') == 1000,
'AM selected stage contains a failed/truncated trial')
require(compare.get('pass_all') is True and compare.get('overlay_name') == stage and compare.get('case_dir') == str(case) and
compare.get('right') == str(folder), 'AM comparison identity/gate mismatch')
require(type(compare.get('checks')) is dict and compare['checks'] and all(value is True for value in compare['checks'].values()) and
compare.get('failed_checks') == [] and 'error' not in compare and 'read_manifest_error' not in compare and
compare.get('full_queue_reconstructed') is True, 'AM comparison has a hidden negative/missing numerical gate')
mode = 'zero' if stage == 'zero' else ('repeat' if row['repeat'] else 'self')
left = root / AK_REL / 'gpu' / row['cohort'] / (row['name'] + '-r' + str(row['repeat'])) if stage == 'zero' else (
experiment / 'gpu' / stage / (row['name'] + '-r0') if row['repeat'] else folder)
require(compare.get('mode') == mode and compare.get('left') == str(left), 'AM comparison reference/mode mismatch')
return receipt
def _layout(root, model, experiment):
root, model = Path(root).resolve(strict=True), Path(model).resolve(strict=True)
experiment = Path(experiment).resolve(strict=False)
require(experiment == root / 'experiments/phase4an-engineering-validation' and model.is_relative_to(root / 'models/male-cns-lif'),
'Unexpected project/output layout')
require(not experiment.exists() or (experiment.is_dir() and not any(experiment.iterdir())), 'Refusing populated experiment; preserve all partials')
return root, model, experiment
def _budget(experiment):
experiment = Path(experiment)
used = sum(path.stat().st_size for path in experiment.rglob('*') if path.is_file()) if experiment.exists() else 0
require(used < BUDGET['output_budget_bytes'], '8 GiB engineering budget exhausted; preserve partials')
free = shutil.disk_usage(experiment.parent).free
require(free >= BUDGET['minimum_free_reserve_bytes'] + BUDGET['output_budget_bytes'] - used,
'Insufficient capacity for remaining 8 GiB budget plus 10 GiB free reserve')
return dict(BUDGET, used_bytes=used, free_bytes=free)
def budget():
return _budget(EXP)
def verify_authority(anchors):
"""Full read-only AM seal recheck for the finalizer; no gate rerun."""
require(sys.dont_write_bytecode, 'Use python -B for immutable authority verification')
root = ROOT.resolve(strict=True)
values = [(root / AM_MODEL_REL, 'am_model_manifest_sha256'), (root / AM_EXP_REL, 'am_experiment_manifest_sha256')]
for directory, key in values:
require(sha(anchors.get(key)) and digest(directory / 'manifest.json') == anchors[key], 'External AM manifest anchor mismatch')
require(sha(anchors.get('am_final_sha256')) and digest(root / AM_EXP_REL / 'audit/final-status.json') == anchors['am_final_sha256'],
'External AM final anchor mismatch')
require(anchors.get('am_preregistration_sha256') == AM_PREREG_SHA and
digest(root / AM_EXP_REL / 'protocol/preregistered.json') == AM_PREREG_SHA, 'AM preregistration anchor mismatch')
require(anchors.get('AI_publication_helper_sha256') == AI_HELPER_SHA, 'Publication helper authority changed')
publication = load_module('phase4an_pinned_AI_publication', root / AI_HELPER_REL, AI_HELPER_SHA)
result = {'model': publication.verify_seal(values[0][0]), 'experiment': publication.verify_seal(values[1][0])}
require(result['model']['manifest_sha256'] == anchors['am_model_manifest_sha256'] and
result['experiment']['manifest_sha256'] == anchors['am_experiment_manifest_sha256'], 'AM seal changed during verification')
return result
def _old_integrity(root, context_sha, expected):
"""Reuse the pinned AM read-only integrity chain, not a brain runtime."""
context_path = root / AM_MODEL_REL / 'context.py'
previous_path = list(sys.path)
torch_before = 'torch' in sys.modules
try:
context = load_module('phase4an_sealed_AM_integrity', context_path, context_sha)
require(context.ROOT.resolve() == root.resolve(), 'AM integrity helper belongs to another root')
result = context.integrity()
finally:
sys.path[:] = previous_path
require(torch_before or 'torch' not in sys.modules, 'Integrity unexpectedly imported Torch')
require(type(result) is dict and len(result) == 28 and set(result) == set(expected) and
all(type(row) is dict and row.get('pass') is True for row in result.values()), 'Current 28-root old-integrity gate failed')
for name, row in result.items():
if 'mismatches' in row:
require(row['mismatches'] == [], 'Current old-integrity mismatch list is nonempty')
for key in ('manifest_sha256', 'checksums_sha256', 'reference_sha256'):
if key in expected[name] or key in row:
require(sha(row.get(key)) and row[key] == expected[name].get(key), 'Old integrity authority changed: ' + name)
return result
def verify_old_integrity(registration):
"""Rehash the pinned 28 old roots/snapshot via original CPU-only code."""
require(sys.dont_write_bytecode, 'Use python -B for old-integrity verification')
path = ROOT / AM_MODEL_REL / 'context.py'
expected_sha = registration['input_sha256'].get(str(path.resolve(strict=True)))
require(sha(expected_sha) and digest(path) == expected_sha, 'AM integrity source is not a registered input')
before_path = EXP / 'regression/old-integrity-before.json'
require(digest(before_path) == registration['input_sha256'].get(str(before_path.resolve(strict=True))), 'Before-integrity evidence changed')
return _old_integrity(ROOT, expected_sha, read(before_path))
def _prepare(root, model, experiment, am_model_manifest_sha, am_experiment_manifest_sha, am_final_sha):
require(sys.dont_write_bytecode, 'Preparation requires python -B; old sources must remain byte-for-byte read-only')
require(all(sha(value) for value in (am_model_manifest_sha, am_experiment_manifest_sha, am_final_sha)), 'Three actual external AM SHA-256 anchors are mandatory')
root, model, experiment = _layout(root, model, experiment)
_budget(experiment)
sources = source_inventory(model)
inputs_module = load_module('phase4an_prepare_inputs', model / 'engineering_inputs.py', sources[str(model / 'engineering_inputs.py')])
plan_module = load_module('phase4an_prepare_plan', model / 'engineering_plan.py', sources[str(model / 'engineering_plan.py')])
runtime = load_module('phase4an_prepare_light', model / 'runtime_light.py', sources[str(model / 'runtime_light.py')])
publication = load_module('phase4an_pinned_AI_publication', root / AI_HELPER_REL, AI_HELPER_SHA)
am_model, am = root / AM_MODEL_REL, root / AM_EXP_REL
require(digest(am_model / 'manifest.json') == am_model_manifest_sha and digest(am / 'manifest.json') == am_experiment_manifest_sha and
digest(am / 'audit/final-status.json') == am_final_sha, 'External AM authority mismatch')
verified = {'model': publication.verify_seal(am_model), 'experiment': publication.verify_seal(am)}
require(verified['model']['manifest_sha256'] == am_model_manifest_sha and verified['experiment']['manifest_sha256'] == am_experiment_manifest_sha,
'AM controls changed during full seal verification')
idx = _index(verified['experiment']['files']); midx = _index(verified['model']['files'])
reg = _sealed_json(am, idx, 'protocol/preregistered.json')
require(idx['protocol/preregistered.json']['sha256'] == AM_PREREG_SHA, 'AM preregistration authority differs from the fixed anchor')
planned = reg.get('planned'); selected = validate_planned(planned)
final = _sealed_json(am, idx, 'audit/final-status.json')
scientific = _sealed_json(am, idx, 'audit/scientific-summary.json')
require(final.get('scientific_summary_sha256') == idx['audit/scientific-summary.json']['sha256'], 'AM final does not bind scientific summary')
before = _sealed_json(am, idx, 'regression/old-integrity-before.json')
after = _sealed_json(am, idx, 'regression/old-integrity-after.json')
refs = _sealed_json(am, idx, 'regression/frozen-five.json')
validate_final_status(final, scientific, before, after, refs)
for stage in ('zero', 'static25'):
validate_stage(am, idx, stage, planned, root)
current_old = _old_integrity(root, midx['context.py']['sha256'], after)
inputs = {}
anchors = {'am_model_manifest_sha256': am_model_manifest_sha, 'am_experiment_manifest_sha256': am_experiment_manifest_sha,
'am_final_sha256': am_final_sha, 'am_preregistration_sha256': AM_PREREG_SHA, 'AI_publication_helper_sha256': AI_HELPER_SHA}
for path, expected in ((am_model / 'manifest.json', am_model_manifest_sha), (am / 'manifest.json', am_experiment_manifest_sha),
(am / 'audit/final-status.json', am_final_sha), (root / AI_HELPER_REL, AI_HELPER_SHA),
(am_model / 'overlay.py', AM_OVERLAY_SHA)):
_bind(inputs, path, expected)
require(midx.get('overlay.py', {}).get('sha256') == AM_OVERLAY_SHA, 'AM model seal does not contain the pinned overlay helper')
for name in ('context.py', 'config.json'):
_bind(inputs, am_model / name, midx[name]['sha256'])
for path in (am_model / 'checksums.sha256', am / 'checksums.sha256'):
_bind(inputs, path)
for relative in ('protocol/preregistered.json', 'protocol/prepared-hashes.json', 'audit/scientific-summary.json', 'audit/zero-complete.json', 'audit/static25-complete.json',
'regression/old-integrity-before.json', 'regression/old-integrity-after.json', 'regression/frozen-five.json'):
_bind(inputs, am / relative, idx[relative]['sha256'])
am_inputs = {}
for row in reg['inputs']:
require(set(row) == {'path', 'sha256'} and sha(row['sha256']), 'Invalid AM registered input row')
_relative(row['path']); require(row['path'] not in am_inputs, 'Duplicate AM registered input')
am_inputs[row['path']] = row['sha256']
def authority(path):
path = Path(path).resolve(strict=True)
require(path.is_relative_to(root), 'AM input leaves project root')
relative = path.relative_to(root).as_posix()
if path.is_relative_to(am):
expected = idx[path.relative_to(am).as_posix()]['sha256']
else:
require(relative in am_inputs, 'Dependency absent from AM preregistration: ' + relative)
expected = am_inputs[relative]
return _bind(inputs, path, expected)
original_dir = root / AK_REL / 'cases' / selected['cohort'] / selected['name']
authority(original_dir / 'spec.json'); authority(original_dir / 'requests.npz')
original = read(original_dir / 'spec.json')
require(original['steps'] == 1000 and original['kind'] == 'fullgraph' and original['name'] == selected['name'], 'Fixed selected AM case changed')
with np.load(original_dir / 'requests.npz', allow_pickle=False) as archive:
require(set(archive.files) == {'tick', 'neuron_idx'} and len(archive.files) == 2, 'Unexpected original request arrays')
requests = {key: archive[key].copy() for key in archive.files}
inputs_module.validate_requests(requests, 1000)
for name in GRAPH_FILES:
authority(Path(original['graph_dir']) / name)
for name in APL_FILES:
authority(Path(original['apl_bundle_dir']) / name)
authority(original['coefficients_path'])
selected_outputs = {}
for stage in ('zero', 'static25'):
prefix = 'gpu/' + stage + '/' + selected['name'] + '-r0/'
selected_outputs[stage] = str(am / prefix.rstrip('/'))
for relative, row in idx.items():
if relative.startswith(prefix):
_bind(inputs, am / relative, row['sha256'])
adapted = {}
for stage in ('zero', 'static25'):
old_manifest = authority(am / 'overlays' / (stage + '.json'))
overlay_file = authority(am / 'overlays' / (stage + '.npz'))
declaration = read(old_manifest)
require(declaration['overlay_name'] == stage and declaration['overlay_sha256'] == digest(overlay_file), 'AM overlay declaration mismatch')
for label in ('mask', 'base_candidate'):
require(digest(authority(declaration[label + '_path'])) == declaration[label + '_sha256'], 'Overlay external array authority mismatch')
adapted[stage] = inputs_module.adapt_overlay_manifest(declaration, str(am_model / 'overlay.py'))
require(adapted[stage]['am_overlay_sha256'] == AM_OVERLAY_SHA, 'Adapted helper pin differs')
for relative, expected in runtime.PINS.items():
_bind(inputs, model.parent / relative, expected)
plan = plan_module.validate_plan(plan_module.build_plan())
require(plan['total_calls'] == 32 and plan['total_ticks'] == 11074, 'Engineering plan matrix changed')
# No writes have occurred before all authority/source/space checks above.
experiment.mkdir(parents=True, exist_ok=True)
try:
for folder in ('protocol', 'cases', 'overlays', 'audit', 'runs', 'receipts', 'logs', 'regression', 'extension-build'):
(experiment / folder).mkdir(exist_ok=False)
save(experiment / 'regression/old-integrity-before.json', current_old)
_bind(inputs, experiment / 'regression/old-integrity-before.json')
for source, expected in sources.items():
target = experiment / 'protocol/source-snapshot' / Path(source).name
publication._copy_verified(Path(source), target, Path(source).stat().st_size, expected)
_bind(inputs, target, expected)
save(experiment / 'protocol/AM-seal-verification.json', verified)
save(experiment / 'protocol/AM-gate-authority.json', dict(anchors=anchors, all_144_per_stage_declared_gates_pass=True,
old_roots=28, old_references=list(REFERENCES), verification_scope='Full AM seals and finalized declared gate evidence; no new scientific execution'))
for name in ('AM-seal-verification.json', 'AM-gate-authority.json'):
_bind(inputs, experiment / 'protocol' / name)
overlays, cases = {}, {}
for stage, value in adapted.items():
manifest_path = experiment / 'overlays' / (stage + '.json')
save(manifest_path, value)
_bind(inputs, manifest_path)
overlays[stage] = {'overlay_file': str(am / 'overlays' / (stage + '.npz')), 'overlay_manifest': str(manifest_path)}
for row in plan['rows']:
case = experiment / 'cases' / row['row_id']; case.mkdir(exist_ok=False)
template = {'schema': 'phase4an-engineering-case-template-v1', 'row_id': row['row_id'], 'original_spec': original,
'start': row['start'], 'stop': row['stop'], 'budget_root': str(experiment), 'output_budget_bytes': BUDGET['output_budget_bytes']}
save(case / 'template-spec.json', template)
part = inputs_module.slice_requests(requests, 1000, row['start'], row['stop'])
with (case / 'requests.npz').open('xb') as stream:
np.savez_compressed(stream, **part)
_bind(inputs, case / 'template-spec.json'); _bind(inputs, case / 'requests.npz')
cases[row['row_id']] = {'case_dir': str(case), 'template_spec_sha256': digest(case / 'template-spec.json'),
'requests_sha256': digest(case / 'requests.npz')}
require(source_inventory(model) == sources and all(digest(path) == expected for path, expected in inputs.items()), 'Source/input changed during preparation')
# Repeat the full AM verification after producing only small new inputs.
after_verified = {'model': publication.verify_seal(am_model), 'experiment': publication.verify_seal(am)}
require(after_verified == verified, 'AM sealed payload changed during preparation')
_budget(experiment)
registration = {'schema': REG_SCHEMA, 'plan': plan, 'selected_case': {'row': selected, 'original_dir': str(original_dir), 'AM_outputs': selected_outputs},
'source_sha256': sources, 'input_sha256': inputs, 'cases': cases, 'overlays': overlays, 'budget': dict(BUDGET), 'anchors': anchors,
'training': False, 'new_brain_runs_during_preparation': 0, 'future_checkpoint_hashes_fabricated': False,
'runtime_spec_derivation': 'engineering_inputs.derive_spec(template.original_spec, interval, budget, actual verified parent SHA or None)',
'scope': 'Fixed 32-call engineering gate only; not conditioning or learning registration'}
save(experiment / 'protocol/preregistered.json', registration)
return {'prepared': True, 'registration_path': str(experiment / 'protocol/preregistered.json'),
'registration_sha256': digest(experiment / 'protocol/preregistered.json'), 'calls': 32, 'brain_runs': 0}
except Exception as error:
failure = experiment / 'audit/preparation-failure.json'
if not failure.exists():
save(failure, {'prepared': False, 'type': type(error).__name__, 'message': str(error), 'partial_outputs_preserved': True})
raise
def prepare(am_model_manifest_sha, am_experiment_manifest_sha, am_final_sha):
return _prepare(ROOT, MODEL, EXP, am_model_manifest_sha, am_experiment_manifest_sha, am_final_sha)
def row_path(registration, row_id):
require(type(row_id) is str and row_id in {row['row_id'] for row in registration['plan']['rows']}, 'Unknown engineering row ID')
expected = EXP.resolve(strict=True) / 'cases' / row_id
actual = Path(registration['cases'][row_id]['case_dir'])
require(actual == expected and actual.resolve(strict=True) == expected and actual.is_dir(), 'Row case path is not its fixed safe mapping')
return actual
def read_registration(expected_sha):
require(sys.dont_write_bytecode and sha(expected_sha), 'Require python -B and an external preregistration SHA')
path = EXP / 'protocol/preregistered.json'
require(digest(path) == expected_sha, 'Preregistration differs from external authority')
require(not (EXP / 'audit/preparation-failure.json').exists(), 'Preparation failed; preserve partials')
registration = read(path)
require(registration.get('schema') == REG_SCHEMA and registration.get('training') is False and registration.get('budget') == BUDGET,
'Unexpected registration schema/scope/budget')
require(registration['source_sha256'] == source_inventory(MODEL), 'Registered engineering sources changed or inventory incomplete')
for name, expected in registration['input_sha256'].items():
require(Path(name).is_absolute() and sha(expected) and digest(name) == expected, 'Registered input changed: ' + name)
plan_module = load_module('phase4an_read_plan', MODEL / 'engineering_plan.py', registration['source_sha256'][str(MODEL / 'engineering_plan.py')])
inputs_module = load_module('phase4an_read_inputs', MODEL / 'engineering_inputs.py', registration['source_sha256'][str(MODEL / 'engineering_inputs.py')])
plan_module.validate_plan(registration['plan'])
selected = validate_planned(read(ROOT / AM_EXP_REL / 'protocol/preregistered.json')['planned'])
original_dir = ROOT / AK_REL / 'cases' / selected['cohort'] / selected['name']
require(registration['selected_case']['row'] == selected and registration['selected_case']['original_dir'] == str(original_dir),
'Registered case is not the fixed first AM planned primary')
original = read(original_dir / 'spec.json')
with np.load(original_dir / 'requests.npz', allow_pickle=False) as archive:
require(set(archive.files) == {'tick', 'neuron_idx'} and len(archive.files) == 2, 'Unexpected original sparse request schema')
requests = {key: archive[key].copy() for key in archive.files}
inputs_module.validate_requests(requests, 1000)
require(set(registration['cases']) == {row['row_id'] for row in registration['plan']['rows']}, 'Registered case inventory differs from plan')
for row in registration['plan']['rows']:
case = row_path(registration, row['row_id']); entry = registration['cases'][row['row_id']]
require(entry['template_spec_sha256'] == digest(case / 'template-spec.json') and entry['requests_sha256'] == digest(case / 'requests.npz'),
'Template/request metadata does not bind registered bytes')
template = read(case / 'template-spec.json')
require(set(template) == {'schema', 'row_id', 'original_spec', 'start', 'stop', 'budget_root', 'output_budget_bytes'} and
template['schema'] == 'phase4an-engineering-case-template-v1' and template['original_spec'] == original and
template['row_id'] == row['row_id'] and template['start'] == row['start'] and template['stop'] == row['stop'] and
template['budget_root'] == str(EXP) and template['output_budget_bytes'] == BUDGET['output_budget_bytes'], 'Template rule differs from canonical row')
expected = inputs_module.slice_requests(requests, 1000, row['start'], row['stop'])
with np.load(case / 'requests.npz', allow_pickle=False) as archive:
require(len(archive.files) == 2 and set(archive.files) == set(expected) and
all(archive[key].dtype == expected[key].dtype and archive[key].shape == expected[key].shape and
archive[key].tobytes() == expected[key].tobytes() for key in expected), 'Registered request slice is not exact')
require(set(registration['overlays']) == {'zero', 'static25'}, 'Registered overlay inventory changed')
for stage in ('zero', 'static25'):
entry = registration['overlays'][stage]
expected_file = ROOT / AM_EXP_REL / 'overlays' / (stage + '.npz')
expected_manifest = EXP / 'overlays' / (stage + '.json')
require(entry == {'overlay_file': str(expected_file), 'overlay_manifest': str(expected_manifest)}, 'Overlay path mapping changed')
original_overlay = read(ROOT / AM_EXP_REL / 'overlays' / (stage + '.json'))
require(read(expected_manifest) == inputs_module.adapt_overlay_manifest(original_overlay, str(ROOT / AM_MODEL_REL / 'overlay.py')),
'Adapted overlay manifest is not the exact declared transformation')
anchors = registration['anchors']
for name, relative in (('am_model_manifest_sha256', AM_MODEL_REL / 'manifest.json'),
('am_experiment_manifest_sha256', AM_EXP_REL / 'manifest.json'),
('am_final_sha256', AM_EXP_REL / 'audit/final-status.json')):
require(sha(anchors[name]) and digest(ROOT / relative) == anchors[name], 'AM external anchor changed')
require(anchors['am_preregistration_sha256'] == AM_PREREG_SHA and digest(ROOT / AM_EXP_REL / 'protocol/preregistered.json') == AM_PREREG_SHA and
anchors['AI_publication_helper_sha256'] == AI_HELPER_SHA and digest(ROOT / AI_HELPER_REL) == AI_HELPER_SHA, 'Fixed AM/helper authority changed')
require(digest(path) == expected_sha, 'Registration changed during verification')
return registration
@@ -0,0 +1,243 @@
"""Independent, read-only CPU queue/ref audit for the draft AN 32-call gate.
The caller supplies externally authenticated records and CSR arrays. No file
I/O, runtime import, GPU, membrane integration, weight update or learning is
performed. Spikes are observed facts, not independently predicted here.
queue digest bytes independently implement frozen AF oracle_full.queue_summary
(source SHA below): header, little-endian signed int64 local tick/count, all
global edge IDs in source/CSR order, then uint8 post-threshold acceptance bits.
The function never target-sorts or drops zero-weight edges from the queue.
"""
from __future__ import annotations
import hashlib
import struct
import numpy as np
AF_QUEUE_SOURCE_SHA256 = 'd72003d98f0f1624b12126eff50bd361c43d888f4a64d90ba9247a58976f1961'
QUEUE_HEADER = b'phase4af-queue-v1\0'
DELAY, REFRACTORY = 18, 22
CHUNK = 1 << 18
QUEUE_KEYS = {'tick', 'count', 'accepted_count', 'ordered_edge_sha256', 'accepted_mask_sha256', 'joint_sha256'}
def require(value, message):
if not value:
raise ValueError(message)
def _array(value, dtype, shape, name):
require(isinstance(value, np.ndarray) and value.dtype == np.dtype(dtype) and value.shape == shape,
'Invalid dtype/shape: ' + name)
def _raw_sha(value):
result = hashlib.sha256()
for start in range(0, len(value), CHUNK):
result.update(np.ascontiguousarray(value[start:start + CHUNK], dtype=value.dtype.newbyteorder('<')).tobytes())
return result.hexdigest()
def _event_schema(tick, node, steps, n, label):
require(isinstance(tick, np.ndarray) and tick.ndim == 1, 'Invalid event vector: ' + label)
_array(tick, np.int32, tick.shape, label + '.tick')
_array(node, np.int32, tick.shape, label + '.neuron_idx')
require(np.all((tick >= 0) & (tick < steps)) and np.all((node >= 0) & (node < n)),
'Out-of-range event: ' + label)
key = tick.astype(np.int64) * n + node
require(len(key) < 2 or np.all(key[1:] > key[:-1]), 'Events must be sorted/unique: ' + label)
def _record_schema(record, n, label):
require(isinstance(record, dict), 'Record must be a dictionary: ' + label)
steps, offset = record['steps'], record['global_tick_offset']
require(type(steps) is int and 0 < steps <= np.iinfo(np.int32).max and
type(offset) is int and offset >= 0, 'Invalid record clock: ' + label)
state = record['state']
require(set(state) == {'step', 'neuron_idx', 'counts', 'v', 'g', 'ref'}, 'Invalid state keys: ' + label)
_event_schema(state['step'], state['neuron_idx'], steps, n, label)
for name, dtype in [('counts', np.int32), ('ref', np.int32), ('v', np.float64), ('g', np.float64)]:
_array(state[name], dtype, (n,), label + '.' + name)
require(np.isfinite(state[name]).all(), 'Nonfinite state: ' + label + '.' + name)
require(np.all(state['counts'] >= 0) and np.all((state['ref'] >= 0) & (state['ref'] <= REFRACTORY)),
'Invalid state count/ref: ' + label)
rows = record['requests']
require(isinstance(rows, np.ndarray) and rows.ndim == 2 and rows.shape[1] == 5,
'Invalid request table: ' + label)
_array(rows, np.int64, rows.shape, label + '.requests')
queues, hashes = record.get('queues'), record.get('state_hashes')
require(isinstance(queues, list) and isinstance(hashes, list) and len(queues) == len(hashes) == steps,
'Complete reference queue and state hashes are required: ' + label)
for t, (queue, state_hash) in enumerate(zip(queues, hashes)):
require(isinstance(queue, dict) and set(queue) == QUEUE_KEYS and type(queue['tick']) is int and queue['tick'] == t,
'Invalid local queue row: ' + label)
require(isinstance(state_hash, dict) and set(state_hash) == {'tick', 'v_sha256', 'g_sha256', 'ref_sha256'} and
type(state_hash['tick']) is int and state_hash['tick'] == t,
'Invalid local state-hash row: ' + label)
return record
def _graph(ptr, dst, edge_id, effective_weights):
require(isinstance(ptr, np.ndarray) and ptr.ndim == 1 and len(ptr) > 1, 'Invalid CSR ptr')
n, e = len(ptr) - 1, len(dst)
_array(ptr, np.int64, (n + 1,), 'ptr')
_array(dst, np.int32, (e,), 'dst')
_array(edge_id, np.int64, (e,), 'edge_id')
_array(effective_weights, np.float64, (e,), 'effective_weights_SI')
require(ptr[0] == 0 and ptr[-1] == e and np.all(ptr[1:] >= ptr[:-1]), 'Invalid CSR pointers')
for start in range(0, e, CHUNK):
stop = min(start + CHUNK, e)
require(np.all((dst[start:stop] >= 0) & (dst[start:stop] < n)), 'CSR destination out of range')
require(np.all(edge_id[start:stop] >= 0) and
np.all(np.diff(edge_id[max(0, start - 1):stop]) > 0), 'Global edge IDs must preserve canonical increasing CSR order')
require(np.isfinite(effective_weights[start:stop]).all(), 'Nonfinite effective SI weight')
return n, e
def _queue(tick, local_tick, sources, ref, ptr, dst, edge_id, weights, boundaries):
"""Two headers, shared source-ordered payload; memory bounded by one chunk."""
count = sum(int(ptr[int(s) + 1] - ptr[int(s)]) for s in sources)
joints = [hashlib.sha256(QUEUE_HEADER + struct.pack('<qq', t, count)) for t in (tick, local_tick)]
edges_hash, accepted_hash = hashlib.sha256(), hashlib.sha256()
for source in sources:
for start in range(int(ptr[source]), int(ptr[source + 1]), CHUNK):
stop = min(start + CHUNK, int(ptr[source + 1]))
data = np.ascontiguousarray(edge_id[start:stop], dtype='<i8').tobytes()
edges_hash.update(data)
for joint in joints:
joint.update(data)
accepted_count, nonzero_count, first = 0, 0, None
for source in sources:
for start in range(int(ptr[source]), int(ptr[source + 1]), CHUNK):
stop = min(start + CHUNK, int(ptr[source + 1]))
mask = np.ascontiguousarray(ref[dst[start:stop]] == 0, dtype=np.uint8)
accepted_count += int(mask.sum(dtype=np.int64))
data = mask.tobytes(); accepted_hash.update(data)
for joint in joints:
joint.update(data)
if boundaries:
live = np.flatnonzero((mask != 0) & (weights[start:stop] != 0))
nonzero_count += len(live)
if first is None and len(live):
index = start + int(live[0])
first = {'source_neuron_idx': int(source), 'target_neuron_idx': int(dst[index]),
'global_edge_id': int(edge_id[index]), 'csr_edge_index': index,
'source_spike_tick': tick - DELAY, 'arrival_tick': tick,
'effective_weight_SI_volt': float(weights[index]),
'effective_weight_float64_le_hex': np.asarray(weights[index], dtype='<f8').tobytes().hex(),
'accepted': True, 'post_threshold_target_ref': 0,
'weight_is_nonzero': True, 'evidence_is_unique_cause_or_necessity': False}
rows = [{'tick': t, 'count': count, 'accepted_count': accepted_count,
'ordered_edge_sha256': edges_hash.hexdigest(), 'accepted_mask_sha256': accepted_hash.hexdigest(),
'joint_sha256': joint.hexdigest()} for t, joint in zip((tick, local_tick), joints)]
return rows, first, nonzero_count
def audit_queues(monolithic, segments, *, ptr, dst, edge_id, effective_weights, requests, cuts):
"""Audit a fresh reference plus its complete reference segmentation.
All inputs are read-only. Return PASS/FAIL/INCOMPLETE JSON-safe evidence.
Structural input damage raises ValueError; valid but differing artifacts
return FAIL. ``cuts`` must be the exact internal segment boundaries.
No initial state is inferred for arbitrary nonzero-origin experiments.
"""
n, e = _graph(ptr, dst, edge_id, effective_weights)
mono = _record_schema(monolithic, n, 'monolithic')
require(mono['global_tick_offset'] == 0, 'This audit requires the registered fresh monolithic origin')
require(isinstance(segments, (list, tuple)) and len(segments) >= 2, 'At least two reference segments required')
parts = [_record_schema(p, n, 'segment-' + str(i)) for i, p in enumerate(segments)]
cursor, actual_cuts = 0, []
for i, part in enumerate(parts):
require(part['global_tick_offset'] == cursor, 'Segment clock gap/overlap')
if i:
actual_cuts.append(cursor)
cursor += part['steps']
require(cursor == mono['steps'] and type(cuts) in (list, tuple) and all(type(t) is int for t in cuts) and
list(cuts) == actual_cuts, 'Cuts/segments must cover the exact complete monolithic interval')
require(isinstance(requests, dict) and set(requests) == {'tick', 'neuron_idx'}, 'Exact original request arrays required')
_event_schema(requests['tick'], requests['neuron_idx'], mono['steps'], n, 'original_requests')
records = [('monolithic', mono), *[('segment-' + str(i), p) for i, p in enumerate(parts)]]
differences, difference_count = [], 0
def check(value, field, **location):
nonlocal difference_count
if not value:
difference_count += 1
if len(differences) < 100:
differences.append({'field': field, **location})
tick, node = mono['state']['step'], mono['state']['neuron_idx']
for label, record in records:
start, stop = record['global_tick_offset'], record['global_tick_offset'] + record['steps']
lo, hi = np.searchsorted(tick, (start, stop))
check(np.array_equal(record['state']['step'].astype(np.int64) + start, tick[lo:hi]), 'events.tick', record=label)
check(np.array_equal(record['state']['neuron_idx'], node[lo:hi]), 'events.neuron_idx', record=label)
check(np.array_equal(record['state']['counts'], np.bincount(record['state']['neuron_idx'], minlength=n)), 'counts', record=label)
expected_requests = np.empty((len(requests['tick']), 5), np.int64)
if len(expected_requests):
expected_requests[:, 0] = requests['tick']; expected_requests[:, 1] = requests['neuron_idx']; expected_requests[:, 2] = 1
ref = np.zeros(n, np.int32)
boundary_report = {t: {'boundary_tick': t, 'accepted_nonzero_crossing_edge_events': 0, 'first_witness': None} for t in cuts}
due_total, accepted_total, segment_index = 0, 0, 0
for t in range(mono['steps']):
while t >= parts[segment_index]['global_tick_offset'] + parts[segment_index]['steps']:
segment_index += 1
part = parts[segment_index]; local = t - part['global_tick_offset']
rlo, rhi = np.searchsorted(requests['tick'], (t, t + 1))
requested = requests['neuron_idx'][rlo:rhi]
accepted_requests = ref[requested] <= 1
expected_requests[rlo:rhi, 3] = accepted_requests
expected_requests[rlo:rhi, 4] = ~accepted_requests
np.maximum(ref - np.int32(1), np.int32(0), out=ref)
slo, shi = np.searchsorted(tick, (t, t + 1)); fired = node[slo:shi]
check(np.all(ref[fired] == 0), 'spike_during_refractory', global_tick=t)
check(np.isin(requested[accepted_requests], fired, assume_unique=True).all(), 'accepted_request_without_spike', global_tick=t)
ref[fired] = REFRACTORY
sources = node[0:0]
if t >= DELAY:
dlo, dhi = np.searchsorted(tick, (t - DELAY, t - DELAY + 1)); sources = node[dlo:dhi]
crossing = [cut for cut in cuts if t - DELAY < cut <= t]
rows, witness, live_count = _queue(t, local, sources, ref, ptr, dst, edge_id, effective_weights, crossing)
due_total += rows[0]['count']; accepted_total += rows[0]['accepted_count']
ref_sha = _raw_sha(ref)
for label, record, at, expected in [('monolithic', mono, t, rows[0]), ('segment-' + str(segment_index), part, local, rows[1])]:
for key in sorted(QUEUE_KEYS):
check(type(record['queues'][at][key]) is type(expected[key]) and record['queues'][at][key] == expected[key],
'queue.' + key, record=label, local_tick=at, global_tick=t)
check(record['state_hashes'][at]['ref_sha256'] == ref_sha, 'state_hash.ref_sha256', record=label, local_tick=at, global_tick=t)
if at == record['steps'] - 1:
check(np.array_equal(record['state']['ref'], ref), 'terminal.ref', record=label)
for cut in crossing:
boundary_report[cut]['accepted_nonzero_crossing_edge_events'] += live_count
if witness is not None and boundary_report[cut]['first_witness'] is None:
boundary_report[cut]['first_witness'] = {'boundary_tick': cut, **witness}
for label, record in records:
start, stop = record['global_tick_offset'], record['global_tick_offset'] + record['steps']
lo, hi = np.searchsorted(requests['tick'], (start, stop))
expected = expected_requests[lo:hi].copy(); expected[:, 0] -= start
check(np.array_equal(record['requests'], expected), 'requests.five_fields', record=label)
first_witness = min((row['first_witness'] for row in boundary_report.values() if row['first_witness'] is not None),
key=lambda row: (row['arrival_tick'], row['source_neuron_idx'], row['csr_edge_index'], row['boundary_tick']), default=None)
exact, live = difference_count == 0, first_witness is not None
return {'schema': 'phase4an-independent-queue-audit-v1', 'pass_all': exact and live,
'status': 'FAIL-QUEUE-OR-REF' if not exact else ('PASS-QUEUE-WITNESS-SCOPE' if live else 'INCOMPLETE-NO-ACCEPTED-NONZERO-CROSSING'),
'queue_ref_request_exact': exact, 'accepted_nonzero_crossing_witness_verified': exact and live,
'difference_count': difference_count, 'differences': differences, 'differences_truncated': difference_count > len(differences),
'neurons': n, 'edges': e, 'steps': mono['steps'], 'segments': len(parts), 'cuts': list(cuts),
'arriving_edges': due_total, 'accepted_arriving_edges': accepted_total, 'rejected_arriving_edges': due_total - accepted_total,
'requests': len(expected_requests), 'accepted_requests': int(expected_requests[:, 3].sum(dtype=np.int64)),
'rejected_requests': int(expected_requests[:, 4].sum(dtype=np.int64)),
'first_witness': first_witness, 'boundaries': list(boundary_report.values()),
'same_edge_event_may_cross_multiple_boundaries': True,
'input_array_sha256': {name: _raw_sha(value) for name, value in [('ptr', ptr), ('dst', dst), ('edge_id', edge_id),
('effective_weights_float64_SI', effective_weights), ('request_tick', requests['tick']), ('request_neuron_idx', requests['neuron_idx'])]},
'digest_definition': {'AF_source_sha256': AF_QUEUE_SOURCE_SHA256, 'header_hex': QUEUE_HEADER.hex(),
'payload_order': 'local tick/count <qq; all canonical source/CSR edge IDs <i8; all post-threshold accepted u1'},
'delay_steps': DELAY, 'refractory_steps': REFRACTORY,
'scope': 'Conditional on recorded spikes: exact ref, original requests, source-ordered real-edge arrival queues and finite-cut witness only',
'membrane_dynamics_independently_replayed': False, 'v_g_hashes_independently_verified': False,
'unique_causal_or_necessity_claim': False, 'external_seals_and_input_authority_verified_here': False,
'training': False, 'GPU_used': False}
@@ -0,0 +1,219 @@
"""Pure CPU comparisons of already-loaded engineering_compare run records.
No paths are accepted, no artifacts are opened, no GPU is imported. Structural
or missing-required-observation errors raise ValueError. Valid but different
records return numerical_exact=False. External seals, parent SHA authority and
independent CSR/local-header queue reconstruction remain coordinator duties.
"""
from __future__ import annotations
import hashlib
import numpy as np
import engineering_compare as ec
SCHEMA = 'phase4an-engineering-scopes-v1'
def _require(condition, message):
if not condition:
raise ValueError(message)
def _raw_sha(value):
return hashlib.sha256(np.ascontiguousarray(value, dtype=value.dtype.newbyteorder('<')).tobytes()).hexdigest()
def _record(value, label):
_require(type(value) is dict, label + ' must be an already-loaded run dictionary, not a path')
try:
ec.validate_run(value)
summary = value['summary']
_require(type(summary) is dict, label + ' requires summary metadata')
steps, offset = value['steps'], value['global_tick_offset']
_require(steps <= np.iinfo(np.int32).max, label + ' steps exceed int32 event capacity')
for key in ('complete', 'pass_all', 'all_finite'):
_require(summary.get(key) is True, label + ' is incomplete or failed: ' + key)
for key in ('steps', 'completed_ticks'):
_require(type(summary.get(key)) is int and summary[key] == steps, label + ' summary clock mismatch')
_require(type(summary.get('nodes')) is int and summary['nodes'] == len(value['state']['v']),
label + ' summary node count mismatch')
_require(summary.get('precision') == 'float64' and summary.get('apl_precision') == 'float32' and
type(summary.get('apl_delay_steps')) is int and summary['apl_delay_steps'] == 18,
label + ' numerical metadata mismatch')
continued = summary.get('checkpoint_continuation_used', False)
_require(type(continued) is bool and continued == (offset > 0), label + ' continuation scope mismatch')
declared_offset = summary.get('global_tick_offset', 0)
_require(type(declared_offset) is int and declared_offset == offset, label + ' summary offset mismatch')
if continued or 'global_next_tick' in summary:
_require(type(summary.get('global_next_tick')) is int and summary['global_next_tick'] == offset + steps,
label + ' summary global end mismatch')
_require(ec._sha(summary.get('effective_weights_raw_sha256')), label + ' effective weight SHA is missing')
mode = summary.get('diagnostic_mode')
legacy = str(value.get('terminal_origin', '')).startswith('AM_')
_require(mode in ('light', 'reference') or (mode is None and legacy), label + ' recording mode is not declared')
observations = {'apl_traces': bool(value.get('traces')), 'state_hashes': value.get('state_hashes') is not None,
'queues': value.get('queues') is not None}
for name, field in (('apl_traces', 'full_APL_history_recorded'),
('state_hashes', 'full_tick_state_hashes_recorded'), ('queues', 'full_queue_hashes_recorded')):
if field in summary:
_require(type(summary[field]) is bool, label + ' observation flag must be Boolean')
if mode == 'reference' or legacy or summary.get(field) is True:
_require(observations[name], label + ' declared reference observation missing: ' + name)
return value, observations
except (KeyError, TypeError, AttributeError, IndexError, OverflowError) as error:
raise ValueError(label + ' has an invalid run-record schema') from error
def _common(left, right):
return {'same_neuron_count': len(left['state']['v']) == len(right['state']['v']),
'same_APL_width': left['terminal']['apl_pending'].shape[1] == right['terminal']['apl_pending'].shape[1],
'same_effective_weights': left['summary']['effective_weights_raw_sha256'] ==
right['summary']['effective_weights_raw_sha256']}
def _report(comparison, checks, observed, limitations):
# All predicates are JSON booleans, not array scalars or success claims.
checks = {key: bool(value) for key, value in checks.items()}
return dict(schema=SCHEMA, comparison=comparison, numerical_exact=all(checks.values()),
checks=checks, failed_checks=[key for key, value in checks.items() if not value],
observed_scope=observed, limitations=list(limitations), learning_pass_claimed=False,
complete_dynamical_equivalence_proven=False, external_authority_verified_here=False,
independent_CSR_queue_reconstruction_performed=False)
def compare_slice(candidate, monolithic):
"""Compare candidate's absolute [start,stop) with an origin-zero full reference.
The final candidate V/g/ref bytes must match the reference's tick stop-1
hashes even for light candidates. Missing light intermediate observations
are disclosed, never replaced by fabricated zero arrays or inferred hashes.
"""
candidate, observed = _record(candidate, 'candidate')
monolithic, full = _record(monolithic, 'monolithic')
_require(monolithic['global_tick_offset'] == 0, 'Monolithic slice reference must have origin zero')
_require(monolithic['summary'].get('diagnostic_mode') == 'reference' or
str(monolithic.get('terminal_origin', '')).startswith('AM_'),
'Monolithic slice source must be declared reference or loaded legacy AM')
_require(all(full.values()), 'Monolithic slice reference needs all four APL traces, state hashes and queues')
start, steps = candidate['global_tick_offset'], candidate['steps']
stop = start + steps
_require(stop <= monolithic['steps'], 'Candidate slice extends beyond monolithic reference')
checks = _common(candidate, monolithic)
state, whole = candidate['state'], monolithic['state']
lo, hi = np.searchsorted(whole['step'], (start, stop))
expected_ticks = (whole['step'][lo:hi].astype(np.int64) - start).astype(np.int32)
expected_nodes = whole['neuron_idx'][lo:hi]
checks['slice.event_steps'] = ec.bit_equal(state['step'], expected_ticks)
checks['slice.event_neurons'] = ec.bit_equal(state['neuron_idx'], expected_nodes)
counts = np.bincount(expected_nodes, minlength=len(whole['v'])).astype(np.int32)
checks['slice.segment_counts'] = ec.bit_equal(state['counts'], counts)
requests = monolithic['requests']
lo, hi = np.searchsorted(requests[:, 0], (start, stop))
expected_requests = requests[lo:hi].copy()
expected_requests[:, 0] -= start
checks['slice.requests_five_fields'] = ec.bit_equal(candidate['requests'], expected_requests)
for name in ('v', 'g', 'ref'):
checks['slice.final_' + name + '_raw_hash'] = _raw_sha(state[name]) == monolithic['state_hashes'][stop - 1][name + '_sha256']
raw_ring = ec.reconstructed_fresh_ring(whole, stop, len(whole['v']))
expected_ring = raw_ring[(np.arange(19) + stop % 19) % 19]
checks['slice.normalized_spike_ring'] = ec.bit_equal(ec.normalized_ring(candidate['terminal']), expected_ring)
traces = monolithic['traces']
checks['slice.APL_state'] = ec.bit_equal(candidate['terminal']['apl_state'], traces['state'][stop - 1])
pending = np.zeros((18, traces['generated'].shape[1]), dtype=np.float32)
kept = min(stop, 18)
pending[-kept:] = traces['generated'][stop - kept:stop]
checks['slice.APL_pending'] = ec.bit_equal(candidate['terminal']['apl_pending'], pending)
if observed['apl_traces']:
for name in ec.TRACE_FILES:
checks['slice.APL_trace.' + name] = ec.bit_equal(candidate['traces'][name], traces[name][start:stop])
if observed['state_hashes']:
checks['slice.per_tick_state_hashes'] = all(
all(left[key] == right[key] for key in ec.HASH_FIELDS)
for left, right in zip(candidate['state_hashes'], monolithic['state_hashes'][start:stop]))
if observed['queues']:
checks['slice.queue_order_and_acceptance'] = all(
all(left[key] == right[key] for key in ec.QUEUE_FIELDS)
for left, right in zip(candidate['queues'], monolithic['queues'][start:stop]))
limitations = ['External input/seal/case/parent authority is not established by these records.',
'Queue joint hashes contain local tick headers; their independent CSR reconstruction is not performed here.',
'Numerical equality of this observed slice is not a learning, live-coverage or complete dynamics proof.']
for key, label in (('apl_traces', 'APL histories'), ('state_hashes', 'intermediate V/g/ref state hashes'),
('queues', 'per-tick queue order/acceptance')):
if not observed[key]:
limitations.append('Candidate ' + label + ' are unobserved; no direct comparison is claimed.')
return _report('slice', checks, dict(start=start, stop=stop, segment_steps=steps,
events=True, segment_counts=True, requests_five_fields=True, final_state_against_reference_tick_hash=True,
normalized_spike_ring_from_reference_prefix_events=True, terminal_APL_state_and_pending=True,
candidate_APL_histories=observed['apl_traces'], candidate_per_tick_state_hashes=observed['state_hashes'],
candidate_per_tick_queue_fields=observed['queues'], queue_joint_hashes_compared=False), limitations)
def compare_checkpoint(candidate, prefix):
"""Compare equal absolute endpoints, not local phases or segment counts."""
candidate, _ = _record(candidate, 'candidate')
prefix, _ = _record(prefix, 'prefix')
checks = _common(candidate, prefix)
end = candidate['global_tick_offset'] + candidate['steps']
reference_end = prefix['global_tick_offset'] + prefix['steps']
checks['checkpoint.same_absolute_end'] = end == reference_end
for name in ('v', 'g', 'ref'):
checks['checkpoint.state.' + name] = ec.bit_equal(candidate['state'][name], prefix['state'][name])
checks['checkpoint.normalized_spike_ring'] = ec.bit_equal(
ec.normalized_ring(candidate['terminal']), ec.normalized_ring(prefix['terminal']))
for name in ('apl_state', 'apl_pending'):
checks['checkpoint.' + name] = ec.bit_equal(candidate['terminal'][name], prefix['terminal'][name])
return _report('checkpoint', checks, dict(candidate_absolute_end=end, reference_absolute_end=reference_end,
candidate_local_steps=candidate['steps'], reference_local_steps=prefix['steps'], terminal_v_g_ref=True,
normalized_spike_ring=True, terminal_APL_state_and_pending=True, counts_compared=False,
event_histories_compared=False, per_tick_state_histories_compared=False), [
'Segment counts and cumulative prefix counts are deliberately not compared in this endpoint scope.',
'Matching terminal records do not prove matching intermediate states or complete event histories.',
'External input/seal/parent authority and live-carry coverage remain separate gates.'])
def compare_parent_carry(candidate, parent):
"""Check recorded short-segment carry invariants, not unobserved restoration.
The parent ends where the candidate starts. Slots already overwritten by
this segment cannot be checked as initial carry from terminal data alone.
When present, the child's applied trace directly tests consumed APL rows.
"""
candidate, observed = _record(candidate, 'candidate')
parent, _ = _record(parent, 'parent')
_require(candidate['global_tick_offset'] > 0, 'Parent-carry candidate must be a continued segment')
checks = _common(candidate, parent)
start, steps = candidate['global_tick_offset'], candidate['steps']
parent_end = parent['global_tick_offset'] + parent['steps']
checks['parent.contiguous_absolute_clock'] = parent_end == start
restored_ring = ec.normalized_ring(parent['terminal'])
unconsumed_spikes, unconsumed_apl = steps < 19, steps < 18
if unconsumed_spikes:
checks['parent.unwritten_spike_slots'] = ec.bit_equal(
candidate['terminal']['spike_ring'][steps:], restored_ring[steps:])
pending = parent['terminal']['apl_pending']
if unconsumed_apl:
checks['parent.unconsumed_APL_prefix'] = ec.bit_equal(
candidate['terminal']['apl_pending'][:18 - steps], pending[steps:])
consume = min(18, steps)
if observed['apl_traces']:
checks['parent.applied_APL_prefix'] = ec.bit_equal(candidate['traces']['applied'][:consume], pending[:consume])
limitations = ['Initial V/g/ref/APL-state restoration is not directly observed by this terminal-only carry scope.',
'Overwritten spike slots and consumed pending slots require the separate full-reference slice/endpoint gates.',
'Parent manifest hashes and ancestor provenance must be independently bound by the coordinator.',
'No queue arrival/acceptance reconstruction, live-carry or learning success is claimed here.']
if not observed['apl_traces']:
limitations.append('Candidate applied APL history is unobserved; consumed parent APL values are not directly compared.')
carry_observed = unconsumed_spikes or unconsumed_apl or observed['apl_traces']
if not carry_observed:
limitations.append('No carry array remains directly observable in this long light segment; this scope checks clock/shape/weight identity only.')
return _report('parent_carry', checks, dict(parent_absolute_start=parent['global_tick_offset'],
parent_absolute_end=parent_end, candidate_absolute_start=start, candidate_absolute_end=start + steps,
candidate_local_steps=steps, unwritten_spike_slots_checked=unconsumed_spikes,
unwritten_spike_slot_count=max(19 - steps, 0), unconsumed_APL_prefix_checked=unconsumed_apl,
unconsumed_APL_row_count=max(18 - steps, 0), applied_parent_APL_prefix_checked=observed['apl_traces'],
applied_parent_APL_rows_compared=consume if observed['apl_traces'] else None,
any_carry_values_directly_compared=carry_observed,
initial_neural_state_restoration_directly_observed=False), limitations)
@@ -0,0 +1,421 @@
"""Finite, no-training execution of the separately registered 32-call gate.
All files are exclusive creations. A partial/failed row is preserved and is
not retried. Runtime outputs are sealed before external reports are written.
The preparation module owns source/input authority; no summary Boolean alone
is accepted as proof that a subprocess or a complete engineering gate passed.
"""
from __future__ import annotations
import argparse
from datetime import datetime, timezone
import importlib
import json
import os
from pathlib import Path
import shutil
import subprocess
import sys
import time
import traceback
import urllib.request
import zipfile
import numpy as np
import engineering_inputs as I
import engineering_plan as PLAN
import runtime_light as R
import runtime_continuation as C
GPU = 'GPU-79f3a85d-adce-7833-7003-6488140ca596'
ROOT = Path('/data/digital-drosophila')
MODEL = Path(__file__).resolve().parent
EXP = ROOT / 'experiments/phase4an-engineering-validation'
require = R.require
def canonical(value):
return json.dumps(value, sort_keys=True, separators=(',', ':'), allow_nan=False)
def same_json(left, right):
return canonical(left) == canonical(right)
def preparation():
return importlib.import_module('engineering_prepare')
def row_from(registration, row_id):
plan = PLAN.validate_plan(registration['plan'])
rows = [row for row in plan['rows'] if row['row_id'] == row_id]
require(len(rows) == 1, 'Only an exact registered row may execute')
return rows[0]
def locations(row_id):
require(type(row_id) is str and row_id and all(ch.isalnum() or ch in '-_' for ch in row_id),
'Unsafe row identity')
return dict(output=EXP / 'runs' / row_id,
runtime=EXP / 'receipts' / (row_id + '.runtime.json'),
execution=EXP / 'receipts' / (row_id + '.json'),
derivation=EXP / 'receipts' / (row_id + '.derivation.json'),
stdout=EXP / 'logs' / (row_id + '.stdout.log'),
stderr=EXP / 'logs' / (row_id + '.stderr.log'))
def disk_budget(registration):
limit = registration['budget']['output_budget_bytes']
reserve = registration['budget']['minimum_free_reserve_bytes']
require(type(limit) is type(reserve) is int and limit > 0 and reserve > 0, 'Invalid registered budget')
files = [p for p in EXP.rglob('*') if p.is_file()]
require(not any(p.is_symlink() for p in EXP.rglob('*')), 'Experiment contains a linked output')
used = sum(p.stat().st_size for p in files)
free = shutil.disk_usage(EXP).free
require(used <= limit and free >= reserve + max(0, limit - used), 'Engineering budget/reserve exceeded; preserve partials')
return dict(used_bytes=used, free_bytes=free, hard_cap_bytes=limit, reserve_bytes=reserve)
def preflight():
"""Never evict an existing service; configure only this child process."""
os.environ['CUDA_VISIBLE_DEVICES'] = GPU
os.environ['CUDA_HOME'] = '/usr/local/cuda'
os.environ['PATH'] = str(ROOT / 'models/male-cns-lif/.venv/bin') + ':/usr/local/cuda/bin:' + os.environ['PATH']
os.environ['MAX_JOBS'] = '2'
os.environ['TORCH_CUDA_ARCH_LIST'] = '8.9'
for port in (8188, 8189, 8190):
with urllib.request.urlopen(f'http://192.168.10.51:{port}/queue', timeout=5) as response:
queue = json.load(response)
require(queue['queue_running'] == [] and queue['queue_pending'] == [],
'ComfyUI busy: defer rather than evict')
def derive_case(template, row, parent_sha):
require(type(template) is dict and set(template) == {'schema', 'row_id', 'original_spec', 'start', 'stop',
'budget_root', 'output_budget_bytes'}, 'Unexpected registered case template fields')
require(template['schema'] == 'phase4an-engineering-case-template-v1' and template['row_id'] == row['row_id'] and
type(template['start']) is int and type(template['stop']) is int and
(template['start'], template['stop']) == (row['start'], row['stop']), 'Template differs from exact plan')
require((row['parent_row_id'] is None) == (parent_sha is None), 'Parent linkage differs from registered plan')
return I.derive_spec(template['original_spec'], start=row['start'], stop=row['stop'],
budget_root=template['budget_root'], output_budget_bytes=template['output_budget_bytes'], parent_sha=parent_sha)
def npz_storage(path):
with zipfile.ZipFile(path) as archive:
rows = archive.infolist()
require(rows and all(r.compress_type == zipfile.ZIP_DEFLATED and r.filename.endswith('.npy') for r in rows),
'Declared lossless compressed state format differs from actual archive')
return dict(file_bytes=Path(path).stat().st_size,
uncompressed_entry_bytes=sum(r.file_size for r in rows),
compressed_entry_bytes=sum(r.compress_size for r in rows),
entries=[dict(name=r.filename, bytes=r.file_size, compressed_bytes=r.compress_size,
compression=r.compress_type) for r in rows])
def runtime_environment():
"""Read settings after the runtime imported Torch; change none of them."""
torch = sys.modules.get('torch')
require(torch is not None and os.environ.get('CUDA_VISIBLE_DEVICES') == GPU and
torch.cuda.device_count() == 1 and '4090' in torch.cuda.get_device_name(0),
'Actual runtime is not the registered single RTX4090')
properties = torch.cuda.get_device_properties(0)
inventory = subprocess.run(['nvidia-smi', '-i', GPU,
'--query-gpu=uuid,name,memory.total,driver_version', '--format=csv,noheader'],
check=True, capture_output=True, text=True, timeout=20).stdout.strip()
require(inventory.startswith(GPU + ','), 'GPU hardware inventory does not match the registered UUID')
return dict(python_executable=sys.executable, python_version=sys.version, numpy_version=np.__version__,
torch_version=str(torch.__version__), torch_cuda_version=torch.version.cuda,
device_name=properties.name, total_memory_bytes=int(properties.total_memory),
gpu_uuid=GPU, nvidia_smi_inventory=inventory,
matmul_allow_tf32=bool(torch.backends.cuda.matmul.allow_tf32),
cudnn_allow_tf32=bool(torch.backends.cudnn.allow_tf32),
deterministic_algorithms=bool(torch.are_deterministic_algorithms_enabled()),
cudnn_deterministic=bool(torch.backends.cudnn.deterministic),
cudnn_benchmark=bool(torch.backends.cudnn.benchmark),
float32_matmul_precision=torch.get_float32_matmul_precision(),
CUBLAS_WORKSPACE_CONFIG=os.environ.get('CUBLAS_WORKSPACE_CONFIG'),
TORCH_CUDA_ARCH_LIST=os.environ.get('TORCH_CUDA_ARCH_LIST'),
settings_changed_by_environment_recorder=False,
environment_recorder_adds_no_timing=True,
build_timing_source='Use inherited summary.build_or_cache_seconds; build/cache is also included in runtime wall time')
def verify_environment(recorded, component):
require(type(recorded) is type(component) is dict and
recorded.get('gpu_uuid') == GPU and recorded.get('settings_changed_by_environment_recorder') is False,
'Missing or invalid actual environment receipt')
require(component.get('cuda_matmul_allow_tf32') is False and
component.get('float32_matmul_precision') == 'highest' and
component.get('precision_configuration_changed_by_component') is False,
'Frozen AM precision configuration changed; never correct it automatically')
pairs = {'matmul_allow_tf32': 'cuda_matmul_allow_tf32', 'cudnn_allow_tf32': 'cudnn_allow_tf32',
'deterministic_algorithms': 'deterministic_algorithms', 'cudnn_deterministic': 'cudnn_deterministic',
'cudnn_benchmark': 'cudnn_benchmark', 'float32_matmul_precision': 'float32_matmul_precision',
'CUBLAS_WORKSPACE_CONFIG': 'CUBLAS_WORKSPACE_CONFIG', 'device_name': 'device_name',
'torch_version': 'torch', 'torch_cuda_version': 'cuda', 'numpy_version': 'numpy'}
for left, right in pairs.items():
require(left in recorded and right in component and same_json(recorded[left], component[right]),
'Actual runtime/APL environment disagreement: ' + left)
require(type(recorded['device_name']) is str and '4090' in recorded['device_name'], 'Wrong physical device class')
def check_runtime_record(reg, reg_sha, row, record):
"""Recheck actual bytes, runtime completion and parent chain, not intent."""
paths = locations(row['row_id'])
require(record.get('complete') is True and record.get('registration_sha256') == reg_sha and
same_json(record.get('row'), row) and record.get('training') is False,
'Runtime record is not bound to this registered row')
case = Path(reg['cases'][row['row_id']]['case_dir'])
checkpoint = C.load_checkpoint(paths['output'], record['checkpoint_manifest_sha256'])
summary = checkpoint['summary']
verify_environment(record['environment'], summary['apl_component']['environment'])
require(checkpoint['steps'] == row['steps'] and checkpoint['global_tick_offset'] == row['start'] and
checkpoint['global_next_tick'] == row['stop'] and summary['nodes'] == 166700 and summary['edges'] == 25582938 and
summary['diagnostic_mode'] == row['mode'], 'Runtime clock/population/mode differs from plan')
require(record['case_sha256'] == R.digest(case / 'spec.json') and
record['requests_sha256'] == R.digest(case / 'requests.npz') and
record['derivation_sha256'] == R.digest(paths['derivation']), 'Derived input/receipt bytes changed')
template = R._read_json(case / 'template-spec.json')
parent_pin = None
if row['parent_row_id'] is not None:
parent = verify_execution(reg, reg_sha, row_from(reg, row['parent_row_id']))
parent_pin = parent['checkpoint_manifest_sha256']
expected, derivation = derive_case(template, row, parent_pin)
require(same_json(R._read_json(case / 'spec.json'), expected) and
same_json(R._read_json(paths['derivation']), derivation), 'Case was not generated by the registered exact rule')
require(record['parent_checkpoint_manifest_sha256'] == parent_pin and
summary.get('parent_checkpoint_manifest_sha256') == parent_pin, 'Execution parent SHA mismatch')
require(record['checkpoint_inventory_sha256'] == R.digest(paths['output'] / C.MANIFEST), 'Checkpoint manifest changed')
for name in ('state.npz', 'terminal-queues.npz'):
require(same_json(record['storage'][name], npz_storage(paths['output'] / name)), 'Storage measurement changed')
return record
def verify_execution(reg, reg_sha, row):
paths = locations(row['row_id'])
receipt = R._read_json(paths['execution'])
require(receipt.get('complete') is True and type(receipt.get('process_exit_code')) is int and receipt['process_exit_code'] == 0 and
receipt.get('registration_sha256') == reg_sha and receipt.get('row_id') == row['row_id'],
'Missing completed process receipt')
for field, key in (('runtime_record_sha256', 'runtime'), ('stdout_sha256', 'stdout'), ('stderr_sha256', 'stderr')):
require(receipt[field] == R.digest(paths[key]), 'Execution log/runtime receipt changed')
require(not (EXP / 'audit/stopped.json').exists(), 'A failed/deferred run requires explicit review, not automatic retry')
record = check_runtime_record(reg, reg_sha, row, R._read_json(paths['runtime']))
require(receipt['checkpoint_manifest_sha256'] == record['checkpoint_manifest_sha256'], 'Execution/checkpoint linkage mismatch')
return record
def worker(reg_sha, row_id):
require(sys.dont_write_bytecode, 'Use the scientific venv with -B')
reg = preparation().read_registration(reg_sha)
row = row_from(reg, row_id)
paths = locations(row_id)
require(not paths['output'].exists() and not paths['runtime'].exists() and not paths['derivation'].exists(),
'Never overwrite/retry a partial row')
budget_before = disk_budget(reg)
preflight()
parent_sha = None
parent_dir = None
if row['parent_row_id'] is not None:
parent_record = verify_execution(reg, reg_sha, row_from(reg, row['parent_row_id']))
parent_sha = parent_record['checkpoint_manifest_sha256']
parent_dir = locations(row['parent_row_id'])['output']
case = Path(reg['cases'][row_id]['case_dir'])
template = R._read_json(case / 'template-spec.json')
spec, derivation = derive_case(template, row, parent_sha)
require(not (case / 'spec.json').exists(), 'Derived spec already exists; preserve partial instead of rerun')
R._save(case / 'spec.json', spec)
R._save(paths['derivation'], derivation)
overlay = reg['overlays'][row['overlay']]
start_utc = datetime.now(timezone.utc).isoformat()
start = time.perf_counter()
if parent_dir is None:
function = R.run_reference if row['mode'] == 'reference' else R.run_light
function(case, paths['output'], EXP / 'extension-build', overlay['overlay_file'], overlay['overlay_manifest'])
pin = C.seal_checkpoint(paths['output'])['sha256']
else:
result = C.run_continued(case, paths['output'], EXP / 'extension-build',
overlay['overlay_file'], overlay['overlay_manifest'], parent_dir, mode=row['mode'])
pin = result['checkpoint_manifest_sha256']
elapsed = time.perf_counter() - start
budget_after = disk_budget(reg)
record = dict(schema='phase4an-engineering-runtime-record-v1', complete=True, training=False,
registration_sha256=reg_sha, row=row, case_sha256=R.digest(case / 'spec.json'),
requests_sha256=R.digest(case / 'requests.npz'), derivation_sha256=R.digest(paths['derivation']),
checkpoint_manifest_sha256=pin, checkpoint_inventory_sha256=R.digest(paths['output'] / C.MANIFEST),
parent_checkpoint_manifest_sha256=parent_sha, runtime_and_immediate_seal_wall_seconds=elapsed,
started_utc=start_utc, completed_utc=datetime.now(timezone.utc).isoformat(),
environment=runtime_environment(),
budget_before=budget_before, budget_after=budget_after,
storage={name: npz_storage(paths['output'] / name) for name in ('state.npz', 'terminal-queues.npz')},
no_result_selected_retry=True, learning_pass_claimed=False)
# Do not write this or any later report INSIDE the sealed checkpoint.
check_runtime_record(reg, reg_sha, row, record)
preparation().read_registration(reg_sha)
R._save(paths['runtime'], record)
print('COMPLETED ENGINEERING ROW', row_id, flush=True)
def run_registered(reg_sha):
require(sys.dont_write_bytecode, 'Use python -B')
reg = preparation().read_registration(reg_sha)
require(not (EXP / 'audit/stopped.json').exists() and not (EXP / 'audit/execution-complete.json').exists(),
'Stopped/completed gate must not be restarted')
started = time.perf_counter()
completed = []
try:
for row in reg['plan']['rows']:
paths = locations(row['row_id'])
if paths['execution'].exists():
verify_execution(reg, reg_sha, row)
completed.append(dict(row_id=row['row_id'], execution_sha256=R.digest(paths['execution']),
reverified_existing=True))
continue
require(not any(paths[k].exists() for k in ('output', 'runtime', 'stdout', 'stderr', 'derivation')),
'Partial row present; no automatic retry')
disk_budget(reg)
preflight()
began = time.perf_counter()
print('START', row['row_id'], flush=True)
with paths['stdout'].open('xb') as out, paths['stderr'].open('xb') as err:
result = subprocess.run([sys.executable, '-B', str(Path(__file__)), 'worker',
'--registration-sha256', reg_sha, '--row-id', row['row_id']], cwd=MODEL, stdout=out, stderr=err)
require(result.returncode == 0, 'Worker failed; preserve its stdout/stderr and partial output')
record = check_runtime_record(reg, reg_sha, row, R._read_json(paths['runtime']))
receipt = dict(schema='phase4an-engineering-execution-receipt-v1', complete=True,
process_exit_code=result.returncode, row_id=row['row_id'], registration_sha256=reg_sha,
runtime_record_sha256=R.digest(paths['runtime']), stdout_sha256=R.digest(paths['stdout']),
stderr_sha256=R.digest(paths['stderr']), process_wall_seconds=time.perf_counter() - began,
checkpoint_manifest_sha256=record['checkpoint_manifest_sha256'], scientific_pass_claimed=False)
R._save(paths['execution'], receipt)
verify_execution(reg, reg_sha, row)
completed.append(dict(row_id=row['row_id'], execution_sha256=R.digest(paths['execution']),
reverified_existing=False))
print('COMPLETE', row['row_id'], flush=True)
require(len(completed) == 32, 'Incomplete registered inventory')
preparation().read_registration(reg_sha)
R._save(EXP / 'audit/execution-complete.json', dict(complete=True, registration_sha256=reg_sha,
completed=completed, expected_calls=32, executed_ticks=11074, wall_seconds=time.perf_counter() - started,
numerical_gate_evaluated=False, training=False, learning_pass_claimed=False))
except Exception as error:
R._save(EXP / 'audit/stopped.json', dict(complete=False, type=type(error).__name__, message=str(error),
traceback=traceback.format_exc(), completed_before_error=completed, partial_outputs_preserved=True,
no_automatic_retry=True, scientific_failure_not_inferred_from_engineering_failure=True))
raise
def verified_inventory(reg, reg_sha):
"""A completed child and every planned descriptor must remain auditable."""
plan = PLAN.validate_plan(reg['plan'])
receipt = R._read_json(EXP / 'audit/execution-complete.json')
require(receipt.get('complete') is True and receipt.get('registration_sha256') == reg_sha and
type(receipt.get('expected_calls')) is int and receipt['expected_calls'] == plan['total_calls'] and
type(receipt.get('executed_ticks')) is int and receipt['executed_ticks'] == plan['total_ticks'] and
receipt.get('training') is False and receipt.get('learning_pass_claimed') is False,
'Execution inventory is not a complete no-training registered gate')
require(type(receipt.get('completed')) is list and len(receipt['completed']) == plan['total_calls'],
'Execution inventory truncated')
records = {}
for row, entry in zip(plan['rows'], receipt['completed']):
require(entry.get('row_id') == row['row_id'] and
entry.get('execution_sha256') == R.digest(locations(row['row_id'])['execution']),
'Execution completion inventory reordered or receipt changed')
records[row['row_id']] = verify_execution(reg, reg_sha, row)
return records
def evaluate(reg_sha):
reg = preparation().read_registration(reg_sha)
records = verified_inventory(reg, reg_sha)
report = importlib.import_module('engineering_analysis').analyze(reg, reg_sha, records)
require(type(report) is dict and report.get('training') is False and report.get('learning_pass_claimed') is False,
'Engineering analysis must not claim training or learning')
preparation().read_registration(reg_sha)
return reg, report
def analyze(reg_sha):
require(sys.dont_write_bytecode, 'Use python -B')
target = EXP / 'audit/engineering-analysis.json'
require(not target.exists() and not (EXP / 'manifest.json').exists(), 'Analysis/seal already exists; do not overwrite')
began = time.perf_counter()
_, report = evaluate(reg_sha)
R._save(target, report)
R._save(EXP / 'audit/analysis-execution.json', dict(registration_sha256=reg_sha,
analysis_sha256=R.digest(target), wall_seconds=time.perf_counter() - began, GPU_used=False,
training=False, learning_pass_claimed=False))
print(report['status'], flush=True)
def seal_experiment():
"""Seal only the new engineering experiment, not the unfinished lab model."""
controls = ('manifest.json', 'checksums.sha256')
require(not any((EXP / name).exists() for name in controls), 'Do not replace an existing or partial seal')
paths = sorted(EXP.rglob('*'))
require(not any(path.is_symlink() for path in paths), 'Cannot seal linked artifacts')
require(not any('__pycache__' in path.relative_to(EXP).parts for path in paths), 'Unexpected bytecode cache in new experiment')
rows = [dict(path=path.relative_to(EXP).as_posix(), bytes=path.stat().st_size, sha256=R.digest(path))
for path in paths if path.is_file()]
require(rows, 'Cannot seal an empty experiment')
# Only ROOT controls are excluded from their own hash. Nested checkpoint
# manifests and source snapshots are ordinary payload and remain included.
R._save(EXP / controls[0], dict(root=str(EXP), files=rows))
with (EXP / controls[1]).open('x', encoding='utf-8', newline='\n') as stream:
stream.writelines(row['sha256'] + ' ' + row['path'] + '\n' for row in rows)
stream.write(R.digest(EXP / controls[0]) + ' manifest.json\n')
return dict(manifest_sha256=R.digest(EXP / controls[0]), checksums_sha256=R.digest(EXP / controls[1]))
def finalize(reg_sha):
require(sys.dont_write_bytecode, 'Use python -B')
require(not (EXP / 'manifest.json').exists() and not (EXP / 'audit/final-status.json').exists(),
'Final/partial seal already exists; preserve it')
reg, actual = evaluate(reg_sha)
target = EXP / 'audit/engineering-analysis.json'
require(same_json(actual, R._read_json(target)), 'Recomputed analysis differs from saved evidence')
execution = R._read_json(EXP / 'audit/analysis-execution.json')
require(execution.get('registration_sha256') == reg_sha and execution.get('analysis_sha256') == R.digest(target),
'Saved analysis is not bound to its execution receipt')
authority = preparation().verify_authority(reg['anchors'])
old = preparation().verify_old_integrity(reg)
R._save(EXP / 'regression/AM-seals-after.json', authority)
R._save(EXP / 'regression/old-integrity-after.json', old)
preparation().read_registration(reg_sha)
final = dict(schema='phase4an-engineering-final-v1', status=actual['status'],
registration_sha256=reg_sha, analysis_sha256=R.digest(target),
analysis_recomputed_exactly=True, completed_calls=32, executed_ticks=11074,
old_integrity_pass=True, old_integrity_roots=len(old), AM_seals_preserved=True,
old_five_reference_evidence='Sealed AM five-trial evidence retained; not rerun by this engineering gate',
training=False, learning_pass_claimed=False, learned_candidate=False,
model_directory_sealed=False, used_engineering_source_snapshots_sealed=True,
budget=disk_budget(reg),
scope='Finite registered 100 ms input and fixed continuation cuts only; no universal determinism, learning or physiology claim')
R._save(EXP / 'audit/final-status.json', final)
pins = seal_experiment()
helper = preparation().load_module('phase4an_pinned_AI_publication',
ROOT / preparation().AI_HELPER_REL, preparation().AI_HELPER_SHA)
verified = helper.verify_seal(EXP)
require(verified['manifest_sha256'] == pins['manifest_sha256'] and
verified['checksums_sha256'] == pins['checksums_sha256'], 'Final experiment seal verification failed')
print(canonical(dict(status=final['status'], **pins)), flush=True)
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('action', choices=('run', 'worker', 'analyze', 'finalize'))
parser.add_argument('--registration-sha256', required=True)
parser.add_argument('--row-id')
args = parser.parse_args()
if args.action == 'worker':
require(args.row_id is not None, 'Worker requires a row ID')
worker(args.registration_sha256, args.row_id)
else:
require(args.row_id is None, 'Full inventory run has no selected row')
{'run': run_registered, 'analyze': analyze, 'finalize': finalize}[args.action](args.registration_sha256)
if __name__ == '__main__':
main()
@@ -0,0 +1,478 @@
"""Unregistered fixed-weight continuation of the reporting-only AN executor.
run_continued(case, out, build, overlay, manifest, checkpoint_dir, mode='light')
also accepts mode='reference'. Inputs/events/counts use segment-local ticks;
summary and receipts expose global_tick_offset/global_next_tick. No training,
weight update, early stopping, or implicit extra tail is performed.
The caller must externally pin spec.parent_checkpoint_manifest_sha256. A fresh
parent's original runtime receipt did NOT bind state.npz: seal_checkpoint adds
that binding, after checking the available original receipts and current bytes.
It cannot retroactively prove that an unsealed state file was never modified.
Full graph/APL/coefficient/preregistered-input authority remains the coordinator's
responsibility, as in runtime_light. No executable/venv path is resolved here.
"""
from __future__ import annotations
import ast
import copy
import hashlib
import importlib.util
from pathlib import Path
import sys
import numpy as np
LIGHT_SHA = '522b8c31ea938890fe15364fccd25ccc5bded54101754909f255532e7ad01aae'
N = 166700
E = 25582938
MANIFEST = 'checkpoint-manifest.json'
RECEIPT = 'continuation-receipt.json'
RESTORE = '_continuation.restore(v, g, ref, counts, ring, apl_ring, apl, upload)'
CLOCK_OLD = 'tick >= 18'
CLOCK_NEW = 'tick + _continuation.global_offset >= 18'
def _light():
path = Path(__file__).resolve().parent / 'runtime_light.py'
raw = path.read_bytes()
if hashlib.sha256(raw).hexdigest() != LIGHT_SHA:
raise ValueError('Fresh runtime source pin changed')
name = 'phase4an_continuation_pinned_light'
existing = sys.modules.get(name)
if existing is not None:
if getattr(existing, '__continuation_pin__', None) != LIGHT_SHA or Path(existing.__file__).resolve() != path:
raise ValueError('Unverified runtime module binding')
return existing
spec = importlib.util.spec_from_file_location(name, path)
module = importlib.util.module_from_spec(spec)
sys.modules[name] = module
try:
exec(compile(raw, str(path), 'exec'), module.__dict__)
except BaseException:
sys.modules.pop(name, None)
raise
module.__continuation_pin__ = LIGHT_SHA
return module
R = _light() # Pure NumPy/stdlib; this does not import Torch or run a brain.
def rotate_spike_ring(raw_ring, phase_tick):
"""Parent's local next-write slot becomes the child's local slot zero."""
R.require(isinstance(raw_ring, np.ndarray) and raw_ring.dtype == np.uint8 and
raw_ring.ndim == 2 and raw_ring.shape[0] == 19, 'Invalid spike ring dtype/shape')
R.require(type(phase_tick) is int and phase_tick >= 0 and np.all(raw_ring <= 1), 'Invalid ring phase/content')
return raw_ring[(np.arange(19) + phase_tick % 19) % 19].copy()
def _tree_hashes(directory):
"""Exact regular-file inventory, no symlink/reparse escapes, no self-hash."""
directory = Path(directory).resolve(strict=True)
R.require(directory.is_dir(), 'Checkpoint must be a directory')
result = {}
for path in sorted(directory.rglob('*')):
R.require(not path.is_symlink() and path.resolve().is_relative_to(directory), 'Checkpoint contains a linked/escaping path')
if path.is_dir():
continue
R.require(path.is_file(), 'Checkpoint contains a nonregular entry')
relative = path.relative_to(directory).as_posix()
if relative == MANIFEST:
continue
R.require('error' not in path.name.lower() and 'failure' not in path.name.lower(), 'Parent has failure/error evidence')
result[relative] = R.digest(path)
return result
def _npz(path, keys):
with np.load(path, allow_pickle=False) as archive:
R.require(len(archive.files) == len(keys) and set(archive.files) == set(keys), 'Unexpected checkpoint array keys: ' + str(path))
return {name: archive[name].copy() for name in keys}
def validate_state(state, terminal, steps, *, fresh):
"""Validate local events/counts and the slots written by this segment."""
R.require(type(steps) is int and 0 < steps <= np.iinfo(np.int32).max, 'Invalid segment length')
for name in ('v', 'g'):
value = state[name]
R.require(value.dtype == np.float64 and value.shape == (N,) and np.isfinite(value).all(), 'Invalid parent ' + name)
for name in ('ref', 'counts'):
value = state[name]
R.require(value.dtype == np.int32 and value.shape == (N,) and np.all(value >= 0), 'Invalid parent ' + name)
R.require(np.all(state['ref'] <= 22), 'Parent refractory counter exceeds frozen 22-step contract')
tick, neuron = state['step'], state['neuron_idx']
R.require(tick.dtype == neuron.dtype == np.int32 and tick.ndim == neuron.ndim == 1 and tick.shape == neuron.shape,
'Invalid sparse event dtype/shape')
R.require(np.all((tick >= 0) & (tick < steps)) and np.all((neuron >= 0) & (neuron < N)), 'Parent event outside segment')
key = tick.astype(np.int64) * N + neuron
R.require(len(key) < 2 or np.all(key[1:] > key[:-1]), 'Parent events must be sorted and unique')
R.require(np.array_equal(np.bincount(neuron, minlength=N), state['counts']), 'Parent counts are not segment-local event counts')
ring = terminal['spike_ring']
R.require(ring.dtype == np.uint8 and ring.shape == (19, N) and np.all(ring <= 1), 'Invalid parent spike ring')
for name, shape in (('apl_state', (2,)), ('apl_pending', (18, 4064))):
value = terminal[name]
R.require(value.dtype == np.float32 and value.shape == shape and np.isfinite(value).all(), 'Invalid terminal ' + name)
phase = terminal['next_tick']
R.require(phase.dtype == np.int64 and phase.shape == () and phase.item() == steps, 'Terminal local phase does not match segment length')
for local_tick in range(max(0, steps - 19), steps):
start, stop = np.searchsorted(tick, (local_tick, local_tick + 1))
expected = np.zeros(N, dtype=np.uint8)
expected[neuron[start:stop]] = 1
R.require(np.array_equal(ring[local_tick % 19], expected), 'Spike ring does not match emitted segment events')
if fresh and steps < 19:
R.require(not np.any(ring[steps:]), 'Fresh parent has nonzero uninitialized spike slots')
return state, terminal
def _verify_receipt_hashes(directory, receipt):
for key, name in (('summary_sha256', 'summary.json'), ('finite_audit_sha256', 'finite-audit.json'),
('terminal_queues_sha256', 'terminal-queues.npz'), ('overlay_audit_sha256', 'overlay-audit.json')):
R.require(receipt.get(key) == R.digest(directory / name), 'Parent receipt hash mismatch: ' + name)
inputs = receipt.get('input_sha256')
R.require(isinstance(inputs, dict) and inputs, 'Parent receipt lacks immutable input bindings')
for name, sha in inputs.items():
R.require(Path(name).is_absolute() and R._sha(sha) and R.digest(name) == sha, 'Parent input/source bytes changed: ' + name)
def _validate_record(directory, seen):
summary = R._read_json(directory / 'summary.json')
finite = R._read_json(directory / 'finite-audit.json')
provenance = R._read_json(directory / 'runtime-provenance.json')
steps = summary.get('steps')
R.require(summary.get('complete') is True and summary.get('pass_all') is True and summary.get('all_finite') is True and
summary.get('completed_ticks') == steps and summary.get('nodes') == N and summary.get('edges') == E and
summary.get('precision') == 'float64' and summary.get('apl_precision') == 'float32' and
summary.get('within_trial_weight_updates') is False, 'Parent is not a complete fixed-weight fullgraph trial')
R.require(finite.get('all_finite') is True and finite.get('first_failure_tick') == -1 and
finite.get('finite_observed_ticks') == steps, 'Parent finite coverage failed')
R.require(provenance.get('complete') is True and provenance.get('weights_updated_during_run') is False and
provenance.get('old_globals_patched') is False, 'Invalid parent runtime receipt')
_verify_receipt_hashes(directory, provenance)
fresh = provenance.get('schema') == 'phase4an-reporting-adapter-v1'
if fresh:
R.require(provenance.get('source_sha256') == LIGHT_SHA and R.digest(provenance['source_file']) == LIGHT_SHA and
summary.get('checkpoint_continuation_used') is False, 'Fresh parent source/continuation scope mismatch')
offset = 0
else:
R.require(provenance.get('schema') == 'phase4an-continuation-runtime-v1' and
provenance.get('source_sha256') == R.digest(__file__) and
R.digest(provenance['source_file']) == R.digest(__file__) and
summary.get('checkpoint_continuation_used') is True, 'Unrecognized continued parent source')
receipt = R._read_json(directory / RECEIPT)
R.require(receipt.get('schema') == 'phase4an-continuation-receipt-v1' and receipt.get('complete') is True,
'Invalid continuation receipt')
R.require(receipt.get('weights_updated_during_run') is False and receipt.get('segment_steps') == steps and
receipt.get('counts_scope') == 'segment_only_not_lifetime' and
provenance.get('state_sha256') == R.digest(directory / 'state.npz'), 'Continued state/scope receipt mismatch')
expected_payload = _tree_hashes(directory)
expected_payload.pop(RECEIPT, None)
R.require(receipt.get('payload_sha256') == expected_payload, 'Continuation receipt does not bind the exact payload')
parent_pin = receipt.get('parent_checkpoint_manifest_sha256')
case_spec = _parent_spec({'summary': summary, 'provenance': provenance})
R.require(R._sha(parent_pin) and provenance.get('parent_checkpoint_manifest_sha256') == parent_pin and
summary.get('parent_checkpoint_manifest_sha256') == parent_pin and
case_spec.get('parent_checkpoint_manifest_sha256') == parent_pin,
'Parent checkpoint pin disagrees across case/summary/provenance/continuation receipts')
R.require(Path(receipt['parent_checkpoint_dir']).is_absolute(), 'Parent checkpoint directory is not absolute')
parent = load_checkpoint(receipt['parent_checkpoint_dir'], parent_pin, _seen=seen)
offset = parent['global_next_tick']
R.require(receipt.get('global_tick_offset') == offset and receipt.get('global_next_tick') == offset + steps and
summary.get('global_tick_offset') == offset and summary.get('global_next_tick') == offset + steps and
provenance.get('global_tick_offset') == offset and provenance.get('global_next_tick') == offset + steps,
'Continuation global clock chain mismatch')
R.require(summary.get('effective_weights_raw_sha256') == parent['summary'].get('effective_weights_raw_sha256'),
'Retention effective weights changed between parent and child')
state = _npz(directory / 'state.npz', ('step', 'neuron_idx', 'counts', 'v', 'g', 'ref'))
terminal = _npz(directory / 'terminal-queues.npz', ('spike_ring', 'apl_state', 'apl_pending', 'next_tick'))
validate_state(state, terminal, steps, fresh=fresh)
initial_padding = max(18 - (offset + steps), 0)
R.require(np.all(terminal['apl_pending'][:initial_padding].view(np.uint32) == 0),
'Unconsumed original APL padding is not positive zero')
if not fresh:
if steps < 19:
restored = rotate_spike_ring(parent['terminal']['spike_ring'], int(parent['terminal']['next_tick']))
R.require(np.array_equal(terminal['spike_ring'][steps:], restored[steps:]), 'Unwritten carried spike slots changed')
if steps < 18:
carried = parent['terminal']['apl_pending'][steps:]
R.require(np.array_equal(terminal['apl_pending'][:18 - steps].view(np.uint32), carried.view(np.uint32)),
'Unconsumed carried APL slots changed')
overlay = R._read_json(directory / 'overlay-audit.json')
R.require(overlay.get('complete') is True and overlay.get('weights_updated_during_run') is False and
overlay.get('effective_weights_raw_sha256') == summary.get('effective_weights_raw_sha256') and
overlay.get('mask_sha256') == R.MASK_SHA and overlay.get('am_helper_sha256') == R.AM_OVERLAY_SHA,
'Parent effective overlay receipt mismatch')
overlay_manifest = R._read_json(directory / 'overlay-manifest.json')
R.require(R.digest(directory / 'overlay-input.npz') == overlay_manifest.get('overlay_sha256') and
R.digest(directory / 'overlay-manifest.json') in provenance['input_sha256'].values() and
overlay_manifest.get('mask_sha256') == R.MASK_SHA and
overlay_manifest.get('base_candidate_sha256') == overlay.get('base_candidate_sha256'),
'Parent copied overlay inputs do not match their original receipt')
timing = R._read_json(directory / 'apl-timing-audit.json')
with (directory / 'apl-pending-tail.float32.npy').open('rb') as stream:
pending = np.load(stream, allow_pickle=False)
R.require(pending.dtype == np.float32 and pending.shape == (18, 4064) and
pending.tobytes() == terminal['apl_pending'].tobytes(), 'Parent terminal/pending output mismatch')
R.require(timing.get('pending_raw_sha256') == R._raw(pending) and timing.get('apl_delay_steps') == 18 and
timing.get('brain_tail_advanced') is False and
timing.get('final_apl_state_u32_bits') == terminal['apl_state'].view(np.uint32).tolist(), 'Parent APL terminal audit mismatch')
return {'directory': directory, 'summary': summary, 'provenance': provenance, 'overlay': overlay,
'state': state, 'terminal': terminal, 'steps': steps, 'global_tick_offset': offset,
'global_next_tick': offset + steps, 'fresh': fresh,
'ancestry_directories': [directory] if fresh else [directory, *parent['ancestry_directories']]}
def seal_checkpoint(checkpoint_dir):
"""Exclusive file-hash seal, not a brain run or proof of pre-seal integrity.
The returned manifest SHA must be externally pinned into the next case spec.
No existing receipt is overwritten. Fresh state.npz is newly bound here.
"""
directory = Path(checkpoint_dir).resolve(strict=True)
R.require(not (directory / MANIFEST).exists(), 'Checkpoint is already sealed')
before = _tree_hashes(directory)
record = _validate_record(directory, {directory})
R.require(before == _tree_hashes(directory), 'Checkpoint changed while sealing')
value = {'schema': 'phase4an-checkpoint-v1', 'complete': True, 'files_sha256': before,
'segment_steps': record['steps'], 'global_tick_offset': record['global_tick_offset'],
'global_next_tick': record['global_next_tick'], 'counts_scope': 'segment_only',
'state_binding_origin': 'checkpoint seal; fresh runtime receipt did not hash state.npz',
'pre_seal_state_integrity_independently_proven': False}
R._save(directory / MANIFEST, value)
return {'path': str(directory / MANIFEST), 'sha256': R.digest(directory / MANIFEST), **value}
def load_checkpoint(checkpoint_dir, expected_sha, *, _seen=None):
directory = Path(checkpoint_dir).resolve(strict=True)
R.require(R._sha(expected_sha) and R.digest(directory / MANIFEST) == expected_sha, 'Parent checkpoint manifest differs from external pin')
seen = set() if _seen is None else set(_seen)
R.require(directory not in seen, 'Checkpoint ancestry cycle')
seen.add(directory)
manifest = R._read_json(directory / MANIFEST)
R.require(manifest.get('schema') == 'phase4an-checkpoint-v1' and manifest.get('complete') is True and
manifest.get('counts_scope') == 'segment_only' and manifest.get('files_sha256') == _tree_hashes(directory),
'Checkpoint file inventory/hash mismatch')
record = _validate_record(directory, seen)
R.require(manifest.get('segment_steps') == record['steps'] and manifest.get('global_tick_offset') == record['global_tick_offset'] and
manifest.get('global_next_tick') == record['global_next_tick'], 'Checkpoint clock metadata mismatch')
R.require(manifest['files_sha256'] == _tree_hashes(directory) and R.digest(directory / MANIFEST) == expected_sha,
'Checkpoint changed while loading')
record.update(manifest=manifest, manifest_sha256=expected_sha)
return record
def derive_continued(source_bytes, mode):
"""One explicit arrival-admission edit plus one pre-loop restore hook."""
tree, inherited = R.derive_ast(source_bytes, mode)
original = copy.deepcopy(tree)
function = tree.body[-1]
body = next(node for node in function.body if isinstance(node, ast.Try)).body
candidates = [i for i, node in enumerate(body) if isinstance(node, ast.Assign) and
len(node.targets) == 1 and isinstance(node.targets[0], ast.Name) and node.targets[0].id == 'apl_ring']
R.require(len(candidates) == 1, 'Expected one APL ring initialization')
body.insert(candidates[0] + 1, ast.parse(RESTORE).body[0])
clocks = [node for node in ast.walk(function) if isinstance(node, ast.If) and ast.unparse(node.test) == CLOCK_OLD]
R.require(len(clocks) == 1, 'Expected exact original local arrival-admission condition')
clocks[0].test = ast.parse(CLOCK_NEW, mode='eval').body
tree = ast.fix_missing_locations(tree)
# Independently reverse exactly the two edits, then require the ENTIRE AST.
rollback = copy.deepcopy(tree)
reverse_body = next(node for node in rollback.body[-1].body if isinstance(node, ast.Try)).body
restores = [i for i, node in enumerate(reverse_body) if ast.unparse(node) == RESTORE]
reverse_clocks = [node for node in ast.walk(rollback) if isinstance(node, ast.If) and ast.unparse(node.test) == CLOCK_NEW]
R.require(len(restores) == len(reverse_clocks) == 1, 'Ambiguous continuation edit rollback')
del reverse_body[restores[0]]
reverse_clocks[0].test = ast.parse(CLOCK_OLD, mode='eval').body
R.require(R._dump(rollback) == R._dump(original), 'Additional unlisted continuation AST edit')
audit = {'base': inherited, 'restore_hook': RESTORE, 'clock_admission_old': CLOCK_OLD,
'clock_admission_new': CLOCK_NEW, 'full_AST_rollback_identical': True,
'kernels_sorting_and_local_ring_math_unchanged': True,
'derived_module_ast_sha256': hashlib.sha256(R._dump(tree).encode()).hexdigest()}
return tree, audit
def audit_carried_sequence(generated, applied, pending, initial_pending):
"""Bitwise FIFO accounting with a carried queue, never a new brain step."""
for name, value in (('generated', generated), ('applied', applied), ('pending', pending), ('initial', initial_pending)):
R.require(isinstance(value, np.ndarray) and value.dtype == np.float32 and value.ndim == 2 and
np.isfinite(value).all() and np.all(value >= 0), 'Invalid continued APL trace: ' + name)
R.require(generated.shape == applied.shape and pending.shape == initial_pending.shape == (18, generated.shape[1]),
'Continued APL sequence shapes differ')
steps = len(generated)
for tick in range(steps + 18):
actual = applied[tick] if tick < steps else pending[tick - steps]
expected = initial_pending[tick] if tick < 18 else generated[tick - 18]
R.require(np.array_equal(actual.view(np.uint32), expected.view(np.uint32)),
'Carried FIFO bitwise accounting failed at local application tick ' + str(tick))
return {'pass_all': True, 'apl_delay_steps': 18, 'steps': steps, 'sample_bits_preserved': True,
'initial_queue_from_parent': True, 'initial_queue_raw_sha256': R._raw(initial_pending),
'full_sequence_bitwise_audit_performed': True, 'brain_tail_advanced': False}
class ContinuedRuntime(R._Runtime):
def __init__(self, *args, parent, **kwargs):
super().__init__(*args, **kwargs)
self.parent = parent
self.global_offset = parent['global_next_tick']
self.restorations = 0
def install(self, *args, **kwargs):
super().install(*args, **kwargs)
R.require(self.effective_sha == self.parent['summary']['effective_weights_raw_sha256'], 'Retention changed effective weights')
def restore(self, v, g, ref, counts, ring, apl_ring, apl, upload):
R.require(self.installations == 1 and self.restorations == 0, 'Restore must occur once after fixed-weight installation')
R.require(not bool(counts.any().item()), 'Segment counts must start at zero')
state, terminal = self.parent['state'], self.parent['terminal']
for target, value, dtype in ((v, state['v'], self.torch.float64), (g, state['g'], self.torch.float64),
(ref, state['ref'], self.torch.int32),
(ring, rotate_spike_ring(terminal['spike_ring'], int(terminal['next_tick'])), self.torch.uint8),
(apl_ring, terminal['apl_pending'], self.torch.float32),
(apl.state, terminal['apl_state'], self.torch.float32)):
target.copy_(upload(value, dtype))
self.restorations += 1
def finish(self, *args, **kwargs):
R.require(self.restorations == 1, 'Continuation was not restored')
return super().finish(*args, **kwargs)
def audit_sequence(self, generated, applied, pending, delay):
R.require(delay == 18, 'Continuation changed D18')
return audit_carried_sequence(generated, applied, pending, self.parent['terminal']['apl_pending'])
def _timing_metadata(self, value):
result = dict(value)
next_tick, steps = self.global_offset + self.spec['steps'], self.spec['steps']
result.pop('initial_padding_positive_zero', None)
result.update(initial_queue_from_parent=True, initial_queue_raw_sha256=R._raw(self.parent['terminal']['apl_pending']),
global_tick_offset=self.global_offset, global_next_tick=next_tick, tick_coordinate='global',
segment_steps=steps, segment_generated_rows_applied=max(steps - 18, 0),
applied_parent_pending_rows=min(steps, 18),
applied_initial_zero_rows=min(steps, max(18 - self.global_offset, 0)),
applied_parent_generated_rows=min(steps, 18) - min(steps, max(18 - self.global_offset, 0)),
pending_segment_generated_rows=min(steps, 18),
pending_parent_generated_rows=max(18 - steps, 0) - max(18 - next_tick, 0),
pending_generated_count=min(next_tick, 18), pending_initial_zero_prefix=max(18 - next_tick, 0),
pending_application_tick_start=next_tick, pending_application_tick_stop_exclusive=next_tick + 18,
pending_generated_tick_start=max(0, next_tick - 18), pending_generated_tick_stop_exclusive=next_tick,
brain_tail_advanced_scope='no extra brain ticks beyond this explicit segment')
if self.mode == 'light':
result.update(generated_trace=None, applied_trace=None, full_sequence_bitwise_audit_performed=False,
scope='Original D18 ring math retained with restored queue; histories not recorded')
return result
def timing(self, steps, pending):
return self._timing_metadata(super().timing(steps, pending))
def summary(self, value):
result = super().summary(value)
result.update(checkpoint_continuation_used=True, global_tick_offset=self.global_offset,
global_next_tick=self.global_offset + self.spec['steps'], event_tick_scope='segment_local',
counts_scope='segment_only_not_lifetime', parent_checkpoint_manifest_sha256=self.parent['manifest_sha256'])
result.update(pending_generated_count=min(result['global_next_tick'], 18),
pending_initial_zero_prefix=max(18 - result['global_next_tick'], 0))
if 'apl_timing' in result:
result['apl_timing'] = self._timing_metadata(result['apl_timing'])
return result
def write_json(self, path, value):
if Path(path).name == 'apl-timing-audit.json':
value = self._timing_metadata(value)
return super().write_json(path, value)
def compile_continued(aj, runtime, mode):
tree, audit = derive_continued(Path(aj.__file__).read_bytes(), mode)
namespace = dict(aj.__dict__)
namespace.update(__name__=__name__ + '.derived_' + mode, __file__=str(Path(__file__).resolve()),
_runtime=runtime, _continuation=runtime, write_json=runtime.write_json,
audit_sequence=runtime.audit_sequence)
audit['diagnostic_namespace_binding'] = 'audit_sequence checks parent_pending + generated, not an assumed zero-initial queue'
exec(compile(tree, str(Path(__file__).resolve()) + '::continued-AJ-AST-' + mode, 'exec'), namespace)
return namespace['run_gpu'], audit
def _parent_spec(parent):
expected = parent['summary']['input_sha256']['spec.json']
paths = [Path(path) for path, sha in parent['provenance']['input_sha256'].items()
if Path(path).name == 'spec.json' and sha == expected]
R.require(len(paths) == 1, 'Ambiguous original parent case spec')
return R._read_json(paths[0])
def run_continued(case_dir, out_dir, build_root, overlay_file, overlay_manifest, checkpoint_dir, *, mode='light'):
R.require(sys.dont_write_bytecode, 'Use python -B; frozen sources must not receive bytecode writes')
R.require(mode in ('light', 'reference'), 'Invalid diagnostic mode')
case_dir, out_dir, build_root = R.execution_paths(case_dir, out_dir, build_root, (checkpoint_dir,))
spec = R._read_json(case_dir / 'spec.json')
parent_sha = spec.get('parent_checkpoint_manifest_sha256')
fresh_admission = dict(spec)
fresh_admission.pop('parent_checkpoint_manifest_sha256', None)
R.validate_spec(fresh_admission, case_dir)
parent = load_checkpoint(checkpoint_dir, parent_sha)
# A continuation may never create output or a build cache inside ANY
# immutable ancestor, not only its immediate checkpoint parent. This check
# precedes overlay imports, Torch imports, compilation and output creation.
case_dir, out_dir, build_root = R.execution_paths(case_dir, out_dir, build_root, parent['ancestry_directories'])
if 'global_tick_offset' in spec:
R.require(type(spec['global_tick_offset']) is int and spec['global_tick_offset'] == parent['global_next_tick'],
'Declared global offset disagrees with checkpoint chain')
R.require(parent['global_next_tick'] + spec['steps'] <= np.iinfo(np.int64).max, 'Absolute tick overflow')
helper, plan, meta, inputs = R.load_overlay(overlay_file, overlay_manifest)
R.require(R.digest(overlay_file) == R.digest(parent['directory'] / 'overlay-input.npz') and
R.digest(overlay_manifest) == R.digest(parent['directory'] / 'overlay-manifest.json'), 'Retention overlay snapshot/manifest changed')
previous = _parent_spec(parent)
for key in ('dt_ms', 'model', 'apl', 'apl_delay_steps'):
R.require(spec[key] == previous[key], 'Continuation changes frozen dynamics: ' + key)
for key in ('graph_dir', 'apl_bundle_dir'):
R.require(Path(spec[key]).resolve(strict=True) == Path(previous[key]).resolve(strict=True), 'Continuation changes input directory: ' + key)
R.require(R.digest(spec['coefficients_path']) == parent['summary']['input_sha256']['coefficients.json'], 'Continuation coefficients changed')
R.require(R.digest(Path(spec['graph_dir']) / 'weights.npy') == parent['summary']['input_sha256']['base_weights.npy'], 'Continuation base weights changed')
aj_dir = Path(meta['aj_model_dir']).resolve(strict=True)
protected = (aj_dir.parent, Path(meta['am_overlay_path']).resolve().parent, spec['graph_dir'], spec['apl_bundle_dir'],
*parent['ancestry_directories'],
*(Path(path).resolve(strict=True).parent for path in
(overlay_file, overlay_manifest, meta['mask_path'], meta['base_candidate_path'], spec['coefficients_path'])))
case_dir, out_dir, build_root = R.execution_paths(case_dir, out_dir, build_root, protected)
for path in (case_dir / 'spec.json', case_dir / 'requests.npz', Path(spec['graph_dir']) / 'weights.npy',
Path(spec['coefficients_path']), Path(checkpoint_dir).resolve() / MANIFEST, Path(R.__file__).resolve()):
inputs[str(path)] = R.digest(path)
source_sha = R.digest(__file__)
R._load('timing_queue', aj_dir / 'timing_queue.py', R.PINS['apl-timing-lab-v1/timing_queue.py'])
aj = R._load('phase4an_continued_sealed_aj_gpu', aj_dir / 'gpu_timing.py', R.AJ_SHA)
runtime = ContinuedRuntime(mode, spec, helper, plan, meta, inputs, aj.torch, overlay_file, overlay_manifest, parent=parent)
function, ast_audit = compile_continued(aj, runtime, mode)
try:
inherited = function(case_dir, out_dir, build_root)
R.require(runtime.finished and runtime.restorations == 1 and runtime.installations == 1 and inherited['complete'], 'Incomplete continuation')
R.require(inherited['effective_weights_raw_sha256'] == runtime.effective_sha, 'Effective weight identity changed')
R.require(source_sha == R.digest(__file__) and all(R.digest(path) == sha for path, sha in inputs.items()), 'Source/input changed during continuation')
load_checkpoint(checkpoint_dir, parent_sha)
summary = R._read_json(out_dir / 'summary.json')
R.require(summary == runtime.summary(inherited), 'Stored continuation summary mismatch')
provenance = {'schema': 'phase4an-continuation-runtime-v1', 'complete': True, 'mode': mode,
'source_file': str(Path(__file__).resolve()), 'source_sha256': source_sha, 'fresh_runtime_sha256': LIGHT_SHA,
'ast': ast_audit, 'input_sha256': inputs, 'old_globals_patched': False, 'weights_updated_during_run': False,
'learning_pass_claimed': False, 'global_tick_offset': runtime.global_offset,
'global_next_tick': runtime.global_offset + spec['steps'],
'parent_checkpoint_manifest_sha256': parent_sha}
for key, name in (('summary_sha256', 'summary.json'), ('finite_audit_sha256', 'finite-audit.json'),
('terminal_queues_sha256', 'terminal-queues.npz'), ('overlay_audit_sha256', 'overlay-audit.json'),
('state_sha256', 'state.npz')):
provenance[key] = R.digest(out_dir / name)
R._save(out_dir / 'runtime-provenance.json', provenance)
receipt = {'schema': 'phase4an-continuation-receipt-v1', 'complete': True,
'parent_checkpoint_dir': str(parent['directory']), 'parent_checkpoint_manifest_sha256': parent_sha,
'global_tick_offset': runtime.global_offset, 'global_next_tick': runtime.global_offset + spec['steps'],
'segment_steps': spec['steps'], 'counts_scope': 'segment_only_not_lifetime',
'payload_sha256': _tree_hashes(out_dir), 'weights_updated_during_run': False,
'clock_admission_edit': CLOCK_NEW, 'checkpoint_continuation_verified_by_GPU_comparison': False}
R._save(out_dir / RECEIPT, receipt)
sealed = seal_checkpoint(out_dir)
return dict(summary, checkpoint_manifest_sha256=sealed['sha256'])
except Exception as error:
if out_dir.is_dir() and not (out_dir / 'continuation-error.json').exists():
R._save(out_dir / 'continuation-error.json', {'complete': False, 'type': type(error).__name__,
'message': str(error), 'learning_update_forbidden': True})
raise
@@ -0,0 +1,559 @@
"""Engineering-only, fixed-weight trial execution derived from frozen AJ AST.
Public APIs run_light(case_dir,out_dir,build_root,overlay_file,overlay_manifest)
and run_reference(...) have identical arguments. Neither trains nor carries a
previous trial's neural state. The reference retains AJ's eager diagnostics;
the light function removes only an explicit diagnostic whitelist. Both accept
an explicit spec.max_steps instead of AJ's 1000-step admission bound.
The parent protocol owns conditioning, cue/tail schedules, frozen input/mask
authority and success criteria. A completed trial is not a learning PASS.
Before any scientific execution, the coordinator must verify externally fixed
hashes for the entire graph, APL bundle, coefficients/model, and preregistered
case spec (including max_steps). Inherited AJ consistency checks and this
adapter's input hashes are not a replacement for those external authorities.
"""
from __future__ import annotations
import ast
import copy
import hashlib
import importlib.util
import json
from pathlib import Path
import re
import sys
import numpy as np
AJ_SHA = 'af1975ba5e5e700208c0a598d3cbe648a3cf10885c6231239b6e4b28f76d173b'
AM_OVERLAY_SHA = '7f891d0ba79b393c96e75fd2dee2eddc51e57d8213eaac2779703bc2c18712b4'
MASK_SHA = '8b0fb6eb2e6c31da4d17dc20d890aa3f367e514f05642de5bad6b42a1611c05b'
V2_SHA = 'abb4574da85f00bd9ef61024c65993ef81d42651ff8d8d63c15dd69c6222471e'
PINS = {
'apl-timing-lab-v1/gpu_timing.py': AJ_SHA,
'apl-timing-lab-v1/timing_queue.py': '54df6d4818e2a0c26e96fb0aaf5d9b71cd964981706da937562d48af79aca14e',
'author-apl-lab-v2/gpu_apl.py': '69f5d893f47dafff4ef7c3186c3df0b9014666b32c8f4e943b57016bec92b18a',
'author-apl-lab-v2/legacy_apl_worker.py': '21ceec65fe42581dc55872010e0025d660e80358905f1cf4df1dadf341a3b291',
'author-runtime-lab-v1/gpu_backend.py': '03758d07572f4b44d272059d9308fac534268ad201b57bdd77d69fee95e67278',
'author-runtime-lab-v1/author_ext.cpp': '208a937d17bbb041b9bf6b25b89264ee566830b60609e58491b44583d8541f15',
'author-runtime-lab-v1/author_ext.cu': '669f166e18ce9a3f63fc723db8b0a8b74f7fa4757194b1bd65b4cf6008a7787e',
'author-fullgraph-lab-v1/oracle_full.py': 'd72003d98f0f1624b12126eff50bd361c43d888f4a64d90ba9247a58976f1961',
}
_LOADED = {}
_INSTALL = '_runtime.install(weights, graph, apl_out_mask, upload, out_dir, source_provenance, device)'
_FINISH = '_runtime.finish(ring, apl_ring, apl, steps, out_dir)'
_MATH_TARGETS = frozenset(('indptr', 'dst', 'global_nodes_gpu', 'base_weights', 'weights',
'local_apl', 'apl_out_mask', 'common_kc', 'local_kc', 'full_kc', 'kc_local_gpu', 'kc_full_gpu',
'v', 'g', 'ref', 'counts', 'requests', 'ring', 'apl_delay', 'apl_ring', 'requested_local_gpu',
'spikes', 'due', 'sources', 'degrees', 'offsets', 'edge_count', 'expanded', 'destination',
'order', 'ordered_edges', 'ordered_targets', 'boundary', 'starts', 'targets', 'full_spikes',
'drive', 'apl_state', 'h', 'slot', 'applied_h', 'delta_si'))
def require(ok, message):
if not ok:
raise ValueError(message)
def digest(path):
h = hashlib.sha256()
with Path(path).open('rb') as stream:
for block in iter(lambda: stream.read(8 << 20), b''):
h.update(block)
return h.hexdigest()
def _read_json(path):
def pairs(items):
out = {}
for key, value in items:
require(key not in out, 'Duplicate JSON key: ' + key)
out[key] = value
return out
return json.loads(Path(path).read_text(encoding='utf-8'), object_pairs_hook=pairs,
parse_constant=lambda value: (_ for _ in ()).throw(ValueError('Nonfinite JSON: ' + value)))
def _save(path, value):
with Path(path).open('x', encoding='utf-8') as stream:
json.dump(value, stream, sort_keys=True, indent=2, allow_nan=False)
stream.write('\n')
def _raw(value):
return hashlib.sha256(np.ascontiguousarray(value, dtype=value.dtype.newbyteorder('<')).tobytes()).hexdigest()
def _sha(value):
return isinstance(value, str) and re.fullmatch('[0-9a-f]{64}', value) is not None
def _load(name, path, expected):
path = Path(path).absolute()
raw = path.read_bytes()
require(hashlib.sha256(raw).hexdigest() == expected, 'Unpinned source: ' + str(path))
if name in sys.modules:
module = sys.modules[name]
require(_LOADED.get(name) == (path, expected, module), 'Existing unverified module binding: ' + name)
return module
spec = importlib.util.spec_from_file_location(name, path)
require(spec is not None and spec.loader is not None, 'Missing module loader')
module = importlib.util.module_from_spec(spec)
sys.modules[name] = module
try:
exec(compile(raw, str(path), 'exec'), module.__dict__)
except BaseException:
sys.modules.pop(name, None)
raise
_LOADED[name] = (path, expected, module)
return module
def validate_spec(spec, case_dir):
"""Admission policy only. No implicit early stopping or checkpoint support."""
require(type(spec.get('max_steps')) is int and 0 < spec['max_steps'] <= np.iinfo(np.int32).max,
'An explicit positive int32-range max_steps is mandatory')
require(type(spec.get('steps')) is int and 0 < spec['steps'] <= spec['max_steps'], 'steps exceeds its declared bound')
require(spec.get('dt_ms') == .1 and spec.get('duration_ms') == spec['steps'] * .1 and
type(spec.get('apl_delay_steps')) is int and spec['apl_delay_steps'] == 18,
'Frozen clock/duration/D18 contract changed')
require(spec.get('record_history') is False, 'Only wholebrain fresh-state trials are supported')
require(not (Path(case_dir) / 'arrays.npz').exists(), 'Initial arrays/checkpoint continuation are not supported in this version')
require(not any('checkpoint' in str(key).lower() or str(key) in ('initial_state', 'resume', 'continuation') for key in spec),
'Checkpoint/continuation configuration is unsupported')
require(spec.get('within_trial_weight_updates', False) is False, 'No within-trial plasticity is supported')
return spec
def load_overlay(overlay_file, manifest_file):
"""New manifest policy, frozen AM array validators; no old-global mutation.
phase4an-overlay-v1 requires overlay_name zero/static25/learned,
overlay_sha256, original mask_path/mask_sha256, external positive f32-mV
base_candidate_path/base_candidate_sha256, aj_model_dir,
am_overlay_path/am_overlay_sha256. Learned snapshots additionally require
parent_snapshot_sha256 and eligibility_v2_source_sha256=the frozen 4J pin.
These hashes bind provenance; the coordinator must verify the parent chain.
NPZ contains ONLY edge_id:int64[14232], plastic_delta_mv:float32[14232].
"""
overlay_file, manifest_file = Path(overlay_file).absolute(), Path(manifest_file).absolute()
manifest_sha = digest(manifest_file)
meta = _read_json(manifest_file)
keys = {'schema', 'overlay_name', 'overlay_sha256', 'mask_path', 'mask_sha256',
'base_candidate_path', 'base_candidate_sha256', 'aj_model_dir', 'am_overlay_path', 'am_overlay_sha256'}
require(isinstance(meta, dict) and keys <= meta.keys() and meta['schema'] == 'phase4an-overlay-v1', 'Invalid overlay manifest schema')
require(meta['overlay_name'] in ('zero', 'static25', 'learned'), 'Unknown overlay snapshot type')
learned_keys = {'parent_snapshot_sha256', 'eligibility_v2_source_sha256'}
allowed = keys | (learned_keys if meta['overlay_name'] == 'learned' else set())
require(not (meta.keys() - allowed), 'Unknown or inapplicable overlay manifest fields')
require(meta['mask_sha256'] == MASK_SHA and meta['am_overlay_sha256'] == AM_OVERLAY_SHA,
'Original mask or frozen AM helper authority changed')
require(_sha(meta['overlay_sha256']) and digest(overlay_file) == meta['overlay_sha256'], 'Overlay SHA mismatch')
inputs = {str(overlay_file): meta['overlay_sha256'], str(manifest_file): manifest_sha}
for label in ('mask', 'base_candidate', 'am_overlay'):
path = Path(meta[label + '_path'])
require(path.is_absolute() and _sha(meta[label + '_sha256']) and digest(path) == meta[label + '_sha256'],
'Missing/mismatched anchored input: ' + label)
inputs[str(path)] = meta[label + '_sha256']
helper = _load('phase4an_sealed_am_overlay', meta['am_overlay_path'], AM_OVERLAY_SHA)
mask = np.load(meta['mask_path'], allow_pickle=False)
base = np.load(meta['base_candidate_path'], allow_pickle=False)
require(mask.dtype == np.int64 and mask.shape == (helper.CANDIDATE_COUNT,), 'Original complete 14,232 mask is required')
with np.load(overlay_file, allow_pickle=False) as archive:
require(len(archive.files) == 2 and set(archive.files) == {'edge_id', 'plastic_delta_mv'}, 'No embedded base or additional overlay arrays allowed')
plan = helper.validate_arrays(archive['edge_id'], archive['plastic_delta_mv'], mask, base)
delta = plan['plastic_delta_mv']
if meta['overlay_name'] == 'zero':
require(not np.any(delta != 0), 'Zero snapshot contains a nonzero delta')
elif meta['overlay_name'] == 'static25':
require(helper.bit_equal(delta, np.multiply(np.float32(-.25), base)), 'static25 must equal the original f32 floor')
else:
require(_sha(meta.get('parent_snapshot_sha256')) and meta.get('eligibility_v2_source_sha256') == V2_SHA,
'Learned snapshot needs parent hash and explicit frozen eligibility-v2 authority')
aj_dir = Path(meta['aj_model_dir'])
require(aj_dir.is_absolute() and aj_dir.name == 'apl-timing-lab-v1', 'AJ must use its real original directory')
for relative, expected in PINS.items():
path = aj_dir.parent / relative
require(digest(path) == expected, 'Sealed dependency changed: ' + relative)
inputs[str(path)] = expected
require(all(digest(path) == expected for path, expected in inputs.items()), 'Input changed while loading')
return helper, plan, meta, inputs
def _dump(node):
return ast.dump(node, annotate_fields=True, include_attributes=False)
def _target_roots(node):
if isinstance(node, ast.Name):
return {node.id}
if isinstance(node, (ast.Tuple, ast.List)):
return set().union(*(_target_roots(value) for value in node.elts))
if isinstance(node, (ast.Subscript, ast.Attribute)):
return _target_roots(node.value)
return set()
def math_projection(function):
"""Ordered/control-structured frozen math statements, excluding diagnostics.
Whitelist explicitly covers state/weight initialization, request mutation,
expansion/stable grouping, original CUDA calls, LegacyAPL and ring/g write.
Tests also require the entire reference loop AST to equal the original.
"""
def block(statements):
result = []
for node in statements:
if isinstance(node, ast.FunctionDef) and node.name == 'upload':
result.append(copy.deepcopy(node))
elif isinstance(node, ast.Assign) and set().union(*(_target_roots(t) for t in node.targets)) & _MATH_TARGETS:
result.append(copy.deepcopy(node))
elif isinstance(node, ast.Expr) and isinstance(node.value, ast.Call):
text = ast.unparse(node.value.func)
if text.startswith('ext.') or text in ('requests.zero_', 'apl.spikes.zero_', 'apl_ring[slot].copy_'):
result.append(copy.deepcopy(node))
elif isinstance(node, (ast.With, ast.Try)):
result.extend(block(node.body))
elif isinstance(node, (ast.If, ast.For)):
body, other = block(node.body), block(node.orelse)
if body or other:
keep = copy.deepcopy(node)
keep.body, keep.orelse = body, other
result.append(keep)
return result
return _dump(ast.Module(body=block(function.body), type_ignores=[]))
def derive_ast(source_bytes, mode):
"""Derive only from exact AJ bytes; return ASTs plus an explicit edit receipt."""
require(mode in ('light', 'reference'), 'Unknown diagnostic mode')
require(hashlib.sha256(source_bytes).hexdigest() == AJ_SHA, 'AJ source pin mismatch')
source = ast.parse(source_bytes)
original = next(node for node in source.body if isinstance(node, ast.FunctionDef) and node.name == 'run_gpu')
loader = copy.deepcopy(next(node for node in source.body if isinstance(node, ast.FunctionDef) and node.name == 'load_inputs'))
bounds = [node for node in ast.walk(loader) if isinstance(node, ast.Compare) and
ast.unparse(node) == '0 < steps <= 1000']
require(len(bounds) == 1, 'Expected one explicit step-admission bound')
bounds[0].comparators[-1] = ast.parse("spec['max_steps']", mode='eval').body
function = copy.deepcopy(original)
body = next(node for node in function.body if isinstance(node, ast.Try)).body
anchor = next(i for i, node in enumerate(body) if ast.unparse(node) == 'effective_hash = hashlib.sha256()')
require(ast.unparse(body[anchor - 1]) == 'if bool(np.any(apl_out_mask)):\n weights[upload(apl_out_mask, torch.bool)] = 0.0',
'Installation no longer follows APL fast-edge zeroing')
body.insert(anchor, ast.parse(_INSTALL).body[0])
sync = [i for i, node in enumerate(body) if ast.unparse(node) == 'torch.cuda.synchronize(device)']
require(len(sync) == 1, 'Expected one post-loop synchronization')
body.insert(sync[0] + 1, ast.parse(_FINISH).body[0])
removed, replacements = [], []
class ReportingOnly(ast.NodeTransformer):
def note(self, node):
removed.append(_dump(node))
return None
def visit_Assign(self, node):
names = set().union(*(_target_roots(t) for t in node.targets))
if names & {'expanded_global', 'accepted_cpu', 'queue', 'apl_cpu', 'actual', 'finite', 'row'}:
return self.note(node)
if names == {'maps'} and isinstance(node.targets[0], ast.Subscript):
return self.note(node)
if names == {'timing'}:
require(ast.unparse(node.value.func) == 'audit_sequence', 'Unexpected timing assignment')
replacements.append(_dump(node))
return ast.parse('timing = _runtime.timing(steps, pending)').body[0]
if names == {'destination'}:
return [node, ast.parse('_runtime.arrivals(expanded, destination, ref)').body[0]]
if ast.unparse(node) == 'g[kc_local_gpu] = g[kc_local_gpu] - delta_si':
return [node, ast.parse('_runtime.after_tick(tick, v, g, ref, drive, apl_state, h, applied_h)').body[0]]
return node
def visit_AugAssign(self, node):
if _target_roots(node.target) & {'due_total', 'accepted_due_total'}:
return self.note(node)
return node
def visit_For(self, node):
text = ast.unparse(node.iter)
if text in ("(('drive', 2), ('state', 2), ('inhibition', 4064))", 'apl_cpu.items()', 'actual.items()'):
return self.note(node)
return self.generic_visit(node)
def visit_If(self, node):
if ast.unparse(node.test) in ("spec['record_history']", 'not finite'):
return self.note(node)
return self.generic_visit(node)
def visit_Expr(self, node):
if isinstance(node.value, ast.Call) and ast.unparse(node.value.func) in ('queue_output.write', 'hash_output.write'):
return self.note(node)
return node
def visit_With(self, node):
targets = [getattr(item.optional_vars, 'id', None) for item in node.items]
if targets == ['queue_output', 'hash_output', 'request_output']:
replacements.append(_dump(node.items[0])); replacements.append(_dump(node.items[1]))
node.items = [node.items[-1]]
return self.generic_visit(node)
if mode == 'light':
function = ReportingOnly().visit(function)
require(len(removed) == 20 and len(replacements) == 3, 'Reporting whitelist changed; stop instead of broad rewriting')
loops = lambda f: [node for node in ast.walk(f) if isinstance(node, ast.For) and
isinstance(node.target, ast.Name) and node.target.id == 'tick']
require(len(loops(original)) == len(loops(function)) == 1, 'Missing/duplicated trial loop')
if mode == 'reference':
require(_dump(loops(original)[0]) == _dump(loops(function)[0]), 'Reference loop must stay entirely unchanged')
original_math, derived_math = math_projection(original), math_projection(function)
require(original_math == derived_math, 'Ordered mathematical AST changed')
extra_calls = [ast.unparse(node) for node in ast.walk(function) if isinstance(node, ast.Call) and
ast.unparse(node.func).startswith('_runtime.')]
expected_calls = [_INSTALL, _FINISH]
if mode == 'light':
expected_calls += ['_runtime.arrivals(expanded, destination, ref)',
'_runtime.after_tick(tick, v, g, ref, drive, apl_state, h, applied_h)',
'_runtime.timing(steps, pending)']
require(sorted(extra_calls) == sorted(expected_calls), 'Unlisted runtime callback added')
tree = ast.fix_missing_locations(ast.Module(body=[loader, function], type_ignores=[]))
sha = lambda value: hashlib.sha256(value.encode('utf-8')).hexdigest()
receipt = {'mode': mode, 'aj_source_sha256': AJ_SHA, 'math_ast_sha256': sha(original_math),
'derived_math_ast_sha256': sha(derived_math), 'mathematical_ast_identical': True,
'reference_entire_tick_loop_identical': mode == 'reference',
'admission_edit': '0 < steps <= 1000 -> 0 < steps <= spec.max_steps',
'pre_loop_installation': _INSTALL, 'post_loop_terminal_recording': _FINISH,
'added_runtime_callbacks': extra_calls,
'removed_diagnostic_statements': removed, 'replaced_diagnostic_nodes': replacements,
'derived_module_ast_sha256': sha(_dump(tree)), 'old_globals_patched': False}
return tree, receipt
class FiniteObserver:
"""GPU scalar diagnostics only; never writes a neuronal/APL input tensor."""
def __init__(self, torch, device):
self.torch = torch
self.ok = torch.ones((), device=device, dtype=torch.bool)
self.first = torch.full((), -1, device=device, dtype=torch.int64)
self.accepted = torch.zeros((), device=device, dtype=torch.int64)
self.due = 0
self.ticks = 0
def arrivals(self, expanded, destination, ref):
self.due += expanded.numel()
self.accepted.add_((ref[destination.long()] == 0).to(self.torch.int64).sum())
def observe(self, tick, *values):
require(tick == self.ticks, 'Finite observer tick coverage changed')
finite = self.torch.ones_like(self.ok)
for value in values:
finite.logical_and_(self.torch.isfinite(value).all())
self.first = self.torch.where((self.first < 0) & (~finite), self.torch.full_like(self.first, tick), self.first)
self.ok.logical_and_(finite)
self.ticks += 1
def result(self):
return {'all_finite': bool(self.ok.item()), 'first_failure_tick': int(self.first.item()),
'finite_observed_ticks': self.ticks, 'arriving_edges': self.due,
'accepted_arriving_edges': int(self.accepted.item())}
def _copy_input(source, target, expected):
with Path(source).open('rb') as src, Path(target).open('xb') as dst:
for block in iter(lambda: src.read(8 << 20), b''):
dst.write(block)
require(digest(target) == expected, 'Copied input differs')
class _Runtime:
def __init__(self, mode, spec, helper, plan, meta, inputs, torch, overlay_file, manifest_file):
self.mode, self.spec, self.helper, self.plan, self.meta, self.inputs = mode, spec, helper, plan, meta, inputs
self.torch, self.overlay_file, self.manifest_file = torch, Path(overlay_file).absolute(), Path(manifest_file).absolute()
self.observer = None
self.installations = 0
self.finished = False
self.result = None
def install(self, weights, graph, apl_mask, upload, out_dir, inherited, device):
require(self.installations == 0, 'Repeated installation is forbidden')
require(len(graph['nodes']) == 166700 and len(graph['edge_id']) == self.helper.EDGE_COUNT,
'Only the intact full graph is permitted')
require(int(apl_mask.sum()) == 5845, 'Original fast APL mask changed')
mapping = self.helper.map_installation(graph['weights'], graph['edge_id'], apl_mask, self.plan)
require(mapping['missing_candidate_count'] == 0, 'All original candidates must be present')
rows = mapping['rows']
if len(rows):
index = upload(rows, self.torch.int64)
nonzero = self.plan['plastic_delta_mv'] != np.float32(0)
change = upload(self.plan['plastic_delta_mv'][nonzero], self.torch.float64) * 0.001
weights[index] = weights[index] + change
require(self.helper.bit_equal(weights[index].cpu().numpy(), mapping['expected_si']), 'Independent SI installation mismatch')
raw_base, expected = self.helper.expected_hashes(graph['weights'], apl_mask, mapping)
actual = hashlib.sha256()
for start in range(0, weights.numel(), 1 << 20):
actual.update(weights[start:start + (1 << 20)].cpu().numpy().astype('<f8', copy=False).tobytes())
require(actual.hexdigest() == expected, 'Effective weight bytes differ from declared overlay')
self.effective_sha = expected
self.observer = FiniteObserver(self.torch, device)
self.installations += 1
_copy_input(self.overlay_file, out_dir / 'overlay-input.npz', self.meta['overlay_sha256'])
_copy_input(self.manifest_file, out_dir / 'overlay-manifest.json', self.inputs[str(self.manifest_file)])
_save(out_dir / 'overlay-audit.json', {'schema': 'phase4an-overlay-install-v1', 'complete': True,
'overlay_name': self.meta['overlay_name'], 'candidate_count': len(self.plan['edge_id']),
'nonzero_delta_count': len(rows), 'am_helper_sha256': AM_OVERLAY_SHA, 'mask_sha256': MASK_SHA,
'base_candidate_sha256': self.meta['base_candidate_sha256'], 'delta_raw_sha256': _raw(self.plan['plastic_delta_mv']),
'base_weights_raw_sha256': raw_base, 'effective_weights_raw_sha256': expected,
'delta_dtype': 'float32', 'delta_units': 'mV', 'state_dtype': 'float64', 'bridge': self.helper.BRIDGE,
'zero_rows_untouched': True, 'weights_updated_during_run': False, 'apl_fast_edges_zeroed': int(apl_mask.sum()),
'parent_snapshot_sha256': self.meta.get('parent_snapshot_sha256'),
'eligibility_v2_source_sha256': self.meta.get('eligibility_v2_source_sha256'),
'parent_chain_verified_by_runtime': False, 'inherited_AJ_provenance_only': inherited})
def arrivals(self, expanded, destination, ref):
self.observer.arrivals(expanded, destination, ref)
def after_tick(self, tick, v, g, ref, drive, state, h, applied):
self.observer.observe(tick, v, g, ref, drive, state, h, applied)
def finish(self, ring, apl_ring, apl, steps, out_dir):
require(not self.finished and self.installations == 1, 'Invalid terminal recording sequence')
if self.mode == 'light':
self.result = self.observer.result()
require(self.result['finite_observed_ticks'] == steps, 'Missing finite checks')
else:
# Reaching this point means AJ's original per-tick finite check passed.
self.result = {'all_finite': True, 'first_failure_tick': -1, 'finite_observed_ticks': steps}
_save(out_dir / 'finite-audit.json', dict(self.result, mode=self.mode,
scope='every completed tick; no neuronal or APL tensor mutated by observer'))
raw_ring, raw_apl = ring.cpu().numpy().copy(), apl_ring.cpu().numpy().copy()
head = steps % len(raw_apl)
pending = np.concatenate((raw_apl[head:], raw_apl[:head]), axis=0)
apl_state = apl.state.cpu().numpy().copy()
require(raw_ring.dtype == np.uint8 and raw_ring.shape == (19, 166700) and
apl_state.dtype == pending.dtype == np.float32 and apl_state.shape == (2,) and pending.shape == (18, 4064),
'Terminal queue dtype/shape changed')
with (out_dir / 'terminal-queues.npz').open('xb') as target:
np.savez_compressed(target, spike_ring=raw_ring, apl_state=apl_state, apl_pending=pending,
next_tick=np.asarray(steps, dtype=np.int64))
self.finished = True
require(self.result['all_finite'], 'Nonfinite value observed; trial is invalid and must not produce a learning update')
def timing(self, steps, pending):
return {'apl_delay_steps': 18, 'steps': steps, 'pending_generated_count': min(steps, 18),
'pending_initial_zero_prefix': max(18 - steps, 0), 'brain_tail_advanced': False,
'full_sequence_bitwise_audit_performed': False,
'scope': 'Original D18 ring AST retained; generated/applied histories not recorded'}
def summary(self, value):
result = dict(value)
result.pop('learning', None)
result.update(diagnostic_mode=self.mode, within_trial_weight_updates=False,
outer_protocol=self.spec.get('outer_protocol'), outer_learning_status='not evaluated by this single-trial executor',
learning_pass_claimed=False, checkpoint_continuation_used=False,
max_steps=self.spec['max_steps'], explicit_steps_executed=self.spec['steps'],
entire_neural_trial_executed=True, hidden_early_stop=False,
full_APL_history_recorded=self.mode == 'reference', full_tick_state_hashes_recorded=self.mode == 'reference',
full_queue_hashes_recorded=self.mode == 'reference',
full_sequence_bitwise_audit_performed=self.mode == 'reference',
first_failure_tick=self.result['first_failure_tick'],
pass_all_scope='single fixed-weight trial finite/recording integrity only; no learning or equivalence claim')
if self.mode == 'light':
result.update(arriving_edges=self.result['arriving_edges'], accepted_arriving_edges=self.result['accepted_arriving_edges'])
if 'apl_timing' in result:
result['apl_timing'] = dict(result['apl_timing'], generated_trace=None, applied_trace=None,
full_sequence_bitwise_audit_performed=False)
return result
def write_json(self, path, value):
path = Path(path)
if path.name == 'summary.json':
value = self.summary(value)
elif path.name == 'apl-timing-audit.json' and self.mode == 'light':
value = dict(value)
value.update(generated_trace=None, applied_trace=None, full_sequence_bitwise_audit_performed=False)
_save(path, value)
def compile_runtime(aj_module, runtime, mode):
"""Fresh namespace; __file__ is this real adapter, not a disguised old file."""
tree, audit = derive_ast(Path(aj_module.__file__).read_bytes(), mode)
namespace = dict(aj_module.__dict__)
namespace.update(__name__=__name__ + '.derived_' + mode, __file__=str(Path(__file__).absolute()),
_runtime=runtime, write_json=runtime.write_json)
exec(compile(tree, str(Path(__file__).absolute()) + '::AJ-AST-' + mode, 'exec'), namespace)
audit.update(derived_namespace_file=namespace['__file__'], original_module_file=str(Path(aj_module.__file__).absolute()),
output_metadata_binding='new write_json only labels recording/within-trial scope; no old global mutated')
return namespace['run_gpu'], audit
def execution_paths(case_dir, out_dir, build_root, protected=()):
"""Resolve directory aliases before containment checks, without any writes.
Only data/source/output directories are canonicalized here. This helper
must not be used for a Python executable: resolving a venv entry changes
interpreter semantics. The coordinator must keep input paths stable during
execution; this is not a defense against concurrent filesystem mutation.
"""
case_dir = Path(case_dir).resolve(strict=True)
require(case_dir.is_dir(), 'Input case must be an existing directory')
out_dir, build_root = (Path(path).resolve(strict=False) for path in (out_dir, build_root))
require(not out_dir.exists(), 'Output must be a new directory')
require(not build_root.exists() or build_root.is_dir(), 'Build cache must be a directory')
roots = [case_dir]
for path in protected:
root = Path(path).resolve(strict=True)
require(root.is_dir(), 'Protected input/source root must be a directory')
roots.append(root)
require(not any(out_dir.is_relative_to(root) or build_root.is_relative_to(root) for root in roots),
'Output/build paths cannot write into frozen source or input directories')
return case_dir, out_dir, build_root
def _run(mode, case_dir, out_dir, build_root, overlay_file, overlay_manifest):
require(sys.dont_write_bytecode, 'Use python -B; frozen dependencies must not receive bytecode writes')
case_dir, out_dir, build_root = execution_paths(case_dir, out_dir, build_root)
spec = validate_spec(_read_json(case_dir / 'spec.json'), case_dir)
helper, plan, meta, inputs = load_overlay(overlay_file, overlay_manifest)
own_sha = digest(__file__)
inputs[str(case_dir / 'spec.json')] = digest(case_dir / 'spec.json')
inputs[str(case_dir / 'requests.npz')] = digest(case_dir / 'requests.npz')
inputs[str(Path(spec['graph_dir']) / 'weights.npy')] = digest(Path(spec['graph_dir']) / 'weights.npy')
aj_dir = Path(meta['aj_model_dir']).resolve(strict=True)
protected = (aj_dir.parent, Path(meta['am_overlay_path']).resolve(strict=True).parent,
Path(spec['graph_dir']), Path(spec['apl_bundle_dir']),
*(Path(path).resolve(strict=True).parent for path in
(overlay_file, overlay_manifest, meta['mask_path'], meta['base_candidate_path'], spec['coefficients_path'])))
case_dir, out_dir, build_root = execution_paths(case_dir, out_dir, build_root, protected)
_load('timing_queue', aj_dir / 'timing_queue.py', PINS['apl-timing-lab-v1/timing_queue.py'])
aj = _load('phase4an_sealed_aj_gpu', aj_dir / 'gpu_timing.py', AJ_SHA)
runtime = _Runtime(mode, spec, helper, plan, meta, inputs, aj.torch, overlay_file, overlay_manifest)
function, ast_audit = compile_runtime(aj, runtime, mode)
try:
inherited = function(case_dir, out_dir, build_root)
require(runtime.finished and runtime.installations == 1 and inherited['complete'], 'Incomplete single-trial execution')
require(inherited['effective_weights_raw_sha256'] == runtime.effective_sha, 'Effective weight identity changed during trial')
require(digest(__file__) == own_sha and all(digest(path) == expected for path, expected in inputs.items()),
'Source or input changed during execution')
summary = _read_json(out_dir / 'summary.json')
require(summary == runtime.summary(inherited), 'Stored summary does not bind actual trial')
_save(out_dir / 'runtime-provenance.json', {'schema': 'phase4an-reporting-adapter-v1', 'complete': True,
'source_file': str(Path(__file__).absolute()), 'source_sha256': own_sha, 'mode': mode, 'ast': ast_audit,
'input_sha256': inputs, 'old_globals_patched': False, 'math_changed': False,
'admission_bound_changed_explicitly': True, 'weights_updated_during_run': False,
'learning_pass_claimed': False, 'parent_chain_verification_owned_by_coordinator': True,
'summary_sha256': digest(out_dir / 'summary.json'), 'finite_audit_sha256': digest(out_dir / 'finite-audit.json'),
'terminal_queues_sha256': digest(out_dir / 'terminal-queues.npz'),
'overlay_audit_sha256': digest(out_dir / 'overlay-audit.json')})
return summary
except Exception as error:
if out_dir.is_dir() and not (out_dir / 'runtime-error.json').exists():
_save(out_dir / 'runtime-error.json', {'complete': False, 'type': type(error).__name__, 'message': str(error),
'learning_update_forbidden': True})
raise
def run_light(case_dir, out_dir, build_root, overlay_file, overlay_manifest):
return _run('light', case_dir, out_dir, build_root, overlay_file, overlay_manifest)
def run_reference(case_dir, out_dir, build_root, overlay_file, overlay_manifest):
return _run('reference', case_dir, out_dir, build_root, overlay_file, overlay_manifest)
@@ -0,0 +1,227 @@
"""Synthetic/mocked dispatch + temporary-array tests; no GPU or brain runs."""
import copy
from contextlib import ExitStack
import hashlib
import json
from pathlib import Path
import tempfile
import unittest
from unittest.mock import Mock, patch
import numpy as np
import engineering_analysis as a
import engineering_plan as p
PIN = 'a' * 64
WEIGHT = 'b' * 64
def raw(value):
return hashlib.sha256(value.tobytes()).hexdigest()
def record(row):
n, k, steps, start = 3, 2, row['steps'], row['start']
state = dict(step=np.empty(0, np.int32), neuron_idx=np.empty(0, np.int32), counts=np.zeros(n, np.int32),
v=np.zeros(n, np.float64), g=np.zeros(n, np.float64), ref=np.zeros(n, np.int32))
reference = row['mode'] == 'reference'
summary = dict(complete=True, pass_all=True, all_finite=True, counts_match_sparse_events=True,
steps=steps, completed_ticks=steps, nodes=n, edges=3, requested=0, total_spikes=0, accepted_requests=0,
rejected_requests=0, arriving_edges=0, accepted_arriving_edges=0, apl_fast_edges_zeroed=1, kc_targets_present=k,
precision='float64', apl_precision='float32', apl_delay_steps=18, diagnostic_mode=row['mode'],
effective_weights_raw_sha256=WEIGHT, within_trial_weight_updates=False, learning_pass_claimed=False,
checkpoint_continuation_used=start > 0, global_tick_offset=start, global_next_tick=start + steps,
wall_seconds=1., diagnostic_simulation_seconds=.5, build_or_cache_seconds=.2,
peak_allocated_vram_bytes=1000, peak_reserved_vram_bytes=2000)
return dict(source=row['row_id'], state=state, steps=steps, global_tick_offset=start, summary=summary,
terminal=dict(spike_ring=np.zeros((19, n), np.uint8), apl_state=np.zeros(2, np.float32),
apl_pending=np.zeros((18, k), np.float32), next_tick=np.asarray(steps, np.int64)),
requests=np.empty((0, 5), np.int64),
traces={name: np.zeros((steps, 2), np.float32) for name in a.compare.TRACE_FILES} if reference else {},
state_hashes=[dict(tick=t, v_sha256=raw(state['v']), g_sha256=raw(state['g']), ref_sha256=raw(state['ref'])) for t in range(steps)] if reference else None,
queues=[dict(tick=t, count=0, accepted_count=0, ordered_edge_sha256='c' * 64,
accepted_mask_sha256='c' * 64, joint_sha256='d' * 64) for t in range(steps)] if reference else None)
def fixture():
plan = p.build_plan(); exp = Path(tempfile.gettempdir()).resolve() / 'synthetic-project' / 'experiments/phase4an-engineering-validation'
reg = dict(schema='phase4an-engineering-registration-v1', plan=plan, training=False,
cases={r['row_id']: dict(case_dir=str(exp / 'cases' / r['row_id'])) for r in plan['rows']},
source_sha256={}, input_sha256={}, selected_case=dict(AM_outputs={'zero': 'mock', 'static25': 'mock'}),
overlays={'zero': {}, 'static25': {}})
verified = {r['row_id']: dict(schema='phase4an-engineering-runtime-record-v1', complete=True, training=False,
row=copy.deepcopy(r), registration_sha256=PIN, checkpoint_manifest_sha256=PIN,
runtime_and_immediate_seal_wall_seconds=2., storage={'state.npz': {'file_bytes': 10}, 'terminal-queues.npz': {'file_bytes': 20}})
for r in plan['rows']}
records = {r['row_id']: record(r) for r in plan['rows']}
reader = Mock(); reader.finish.return_value = []; reader.close.return_value = None
context = dict(reader=reader, records=records,
legacy={stage: record(next(r for r in plan['rows'] if r['overlay'] == stage and r['kind'] == 'fresh' and r['mode'] == 'reference')) for stage in p.OVERLAYS},
graph=dict(ptr=np.array([0, 1, 2, 3], np.int64), dst=np.array([1, 2, 0], np.int32), edge_id=np.arange(3, dtype=np.int64)),
requests=dict(tick=np.empty(0, np.int32), neuron_idx=np.empty(0, np.int32)), storage={r['row_id']: 600 for r in plan['rows']})
return reg, verified, context
class Orchestration(unittest.TestCase):
def mocked(self, reg, verified, context, *, fail=None, live=True, queue_exact=True, joint=True):
calls = []
def ordinary(name):
def f(*args):
calls.append(name)
return dict(numerical_exact=name != fail, checks={}, observed_scope={'mock_only': True})
return f
def chain(mono, parts):
calls.append('chain_exact')
reference = mono['summary']['diagnostic_mode'] == 'reference'
return dict(numerical_exact=fail != ('reference_chain' if reference else 'light_chain'),
pass_all=reference and live, nonvacuous_temporal_coverage=reference and live,
status='PASS' if reference else 'INCOMPLETE-COVERAGE',
coverage={'carried_APL_nonzero_applied_values': 1 if live and reference else 0,
'boundaries': [{'pending_source_spike_events': 1, 'carried_APL_nonzero_pending_values': 1 if joint else 0}]} if reference else None)
with ExitStack() as stack:
for name, value in dict(N=3, E=3, K=2, FAST_APL_EDGES=1).items():
stack.enter_context(patch.object(a, name, value))
stack.enter_context(patch.object(a, '_load_context', return_value=context))
stack.enter_context(patch.object(a, '_materialize', return_value=(np.ones(3, np.float64), {'effective_weights_raw_sha256': WEIGHT})))
stack.enter_context(patch.object(a.compare, 'compare_fresh', side_effect=ordinary('fresh')))
stack.enter_context(patch.object(a.scopes, 'compare_slice', side_effect=ordinary('slice')))
stack.enter_context(patch.object(a.scopes, 'compare_checkpoint', side_effect=ordinary('checkpoint')))
stack.enter_context(patch.object(a.scopes, 'compare_parent_carry', side_effect=ordinary('carry')))
stack.enter_context(patch.object(a, '_pair', side_effect=ordinary('pair')))
stack.enter_context(patch.object(a.compare, 'compare_segments', side_effect=chain))
queue_mock = stack.enter_context(patch.object(a.queue_audit, 'audit_queues', return_value=dict(
queue_ref_request_exact=queue_exact, accepted_nonzero_crossing_witness_verified=live and queue_exact)))
result = a.analyze(reg, PIN, verified)
return result, calls, queue_mock
def test_executes_all_descriptors_and_both_independent_queues(self):
reg, verified, ctx = fixture()
report, calls, queued = self.mocked(reg, verified, ctx)
expected = [r['row_id'] + '/' + str(i) for r in reg['plan']['rows'] for i, _ in enumerate(r['expected_comparisons'])]
self.assertEqual([r['descriptor_id'] for r in report['comparisons']], expected)
self.assertEqual(len(calls), len(expected))
self.assertEqual(report['evaluated_comparisons'], 74)
self.assertEqual(queued.call_count, 2)
self.assertEqual(queued.call_args.kwargs['cuts'], [5, 18, 19, 500])
self.assertEqual(report['status'], a.SUCCESS)
self.assertFalse(report['learning_pass_claimed'])
self.assertEqual(len(report['metrics']), 32)
self.assertTrue(all(row['gpu_utilization'] is None for row in report['metrics']))
self.assertTrue(all(r['light_standalone_coverage'] is None for r in report['overlay_reports'].values()))
json.dumps(report, allow_nan=False)
ctx['reader'].finish.assert_called_once()
ctx['reader'].close.assert_called_once()
def test_numerical_failure_still_evaluates_every_descriptor(self):
for failure in ('fresh', 'slice', 'checkpoint', 'carry', 'pair', 'reference_chain', 'light_chain'):
reg, verified, ctx = fixture()
with self.subTest(failure=failure):
report, calls, _ = self.mocked(reg, verified, ctx, fail=failure)
self.assertEqual(len(calls), 74)
self.assertEqual(report['status'], 'FAIL-NUMERICAL')
self.assertFalse(report['pass_all'])
def test_queue_failure_and_absent_live_coverage_are_distinct(self):
for arguments, expected in [({'queue_exact': False}, 'FAIL-NUMERICAL'),
({'live': False}, 'INCOMPLETE-COVERAGE'), ({'joint': False}, 'INCOMPLETE-COVERAGE')]:
reg, verified, ctx = fixture()
report, _, _ = self.mocked(reg, verified, ctx, **arguments)
self.assertEqual(report['status'], expected)
self.assertFalse(report['pass_all'])
def test_receipt_inventory_alias_and_descriptor_removal_fail_closed(self):
for mutation in ('receipt', 'descriptor', 'pin', 'case', 'row_alias'):
reg, verified, ctx = fixture(); identity = reg['plan']['rows'][0]['row_id']
if mutation == 'receipt':
verified.pop(identity)
elif mutation == 'descriptor':
reg['plan']['rows'][0]['expected_comparisons'] = []
elif mutation == 'pin':
verified[identity]['registration_sha256'] = 'f' * 64
elif mutation == 'row_alias':
verified[identity]['row']['steps'] = True
else:
reg['cases'][identity]['case_dir'] += '-another'
with self.subTest(mutation=mutation), self.assertRaises(ValueError):
self.mocked(reg, verified, ctx)
def test_summary_uses_actual_AJ_requested_names_and_checks_total_accounting(self):
for key in ('requested', 'accepted_requests', 'arriving_edges', 'accepted_arriving_edges', 'edges', 'effective_weights_raw_sha256'):
reg, verified, ctx = fixture(); identity = reg['plan']['rows'][0]['row_id']
ctx['records'][identity]['summary'][key] = 'e' * 64 if key.endswith('sha256') else 1
with self.subTest(key=key):
report, _, _ = self.mocked(reg, verified, ctx)
self.assertEqual(report['status'], 'FAIL-NUMERICAL')
self.assertIn(identity, report['failed_run_checks'])
def test_identical_bytes_give_deterministic_report_and_unknown_is_null(self):
reg, verified, ctx = fixture(); identity = reg['plan']['rows'][0]['row_id']
ctx['records'][identity]['summary'].pop('peak_reserved_vram_bytes')
one, _, _ = self.mocked(reg, verified, ctx)
two, _, _ = self.mocked(reg, verified, ctx)
self.assertEqual(a._canonical(one), a._canonical(two))
row = next(r for r in one['metrics'] if r['row_id'] == identity)
self.assertIsNone(row['peak_reserved_vram_bytes'])
def test_AM_reference_full_comparison_is_not_hidden_by_light(self):
reg, _, ctx = fixture()
row = reg['plan']['rows'][0]; descriptor = row['expected_comparisons'][0]
ctx['records'][row['row_id']]['traces']['drive'][10, 0] = np.float32(1)
report = a._dispatch(descriptor, ctx['records'], ctx['legacy'], row['overlay'])
self.assertFalse(report['numerical_exact'])
self.assertIn('pair.trace.drive', report['failed_checks'])
def test_missing_loaded_population_or_reference_inventory_rejected(self):
reg, verified, ctx = fixture(); ctx['records'].pop(next(iter(ctx['records'])))
with self.assertRaises(ValueError):
self.mocked(reg, verified, ctx)
class FileAndWeightInputs(unittest.TestCase):
def test_reader_rehashes_actual_content_and_never_uses_stat_cache(self):
with tempfile.TemporaryDirectory() as folder:
path = Path(folder).resolve() / 'data.bin'; path.write_bytes(b'AAAA')
reader = a._Reader({str(path): a._sha_file(path)})
reader.bind(path); path.write_bytes(b'BBBB')
with self.assertRaises(ValueError):
reader.bind(path)
with self.assertRaises(ValueError):
reader.finish()
def test_original_pinned_overlay_math_signed_zero_and_nonzero_only_bridge(self):
helper = Path(__file__).resolve().parents[1] / 'wholebrain-plastic-interface-v1/overlay.py'
self.assertEqual(a._sha_file(helper), a.AM_OVERLAY_SHA)
with tempfile.TemporaryDirectory() as folder:
folder = Path(folder).resolve()
mask = folder / 'mask.npy'; base = folder / 'base.npy'
np.save(mask, np.array([0], np.int64)); np.save(base, np.array([1.5], np.float32))
original = np.array([.002, -0., -0., .009], np.float64)
graph = dict(weights=original, edge_id=np.arange(4, dtype=np.int64), apl_out_mask=np.array([False, True, False, False]))
before = original.tobytes()
for stage in ('zero', 'static25'):
delta = np.array([-0. if stage == 'zero' else -.375], np.float32)
overlay = folder / (stage + '.npz'); manifest = folder / (stage + '.json')
np.savez_compressed(overlay, edge_id=np.array([0], np.int64), plastic_delta_mv=delta)
meta = dict(schema='phase4an-overlay-v1', overlay_name=stage, am_overlay_path=str(helper), am_overlay_sha256=a.AM_OVERLAY_SHA,
mask_path=str(mask), mask_sha256=a._sha_file(mask), base_candidate_path=str(base), base_candidate_sha256=a._sha_file(base),
overlay_sha256=a._sha_file(overlay))
manifest.write_text(json.dumps(meta), encoding='utf-8')
paths = [mask, base, overlay, manifest, helper]
reader = a._Reader({str(path): a._sha_file(path) for path in paths})
reg = {'overlays': {stage: dict(overlay_file=str(overlay), overlay_manifest=str(manifest))}}
with patch.object(a, 'MASK_SHA', meta['mask_sha256']), patch.object(a, 'CANDIDATES', 1), patch.object(a, 'FAST_APL_EDGES', 1):
actual, info = a._materialize(reader, reg, graph, stage)
expected = original.copy(); expected[1] = 0.
if stage == 'static25':
expected[0] = np.add(original[0], np.multiply(delta[0].astype(np.float64), np.float64(.001)))
self.assertEqual(actual.tobytes(), expected.tobytes())
self.assertEqual(info['effective_weights_raw_sha256'], raw(expected))
self.assertEqual(info['installed_nonzero_count'], 0 if stage == 'zero' else 1)
self.assertEqual(original.tobytes(), before)
reader.finish(); reader.close()
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,246 @@
"""Synthetic artifact comparisons only; no neural equation or GPU execution."""
import copy
import hashlib
import json
from pathlib import Path
import tempfile
import unittest
import numpy as np
import engineering_compare as c
def sha(value):
return hashlib.sha256(np.ascontiguousarray(value).tobytes()).hexdigest()
def synthetic_parts(lengths, *, active=True, traces=True, queues=True):
n, k = 3, 2
total = sum(lengths)
all_ticks = np.array([value for value in (0, 7, 19, 21, 41, 61) if value < total], np.int32) if active else np.empty(0, np.int32)
all_nodes = np.array([0, 1, 2, 0, 2, 1][:len(all_ticks)], np.int32)
ring, apl_ring = np.zeros((19, n), np.uint8), np.zeros((18, k), np.float32)
phase, offset, result = 0, 0, []
for length in lengths:
ring = ring[(np.arange(19) + phase % 19) % 19].copy()
head = phase % 18
apl_ring = np.concatenate((apl_ring[head:], apl_ring[:head]))
event_tick, event_node, requested, hashes, queue_rows = [], [], [], [], []
generated, applied, drives, apl_states = [], [], [], []
for tick in range(length):
absolute = offset + tick
fired = all_nodes[all_ticks == absolute]
ring[tick % 19] = 0; ring[tick % 19, fired] = 1
for neuron in fired:
event_tick.append(tick); event_node.append(neuron)
requested.append([tick, int(neuron), 1, 1, 0])
due = ring[(tick - 18) % 19] if absolute >= 18 else np.zeros(n, np.uint8)
edges = np.array([int(node) * 2 + i for node in np.flatnonzero(due) for i in (0, 1)], np.int64)
accepted = np.ones(len(edges), np.uint8)
queue_rows.append({'tick': tick, 'count': len(edges), 'accepted_count': len(edges),
'ordered_edge_sha256': sha(edges), 'accepted_mask_sha256': sha(accepted),
'joint_sha256': hashlib.sha256(str(tick).encode() + edges.tobytes() + accepted.tobytes()).hexdigest()})
v = np.array([-.05, -.04, -.03], np.float64)
g = np.array([-0., float(absolute), -.001], np.float64)
ref = np.zeros(n, np.int32)
hashes.append({'tick': tick, 'v_sha256': sha(v), 'g_sha256': sha(g), 'ref_sha256': sha(ref)})
h = np.full(k, np.float32(absolute + 1 if active else 0), np.float32)
application = apl_ring[tick % 18].copy(); apl_ring[tick % 18] = h
generated.append(h); applied.append(application)
drives.append(np.full(2, np.float32(absolute if active else 0), np.float32))
apl_states.append(np.full(2, np.float32(absolute + 2 if active else 0), np.float32))
local_ticks, nodes = np.array(event_tick, np.int32), np.array(event_node, np.int32)
state = {'step': local_ticks, 'neuron_idx': nodes, 'counts': np.bincount(nodes, minlength=n).astype(np.int32), 'v': v, 'g': g, 'ref': ref}
head = length % 18
pending = np.concatenate((apl_ring[head:], apl_ring[:head]))
terminal = {'spike_ring': ring.copy(), 'apl_state': apl_states[-1].copy(), 'apl_pending': pending,
'next_tick': np.asarray(length, np.int64)}
summary = {'complete': True, 'pass_all': True, 'all_finite': True, 'steps': length, 'completed_ticks': length,
'nodes': n, 'precision': 'float64', 'apl_precision': 'float32', 'apl_delay_steps': 18,
'checkpoint_continuation_used': offset > 0, 'global_tick_offset': offset, 'global_next_tick': offset + length,
'effective_weights_raw_sha256': 'e' * 64, 'diagnostic_mode': 'reference' if traces else 'light'}
result.append({'source': 'synthetic', 'steps': length, 'global_tick_offset': offset, 'state': state,
'requests': np.asarray(requested, np.int64).reshape(-1, 5), 'terminal': terminal,
'summary': summary, 'terminal_origin': 'synthetic_recorded',
'traces': {'generated': np.asarray(generated), 'applied': np.asarray(applied),
'drive': np.asarray(drives), 'state': np.asarray(apl_states)} if traces else {},
'state_hashes': hashes if traces else None, 'queues': queue_rows if queues else None})
offset += length; phase = length
return result
def write_output(path, run, *, legacy=False):
path.mkdir()
for name, value in (('summary.json', run['summary']), ('apl-timing-audit.json',
{'final_apl_state_u32_bits': run['terminal']['apl_state'].view(np.uint32).tolist()})):
(path / name).write_text(json.dumps(value), encoding='utf-8')
np.savez_compressed(path / 'state.npz', **run['state'])
np.save(path / 'apl-pending-tail.float32.npy', run['terminal']['apl_pending'])
if legacy:
(path / 'adapter-provenance.json').write_text('{}', encoding='utf-8')
else:
np.savez_compressed(path / 'terminal-queues.npz', **run['terminal'])
with (path / 'requests.csv').open('w', encoding='utf-8', newline='') as stream:
import csv
writer = csv.writer(stream); writer.writerow(c.REQUEST_FIELDS); writer.writerows(run['requests'].tolist())
for name, file in c.TRACE_FILES.items():
if name in run['traces']:
np.save(path / file, run['traces'][name])
for name, file in (('state_hashes', 'tick-state-hashes.jsonl'), ('queues', 'queue.jsonl')):
if run[name] is not None:
(path / file).write_text(''.join(json.dumps(row) + '\n' for row in run[name]), encoding='utf-8')
class Comparisons(unittest.TestCase):
def test_fresh_light_reference_exact_without_fabricating_light_traces(self):
reference = synthetic_parts([64])[0]
light = synthetic_parts([64], traces=False, queues=False)[0]
report = c.compare_fresh(light, reference)
self.assertTrue(report['pass_all'])
self.assertFalse(report['reference_comparisons_available']['APL_traces'])
self.assertFalse(report['temporal_validation_complete'])
def test_light_does_not_hide_reference_vs_AM_interior_trace_mismatch(self):
light = synthetic_parts([64], traces=False, queues=False)[0]
reference = synthetic_parts([64])[0]
legacy = copy.deepcopy(reference)
reference['traces']['drive'][10, 0] += np.float32(1.)
report = c.compare_fresh(light, reference, legacy)
self.assertFalse(report['pass_all'])
self.assertIn('reference_vs_AM.trace.drive', report['failed_checks'])
def test_nonzero_origin_monolithic_and_split_require_same_parent(self):
mono = synthetic_parts([20, 44])[1]
parts = synthetic_parts([20, 5, 39])[1:]
with self.assertRaisesRegex(ValueError, 'same starting parent'):
c.compare_segments(mono, parts)
mono['summary']['parent_checkpoint_manifest_sha256'] = 'a' * 64
parts[0]['summary']['parent_checkpoint_manifest_sha256'] = 'a' * 64
report = c.compare_segments(mono, parts)
self.assertTrue(report['pass_all'], report['failed_checks'])
self.assertEqual(report['global_tick_offset'], 20)
parts[0]['summary']['parent_checkpoint_manifest_sha256'] = 'b' * 64
with self.assertRaisesRegex(ValueError, 'same starting parent'):
c.compare_segments(mono, parts)
def test_monolithic_segments_exact_and_cross_boundary_coverage(self):
mono = synthetic_parts([64])[0]
parts = synthetic_parts([5, 12, 1, 1, 18, 19, 8])
report = c.compare_segments(mono, parts)
self.assertTrue(report['pass_all'], report['failed_checks'])
self.assertTrue(report['reference_comparisons_available']['APL_traces'])
self.assertTrue(report['reference_comparisons_available']['state_hashes'])
self.assertGreater(report['coverage']['observed_carried_arriving_edges'], 0)
self.assertGreater(report['coverage']['carried_APL_nonzero_applied_values'], 0)
self.assertFalse(report['queue_joint_hash_compared'])
self.assertNotEqual(mono['queues'][18]['joint_sha256'], parts[3]['queues'][0]['joint_sha256'])
def test_raw_ring_phase_must_be_normalized(self):
mono = synthetic_parts([64])[0]
parts = synthetic_parts([20, 44])
self.assertFalse(c.bit_equal(mono['terminal']['spike_ring'], parts[-1]['terminal']['spike_ring']))
self.assertTrue(c.bit_equal(c.normalized_ring(mono['terminal']), c.normalized_ring(parts[-1]['terminal'])))
self.assertTrue(c.compare_segments(mono, parts)['pass_all'])
def test_signed_zero_final_state_difference_is_not_equal(self):
left = synthetic_parts([64], traces=False, queues=False)[0]
right = copy.deepcopy(left); right['state']['g'][0] = 0.
report = c.compare_fresh(left, right)
self.assertFalse(report['numerical_exact'])
self.assertIn('pair.state.g', report['failed_checks'])
def test_zero_activity_is_not_a_complete_temporal_pass(self):
mono = synthetic_parts([64], active=False)[0]
parts = synthetic_parts([20, 44], active=False)
report = c.compare_segments(mono, parts)
self.assertTrue(report['numerical_exact'])
self.assertFalse(report['pass_all'])
self.assertFalse(report['temporal_validation_complete'])
self.assertEqual(report['status'], 'INCOMPLETE-COVERAGE')
self.assertFalse(c.compare_fresh(mono, copy.deepcopy(mono))['pass_all'])
def test_missing_observed_queue_coverage_is_reported_unknown_not_zero_or_pass(self):
mono = synthetic_parts([64], traces=False, queues=False)[0]
parts = synthetic_parts([20, 44], traces=False, queues=False)
report = c.compare_segments(mono, parts)
self.assertTrue(report['numerical_exact'])
self.assertIsNone(report['coverage']['observed_carried_arriving_edges'])
self.assertFalse(report['temporal_validation_complete'])
def test_request_acceptance_difference_fails_even_if_events_are_equal(self):
left = synthetic_parts([64])[0]; right = copy.deepcopy(left)
right['requests'][0, 3:] = [0, 1]
report = c.compare_fresh(left, right)
self.assertIn('pair.requests_five_fields', report['failed_checks'])
def test_clock_gaps_and_empty_segment_list_rejected(self):
mono = synthetic_parts([64])[0]
with self.assertRaises(ValueError):
c.compare_segments(mono, [])
parts = synthetic_parts([20, 44]); parts[1]['global_tick_offset'] += 1
with self.assertRaisesRegex(ValueError, 'clock gap'):
c.compare_segments(mono, parts)
def test_partial_traces_hash_coverage_and_bad_dtype_rejected(self):
for mutation in ('trace', 'hash', 'dtype', 'duplicate'):
run = synthetic_parts([64])[0]
if mutation == 'trace':
run['traces'].pop('applied')
elif mutation == 'hash':
run['state_hashes'].pop()
elif mutation == 'dtype':
run['state']['v'] = run['state']['v'].astype(np.float32)
else:
run['state']['step'][1] = run['state']['step'][0]
run['state']['neuron_idx'][1] = run['state']['neuron_idx'][0]
with self.subTest(mutation=mutation), self.assertRaises(ValueError):
c.validate_run(run)
def test_per_step_state_hash_and_queue_mask_mismatch_fail(self):
mono = synthetic_parts([64])[0]; parts = synthetic_parts([20, 44])
parts[1]['state_hashes'][1]['g_sha256'] = '0' * 64
parts[1]['queues'][1]['accepted_mask_sha256'] = '1' * 64
report = c.compare_segments(mono, parts)
self.assertFalse(report['pass_all'])
self.assertIn('concatenated_per_tick_state_hashes', report['failed_checks'])
self.assertIn('concatenated_queue_order_and_acceptance', report['failed_checks'])
def test_effective_weight_change_fails(self):
mono = synthetic_parts([64])[0]; parts = synthetic_parts([20, 44])
parts[1]['summary']['effective_weights_raw_sha256'] = 'f' * 64
self.assertFalse(c.compare_segments(mono, parts)['checks']['unchanged_effective_weights'])
class FileSchemas(unittest.TestCase):
def test_actual_five_request_fields_and_AM_derived_ring_are_labeled(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
run = synthetic_parts([64])[0]
write_output(root / 'AN', run)
write_output(root / 'AM', run, legacy=True)
actual, legacy = c.load_run(root / 'AN'), c.load_run(root / 'AM', allow_legacy_am=True)
try:
self.assertIn('derived_from_fresh_events', legacy['terminal_origin'])
self.assertTrue(c.compare_fresh(actual, copy.deepcopy(actual), legacy)['pass_all'])
with self.assertRaisesRegex(ValueError, 'missing'):
c.load_run(root / 'AM')
finally:
c.close_run(actual); c.close_run(legacy)
def test_missing_reference_file_or_wrong_request_header_rejected(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
run = synthetic_parts([64])[0]
write_output(root / 'partial', run)
(root / 'partial' / 'apl-applied.float32.npy').unlink()
with self.assertRaisesRegex(ValueError, 'incomplete'):
c.load_run(root / 'partial')
write_output(root / 'header', run)
(root / 'header' / 'requests.csv').write_text('tick,neuron_idx,accepted\n', encoding='utf-8')
with self.assertRaisesRegex(ValueError, 'five AJ'):
c.load_run(root / 'header')
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,115 @@
import copy
import unittest
import numpy as np
import engineering_inputs as e
def inputs():
return dict(tick=np.array([0, 4, 5, 17, 18, 18, 19, 499, 500, 999], np.int32),
neuron_idx=np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 166699], np.int32))
def spec():
return dict(name='unchanged', kind='fullgraph', steps=1000, dt_ms=.1, duration_ms=100.,
apl_delay_steps=18, record_history=False, training=False,
model={'threshold_mv': -45, 'w_syn_mv_per_anatomical_synapse': .275},
apl={'gain': .01}, budget_root='/project/old', output_budget_bytes=2048,
graph_dir='/project/frozen-graph', seed=1073741825)
class ExactInputs(unittest.TestCase):
def test_exact_boundaries_reconstruct_all_requests(self):
original = inputs()
intervals = [(0, 5), (5, 18), (18, 19), (19, 500), (500, 1000)]
parts = [e.slice_requests(original, 1000, *span) for span in intervals]
report = e.verify_partition(original, 1000, intervals, parts)
self.assertTrue(report['complete'])
self.assertEqual(parts[2]['tick'].tolist(), [0, 0])
self.assertEqual(parts[-1]['tick'].tolist(), [0, 499])
self.assertEqual(report['random_draws'], 0)
for value in parts:
for key in original:
self.assertFalse(np.shares_memory(original[key], value[key]))
def test_empty_intervals_are_explicit_not_lost(self):
original = inputs()
intervals = [(0, 20), (20, 499), (499, 1000)]
parts = [e.slice_requests(original, 1000, *span) for span in intervals]
self.assertEqual(len(parts[1]['tick']), 0)
self.assertTrue(e.verify_partition(original, 1000, intervals, parts)['complete'])
def test_bad_requests_never_silently_fixed(self):
for mut in ('dtype', 'duplicate', 'order', 'bounds', 'shape', 'extra'):
req = inputs()
if mut == 'dtype': req['tick'] = req['tick'].astype(np.int64)
elif mut == 'duplicate': req['tick'][1], req['neuron_idx'][1] = 0, 1
elif mut == 'order': req['tick'][1] = 10
elif mut == 'bounds': req['neuron_idx'][-1] = 166700
elif mut == 'shape': req['tick'] = req['tick'][:-1]
else: req['seed'] = 1
with self.subTest(mut=mut), self.assertRaises(ValueError):
e.slice_requests(req, 1000, 0, 1000)
def test_partition_gap_overlap_changed_input_and_missing_tail_rejected(self):
req = inputs()
spans = [(0, 500), (500, 1000)]
parts = [e.slice_requests(req, 1000, *x) for x in spans]
for bad in ([(0, 500), (501, 1000)], [(0, 500), (499, 1000)], [(0, 500)]):
with self.assertRaises(ValueError):
e.verify_partition(req, 1000, bad, parts[:len(bad)])
parts[0]['neuron_idx'][0] = 123
with self.assertRaisesRegex(ValueError, 'changes actual requests'):
e.verify_partition(req, 1000, spans, parts)
def test_fresh_prefix_and_continued_specs_preserve_original(self):
original = spec()
before = copy.deepcopy(original)
for start, stop, parent in ((0, 1000, None), (0, 18, None), (18, 19, 'a' * 64)):
derived, audit = e.derive_spec(original, start=start, stop=stop, budget_root='/project/new',
output_budget_bytes=8 * 1024**3, parent_sha=parent)
self.assertEqual(original, before)
self.assertEqual(derived['model'], original['model'])
self.assertEqual(derived['apl'], original['apl'])
self.assertIsNot(derived['model'], original['model'])
self.assertEqual(derived['max_steps'], 1000)
self.assertEqual(derived['steps'], stop - start)
self.assertTrue(audit['all_other_fields_unchanged'])
if start: self.assertEqual(derived['global_tick_offset'], start)
def test_invalid_continuation_root_and_duration_rejected(self):
for root in ('/project/old/sub', '/project', 'relative', '/project/../old', '//project/old', '//project/new'):
with self.assertRaises(ValueError):
e.derive_spec(spec(), start=0, stop=1000, budget_root=root, output_budget_bytes=10)
for start, stop, parent in ((1, 20, None), (0, 10, 'a'*64), (True, 10, None), (0, 1001, None)):
with self.assertRaises(ValueError):
e.derive_spec(spec(), start=start, stop=stop, budget_root='/project/new', output_budget_bytes=10, parent_sha=parent)
original = spec(); original['duration_ms'] = 1.
with self.assertRaises(ValueError):
e.derive_spec(original, start=0, stop=10, budget_root='/project/new', output_budget_bytes=10)
def test_original_clock_type_and_no_within_trial_update_match_runtime(self):
for key, value in (('apl_delay_steps', 18.), ('apl_delay_steps', True),
('within_trial_weight_updates', True), ('budget_root', '//project/old')):
original = spec(); original[key] = value
with self.subTest(key=key, value=value), self.assertRaises(ValueError):
e.derive_spec(original, start=0, stop=10, budget_root='/project/new', output_budget_bytes=10)
def test_overlay_is_declaration_translation_not_self_authorized_learned(self):
original = dict(schema='phase4am-overlay-v1', overlay_name='zero', training=False, fixed_for_whole_trial=True,
overlay_sha256='b'*64, mask_path='/project/mask.npy', mask_sha256=e.MASK_SHA,
base_candidate_path='/project/base.npy', base_candidate_sha256='c'*64,
aj_model_dir='/project/apl-timing-lab-v1', old_extra_field='preserved in original only')
before = copy.deepcopy(original)
result = e.adapt_overlay_manifest(original, '/project/overlay.py')
self.assertEqual(original, before)
self.assertEqual(result['schema'], 'phase4an-overlay-v1')
self.assertNotIn('old_extra_field', result)
self.assertEqual(result['overlay_sha256'], original['overlay_sha256'])
for key, val in (('overlay_name', 'learned'), ('mask_sha256', 'd'*64), ('training', True), ('base_candidate_path', 'local')):
bad = dict(original); bad[key] = val
with self.subTest(key=key), self.assertRaises(ValueError):
e.adapt_overlay_manifest(bad, '/project/overlay.py')
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,152 @@
"""Pure inventory/dependency tests; no I/O, GPU, scientific input or runtime."""
import copy
import json
import unittest
import engineering_plan as p
class FixedPlan(unittest.TestCase):
def test_complete_current_inventory_not_future_supplements(self):
plan = p.build_plan()
self.assertEqual(plan['total_calls'], len(plan['rows']))
self.assertEqual(plan['total_calls'], 32)
self.assertEqual(plan['total_ticks'], sum(row['steps'] for row in plan['rows']))
self.assertEqual(plan['total_ticks'], 11074)
self.assertEqual(plan['overlays'], ['zero', 'static25'])
self.assertEqual(plan['chain_lengths'], [5, 13, 1, 481, 500])
self.assertFalse(plan['training'])
self.assertFalse(plan['candidate_generated'])
self.assertEqual(p.validate_plan(json.loads(json.dumps(plan))), plan)
for overlay in plan['overlays']:
rows = [r for r in plan['rows'] if r['overlay'] == overlay]
self.assertEqual(len(rows), 16)
self.assertEqual(sum(r['steps'] for r in rows), 5537)
self.assertEqual([(r['mode'], r['steps']) for r in rows if r['kind'] == 'fresh'],
[('reference', 1000), ('light', 1000), ('light', 1000)])
self.assertEqual([r['stop'] for r in rows if r['kind'] == 'prefix'], [18, 19, 500])
self.assertFalse(any(r['stop'] > 1000 for r in rows))
def test_all_prerequisites_precede_rows_and_fresh_prefixes_precede_chains(self):
rows = p.build_plan()['rows']
self.assertTrue(all(r['kind'] in ('fresh', 'prefix') for r in rows[:12]))
self.assertTrue(all(r['kind'] == 'chain' for r in rows[12:]))
positions = {r['row_id']: i for i, r in enumerate(rows)}
self.assertEqual(len(positions), 32)
for index, row in enumerate(rows):
if row['parent_row_id'] is not None:
self.assertLess(positions[row['parent_row_id']], index)
for check in row['expected_comparisons']:
reference = check['reference_row_id']
if reference is not None:
self.assertLess(positions[reference], index)
for candidate in check['candidate_row_ids']:
self.assertLessEqual(positions[candidate], index)
self.assertLess(check['start'], check['stop'])
def test_each_chain_is_exact_partition_with_local_phase_distinct_from_global(self):
for overlay in p.OVERLAYS:
for mode in p.MODES:
chain = [r for r in p.build_plan()['rows'] if
(r['overlay'], r['mode'], r['kind']) == (overlay, mode, 'chain')]
self.assertEqual([(r['start'], r['stop']) for r in chain],
[(0, 5), (5, 18), (18, 19), (19, 500), (500, 1000)])
self.assertEqual([r['steps'] % 19 for r in chain], [5, 13, 1, 6, 6])
self.assertEqual([r['stop'] % 19 for r in chain], [5, 18, 0, 6, 12])
self.assertIsNone(chain[0]['parent_row_id'])
for earlier, later in zip(chain, chain[1:]):
self.assertEqual(later['parent_row_id'], earlier['row_id'])
self.assertEqual(later['start'], earlier['stop'])
self.assertEqual(chain[2]['start'], 18) # source tick 0 arrives at this child's local 0.
self.assertEqual(chain[2]['steps'], 1)
last = next(c for c in chain[-1]['expected_comparisons'] if c['check'] == 'chain_exact')
self.assertEqual(last['candidate_row_ids'], [r['row_id'] for r in chain])
self.assertEqual((last['start'], last['stop']), (0, 1000))
def test_prefix5_is_reference_first_segment_without_extra_run(self):
rows = p.build_plan()['rows']
self.assertFalse(any(r['kind'] == 'prefix' and r['stop'] == 5 for r in rows))
for overlay in p.OVERLAYS:
first = {r['mode']: r for r in rows if r['kind'] == 'chain' and
r['overlay'] == overlay and r['start'] == 0}
refs = [c for c in first['light']['expected_comparisons'] if c['check'] == 'checkpoint_exact']
self.assertEqual(len(refs), 1)
self.assertEqual(refs[0]['reference_row_id'], first['reference']['row_id'])
self.assertFalse(any(c['reference_row_id'] == first['reference']['row_id']
for c in first['reference']['expected_comparisons']))
def test_fresh_am_and_light_repeat_comparisons_are_explicit(self):
rows = p.build_plan()['rows']
for overlay in p.OVERLAYS:
fresh = [r for r in rows if r['overlay'] == overlay and r['kind'] == 'fresh']
self.assertEqual(len(fresh), 3)
for row in fresh:
am = [c for c in row['expected_comparisons'] if c['check'] == 'am_exact']
self.assertEqual(len(am), 1)
self.assertEqual(am[0]['external_reference'], 'sealed_AM_selected_case_same_overlay_r0')
self.assertIsNone(am[0]['reference_row_id'])
self.assertEqual(fresh[1]['expected_comparisons'][1]['reference_row_id'], fresh[0]['row_id'])
self.assertEqual(fresh[2]['expected_comparisons'][1]['reference_row_id'], fresh[1]['row_id'])
def test_every_result_is_independent_not_shared_mutable_inventory(self):
original = p.build_plan()
checked = p.validate_plan(original)
checked['rows'][0]['expected_comparisons'][0]['candidate_row_ids'].append('mutation')
checked['chain_lengths'][0] = 99
self.assertEqual(original, p.build_plan())
original['rows'][0]['expected_comparisons'][0]['scope'] = 'mutation'
self.assertNotEqual(original, p.build_plan())
def test_exact_types_reject_bool_int_float_and_sequence_conflation(self):
mutations = (
lambda x: x.update(total_calls=32.0),
lambda x: x.update(training=0),
lambda x: x['rows'][0].update(start=False),
lambda x: x['rows'][14].update(steps=True),
lambda x: x['rows'][14].update(steps=1.0),
lambda x: x['rows'][0]['expected_comparisons'][0].update(start=False),
lambda x: x.update(rows=tuple(x['rows'])),
)
for mutate in mutations:
plan = p.build_plan(); mutate(plan)
with self.subTest(mutation=repr(mutate)), self.assertRaisesRegex(ValueError, 'type'):
p.validate_plan(plan)
def test_unknown_fields_missing_overlays_and_reordering_are_rejected(self):
mutations = (
lambda x: x.update(extra=True),
lambda x: x['rows'][0].update(extra='not allowed'),
lambda x: x['rows'][0]['expected_comparisons'][0].update(tolerance=0),
lambda x: x.update(overlays=['zero']),
lambda x: x['rows'].pop(0),
lambda x: x['rows'].append(copy.deepcopy(x['rows'][-1])),
lambda x: x['rows'].__setitem__(slice(0, 2), x['rows'][1::-1]),
lambda x: x['rows'][0].update(overlay='static25'),
lambda x: x['rows'][0].update(mode='light'),
lambda x: x['rows'][0].update(row_id='arbitrary'),
)
for mutate in mutations:
plan = p.build_plan(); mutate(plan)
with self.subTest(mutation=repr(mutate)), self.assertRaises(ValueError):
p.validate_plan(plan)
def test_gaps_wrong_parents_and_dropped_comparisons_never_pass(self):
mutations = (
lambda x: x['rows'][13].update(start=6, steps=12),
lambda x: x['rows'][13].update(stop=17, steps=12),
lambda x: x['rows'][13].update(parent_row_id=None),
lambda x: x['rows'][12].update(parent_row_id=x['rows'][13]['row_id']),
lambda x: x['rows'][16].update(stop=999, steps=499),
lambda x: x['rows'][0].update(expected_comparisons=[]),
lambda x: x['rows'][2]['expected_comparisons'].pop(),
lambda x: x['rows'][16]['expected_comparisons'][-1]['candidate_row_ids'].pop(),
lambda x: x['rows'][16]['expected_comparisons'][-1].update(scope='final_only'),
)
for mutate in mutations:
plan = p.build_plan(); mutate(plan)
with self.subTest(mutation=repr(mutate)), self.assertRaises(ValueError):
p.validate_plan(plan)
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,216 @@
"""Temporary-directory and small authority tests only; never touch real ROOT."""
import copy
import hashlib
import json
from pathlib import Path
import tempfile
import types
import unittest
from unittest.mock import patch
import engineering_prepare as p
LOCAL = Path(__file__).resolve().parent
AI = LOCAL.parent / 'pn-recruitment-audit-v1/publication.py'
def valid_status():
final = dict(status=p.FINAL_STATUS, primary_trials_per_stage=120, repeats_per_stage=24,
zero_overlay_exact=True, static25_repeat_exact=True, old_integrity_pass=True,
frozen_five_reference_pass=True, frozen_eligibility_and_mask_unchanged=True,
base_weights_unchanged=True, training=False, learned_candidate=False)
scientific = dict(status=p.FINAL_STATUS, primary_trials_per_stage=120, technical_repeats_per_stage=24,
zero_overlay_exact=True, static25_repeat_exact=True, training=False, learned_candidate=False)
old = {'root-' + str(i): {'pass': True, 'checked': 1, 'mismatches': [], 'manifest_sha256': 'a'*64, 'checksums_sha256': 'b'*64}
for i in range(28)}
refs = {'pass_all': True, 'trials': [{'trial': name, 'exact': {key: True for key in ('step', 'neuron_idx', 'counts', 'v', 'g')}} for name in p.REFERENCES]}
return final, scientific, old, copy.deepcopy(old), refs
def planned():
return [dict(name='fixed-' + str(i), cohort='seed_validation' if i < 60 else 'novel_to_aj', repeat=r)
for i in range(120) for r in ((0, 1) if i < 24 else (0,))]
class AuthorityTests(unittest.TestCase):
def test_required_final_status_and_all_28_roots_five_references(self):
p.validate_final_status(*valid_status())
for mutation in ('status', 'hidden-learning', 'root', 'reference', 'root-hash'):
args = copy.deepcopy(valid_status())
if mutation == 'status':
args[0]['status'] = 'INCOMPLETE'
elif mutation == 'hidden-learning':
args[1]['learned_candidate'] = True
elif mutation == 'root':
args[3].pop('root-27')
elif mutation == 'reference':
args[4]['trials'][2]['exact']['g'] = False
else:
args[3]['root-1']['manifest_sha256'] = 'c' * 64
with self.subTest(mutation=mutation), self.assertRaises(ValueError):
p.validate_final_status(*args)
def test_first_actual_planned_primary_is_selected_without_dynamic_ranking(self):
rows = planned()
self.assertEqual(p.validate_planned(rows), rows[0])
for altered in (rows[:-1], rows + [rows[0]], [dict(rows[0], name='../escape'), *rows[1:]],
[rows[1], rows[1], *rows[2:]]):
with self.assertRaises(ValueError):
p.validate_planned(altered)
def test_three_external_sha_anchors_have_no_defaults_or_placeholders(self):
for anchors in ((None, 'a'*64, 'b'*64), ('', '', ''), ('x'*64, 'a'*64, 'b'*64)):
with self.assertRaisesRegex(ValueError, 'actual external'):
p._prepare(Path('not-read'), Path('not-read'), Path('not-read'), *anchors)
def test_path_inventory_rejects_duplicates_and_traversal(self):
for name in ('../bad', '/absolute', 'a\\b', 'a/../b', 'C:/bad'):
with self.assertRaises(ValueError):
p._relative(name)
row = {'path': 'good.json', 'bytes': 3, 'sha256': 'a'*64}
with self.assertRaisesRegex(ValueError, 'Duplicate'):
p._index([row, row])
def test_pinned_AI_verifier_rechecks_complete_synthetic_seal(self):
helper = p.load_module('phase4an_test_AI_publication', AI, p.AI_HELPER_SHA)
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()
payload = root / 'small.json'; payload.write_text('{}', encoding='utf-8')
row = {'path': payload.name, 'bytes': payload.stat().st_size, 'sha256': p.digest(payload)}
p.save(root / 'manifest.json', {'root': str(root), 'files': [row]})
(root / 'checksums.sha256').write_text(row['sha256'] + ' small.json\n' + p.digest(root / 'manifest.json') + ' manifest.json\n', encoding='utf-8')
self.assertTrue(helper.verify_seal(root)['pass_all'])
payload.write_text('{"changed":true}', encoding='utf-8')
with self.assertRaises(ValueError):
helper.verify_seal(root)
def test_finalizer_authority_helper_checks_external_anchors_and_both_full_seals(self):
helper = p.load_module('phase4an_test_AI_publication', AI, p.AI_HELPER_SHA)
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()
model, experiment = root / p.AM_MODEL_REL, root / p.AM_EXP_REL
model.mkdir(parents=True); experiment.mkdir(parents=True)
p.save(model / 'payload.json', {'synthetic': True})
p.save(experiment / 'protocol/preregistered.json', {'planned': []})
p.save(experiment / 'audit/final-status.json', {'status': p.FINAL_STATUS})
for tree in (model, experiment):
rows = [dict(path=path.relative_to(tree).as_posix(), bytes=path.stat().st_size, sha256=p.digest(path))
for path in sorted(tree.rglob('*')) if path.is_file()]
p.save(tree / 'manifest.json', {'root': str(tree), 'files': rows})
(tree / 'checksums.sha256').write_text(''.join(row['sha256'] + ' ' + row['path'] + '\n' for row in rows) +
p.digest(tree / 'manifest.json') + ' manifest.json\n', encoding='utf-8')
prereg = p.digest(experiment / 'protocol/preregistered.json')
anchors = dict(am_model_manifest_sha256=p.digest(model / 'manifest.json'),
am_experiment_manifest_sha256=p.digest(experiment / 'manifest.json'), am_final_sha256=p.digest(experiment / 'audit/final-status.json'),
am_preregistration_sha256=prereg, AI_publication_helper_sha256=p.AI_HELPER_SHA)
with patch.object(p, 'ROOT', root), patch.object(p, 'AM_PREREG_SHA', prereg), patch.object(p, 'load_module', return_value=helper):
self.assertEqual(set(p.verify_authority(anchors)), {'model', 'experiment'})
(model / 'payload.json').write_text('changed', encoding='utf-8')
with self.assertRaises(ValueError):
p.verify_authority(anchors)
def test_old_integrity_bridge_restores_import_path_and_rejects_failed_root(self):
import sys
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve(); expected = valid_status()[3]
before_path = list(sys.path)
def good():
sys.path.insert(0, 'synthetic-old-context-search-path')
return copy.deepcopy(expected)
module = types.SimpleNamespace(ROOT=root, integrity=good)
with patch.object(p, 'load_module', return_value=module):
self.assertEqual(p._old_integrity(root, 'a'*64, expected), expected)
self.assertEqual(sys.path, before_path)
bad = copy.deepcopy(expected); bad['root-1']['pass'] = False
module.integrity = lambda: bad
with patch.object(p, 'load_module', return_value=module), self.assertRaisesRegex(ValueError, '28-root'):
p._old_integrity(root, 'a'*64, expected)
self.assertEqual(sys.path, before_path)
def test_all_144_stage_completions_must_match_sealed_inventory(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve(); experiment = root / p.AM_EXP_REL; experiment.mkdir(parents=True)
rows, index, completed = planned(), {}, []
def add(relative, value):
path = experiment / relative; p.save(path, value)
index[relative] = {'path': relative, 'bytes': path.stat().st_size, 'sha256': p.digest(path)}
for row in rows:
prefix = 'gpu/zero/' + row['name'] + '-r' + str(row['repeat']) + '/'
folder = experiment / prefix.rstrip('/')
case = root / p.AK_REL / 'cases' / row['cohort'] / row['name']
add(prefix + 'summary.json', {'case': row['name'], 'complete': True, 'pass_all': True, 'all_finite': True,
'steps': 1000, 'completed_ticks': 1000})
add(prefix + 'comparison.json', {'pass_all': True, 'overlay_name': 'zero', 'case_dir': str(case), 'right': str(folder),
'mode': 'zero', 'left': str(root / p.AK_REL / 'gpu' / row['cohort'] / (row['name'] + '-r' + str(row['repeat']))),
'checks': {'synthetic_gate': True}, 'failed_checks': [], 'full_queue_reconstructed': True})
inventory = [dict(path=k[len(prefix):], bytes=v['bytes'], sha256=v['sha256']) for k, v in sorted(index.items()) if k.startswith(prefix)]
add(prefix + 'output-hashes.json', inventory)
completed.append(dict(name=row['name'], repeat=row['repeat'], output_hashes_sha256=index[prefix+'output-hashes.json']['sha256'],
comparison_sha256=index[prefix+'comparison.json']['sha256']))
receipt = {'stage': 'zero', 'pass_all': True, 'trials': 144, 'primary_trials': 120, 'technical_repeats': 24,
'training': False, 'completed': completed}
add('audit/zero-complete.json', receipt)
self.assertTrue(p.validate_stage(experiment, index, 'zero', rows, root)['pass_all'])
wrong = copy.deepcopy(rows); wrong[0], wrong[1] = wrong[1], wrong[0]
with self.assertRaisesRegex(ValueError, 'order/identity'):
p.validate_stage(experiment, index, 'zero', wrong, root)
# A missing sealed payload entry cannot be hidden by a PASS flag.
altered = dict(index); altered.pop('gpu/zero/fixed-0-r0/summary.json')
with self.assertRaises(ValueError):
p.validate_stage(experiment, altered, 'zero', rows, root)
class PreparationSafetyTests(unittest.TestCase):
def test_exclusive_layout_allows_only_nonexistent_or_empty_experiment(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()
model = root / 'models/male-cns-lif/olfactory-conditioning-lab-v1'; model.mkdir(parents=True)
experiment = root / 'experiments/phase4an-engineering-validation'; experiment.parent.mkdir()
self.assertEqual(p._layout(root, model, experiment)[2], experiment)
experiment.mkdir(); self.assertEqual(p._layout(root, model, experiment)[2], experiment)
(experiment / 'preserve.txt').write_text('partial', encoding='utf-8')
with self.assertRaisesRegex(ValueError, 'populated'):
p._layout(root, model, experiment)
self.assertEqual((experiment / 'preserve.txt').read_text(), 'partial')
def test_required_runtime_workflow_scope_sources_must_all_exist(self):
with tempfile.TemporaryDirectory() as directory:
model = Path(directory).resolve()
for name in p.SOURCE_FILES:
(model / name).write_text('# pure test source\n', encoding='utf-8')
result = p.source_inventory(model)
self.assertEqual(len(result), 21)
self.assertIn(str(model / 'engineering_workflow.py'), result)
self.assertIn(str(model / 'engineering_scopes.py'), result)
self.assertIn(str(model / 'engineering_analysis.py'), result)
(model / 'engineering_workflow.py').unlink()
with self.assertRaisesRegex(ValueError, 'Required engineering source'):
p.source_inventory(model)
def test_byte_budget_reserve_is_not_a_license_to_delete_partials(self):
with tempfile.TemporaryDirectory() as directory:
experiment = Path(directory).resolve() / 'new'
experiment.mkdir(); (experiment / 'partial').write_bytes(b'x')
with patch.object(p.shutil, 'disk_usage', return_value=types.SimpleNamespace(free=30 * 1024**3)):
self.assertEqual(p._budget(experiment)['used_bytes'], 1)
with patch.object(p.shutil, 'disk_usage', return_value=types.SimpleNamespace(free=5 * 1024**3)):
with self.assertRaisesRegex(ValueError, 'Insufficient'):
p._budget(experiment)
self.assertTrue((experiment / 'partial').is_file())
def test_row_mapping_rejects_unplanned_or_escaping_paths(self):
with tempfile.TemporaryDirectory() as directory:
exp = Path(directory).resolve(); case = exp / 'cases/safe-row'; case.mkdir(parents=True)
reg = {'plan': {'rows': [{'row_id': 'safe-row'}]}, 'cases': {'safe-row': {'case_dir': str(case)}}}
with patch.object(p, 'EXP', exp):
self.assertEqual(p.row_path(reg, 'safe-row'), case)
with self.assertRaises(ValueError):
p.row_path(reg, '../outside')
reg['cases']['safe-row']['case_dir'] = str(exp)
with self.assertRaises(ValueError):
p.row_path(reg, 'safe-row')
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,229 @@
"""Small synthetic CSR/record tests only; no brain, GPU or artifact writes."""
import ast
import copy
import hashlib
import json
from pathlib import Path
import struct
import unittest
import numpy as np
import engineering_queue_audit as audit
def digest(array):
return hashlib.sha256(np.ascontiguousarray(array).tobytes()).hexdigest()
def queue_row(tick, edges, accepted):
# Independent simple whole-payload construction, unlike the chunked audit.
edge_bytes = np.asarray(edges, dtype='<i8').tobytes()
mask_bytes = np.asarray(accepted, dtype=np.uint8).tobytes()
return dict(tick=tick, count=len(edges), accepted_count=int(sum(accepted)),
ordered_edge_sha256=hashlib.sha256(edge_bytes).hexdigest(),
accepted_mask_sha256=hashlib.sha256(mask_bytes).hexdigest(),
joint_sha256=hashlib.sha256(b'phase4af-queue-v1\0' + struct.pack('<qq', tick, len(edges)) + edge_bytes + mask_bytes).hexdigest())
def fixture(*, target_fire=True, weights=None, events=None, cuts=(5, 18, 19, 37)):
ptr = np.array([0, 2, 3, 4, 5], np.int64)
dst = np.array([2, 1, 2, 0, 2], np.int32)
edge_id = np.array([10, 11, 20, 30, 40], np.int64)
weights = np.array([.001, 0., -.002, .003, .004], np.float64) if weights is None else weights
total, n = 60, 4
if events is None:
events = [(0, 0), (0, 1), *(([(18, 2)]) if target_fire else []), (22, 0), (23, 1), (44, 0)]
events = sorted(events)
request_events = sorted(set(events + [(1, 0)]))
ticks, nodes = np.array([x[0] for x in events], np.int32), np.array([x[1] for x in events], np.int32)
requests = {'tick': np.array([x[0] for x in request_events], np.int32),
'neuron_idx': np.array([x[1] for x in request_events], np.int32)}
refs, expanded, masks, request_rows = [], [], [], []
last = np.full(n, -10000, np.int64)
for tick in range(total):
for when, neuron in request_events:
if when == tick:
accepted = int(tick - last[neuron] >= 22)
request_rows.append([tick, neuron, 1, accepted, 1 - accepted])
for when, neuron in events:
if when == tick:
last[neuron] = tick
ref = np.maximum(22 - (tick - last), 0).astype(np.int32)
refs.append(ref)
edges = [j for when, neuron in events if when == tick - 18 for j in range(int(ptr[neuron]), int(ptr[neuron + 1]))]
expanded.append(np.asarray(edges, np.int64))
masks.append((ref[dst[edges]] == 0).astype(np.uint8))
def record(start, stop):
selected = (ticks >= start) & (ticks < stop)
ev_t, ev_n = (ticks[selected] - start).copy(), nodes[selected].copy()
rows = np.array([row for row in request_rows if start <= row[0] < stop], np.int64).reshape(-1, 5)
rows[:, 0] -= start
return {'steps': stop - start, 'global_tick_offset': start,
'summary': {'pass_all': False}, # Never treated as authority.
'state': {'step': ev_t, 'neuron_idx': ev_n, 'counts': np.bincount(ev_n, minlength=n).astype(np.int32),
'v': np.zeros(n, np.float64), 'g': np.zeros(n, np.float64), 'ref': refs[stop - 1].copy()},
'requests': rows,
'queues': [queue_row(t - start, edge_id[expanded[t]], masks[t]) for t in range(start, stop)],
'state_hashes': [{'tick': t - start, 'v_sha256': digest(np.zeros(n, np.float64)),
'g_sha256': digest(np.zeros(n, np.float64)), 'ref_sha256': digest(refs[t])} for t in range(start, stop)]}
intervals = list(zip([0, *cuts], [*cuts, total]))
return {'mono': record(0, total), 'parts': [record(a, b) for a, b in intervals], 'ptr': ptr, 'dst': dst,
'edge_id': edge_id, 'effective_weights': weights, 'requests': requests, 'cuts': list(cuts)}
def run(data):
return audit.audit_queues(data['mono'], data['parts'], **{k: data[k] for k in
('ptr', 'dst', 'edge_id', 'effective_weights', 'requests', 'cuts')})
class QueueAudit(unittest.TestCase):
def test_real_edge_witness_and_local_headers_cross_18_19(self):
data = fixture()
result = run(data)
self.assertTrue(result['pass_all'], result['differences'])
witness = result['first_witness']
self.assertEqual((witness['boundary_tick'], witness['source_spike_tick'], witness['arrival_tick']), (37, 22, 40))
self.assertEqual((witness['source_neuron_idx'], witness['target_neuron_idx'], witness['global_edge_id']), (0, 2, 10))
self.assertEqual(witness['effective_weight_float64_le_hex'], np.asarray(.001, dtype='<f8').tobytes().hex())
self.assertFalse(result['membrane_dynamics_independently_replayed'])
self.assertFalse(result['v_g_hashes_independently_verified'])
self.assertEqual(result['rejected_requests'], 1)
# The tick44 source has arrival62, beyond this 60-tick record; no tail
# simulation or accidental drain is performed by the audit.
self.assertEqual(result['arriving_edges'], 7)
self.assertEqual(result['accepted_arriving_edges'], 2)
json.dumps(result, allow_nan=False)
self.assertNotEqual(data['mono']['queues'][18]['joint_sha256'], data['parts'][2]['queues'][0]['joint_sha256'])
def test_current_threshold_reset_rejects_arrival_and_available_target_accepts(self):
rejected = fixture(); accepted = fixture(target_fire=False)
self.assertEqual(rejected['mono']['queues'][18]['accepted_count'], 0)
self.assertEqual(accepted['mono']['queues'][18]['accepted_count'], 2)
self.assertTrue(run(rejected)['pass_all'])
witness = run(accepted)['first_witness']
self.assertEqual((witness['source_spike_tick'], witness['arrival_tick']), (0, 18))
def test_zero_weight_edges_stay_in_queue_but_cannot_supply_witness(self):
data = fixture(weights=np.zeros(5, np.float64))
result = run(data)
self.assertTrue(result['queue_ref_request_exact'])
self.assertGreater(result['arriving_edges'], 0)
self.assertFalse(result['pass_all'])
self.assertEqual(result['status'], 'INCOMPLETE-NO-ACCEPTED-NONZERO-CROSSING')
self.assertIsNone(result['first_witness'])
def test_target_sorted_or_reversed_queue_fails_even_when_both_outputs_agree(self):
data = fixture()
bad = queue_row(18, [20, 11, 10], [0, 0, 0])
data['mono']['queues'][18] = bad
data['parts'][2]['queues'][0] = queue_row(0, [20, 11, 10], [0, 0, 0])
result = run(data)
self.assertFalse(result['pass_all'])
self.assertTrue(any(d['field'] == 'queue.ordered_edge_sha256' for d in result['differences']))
def test_global_tick_in_segment_joint_header_fails(self):
data = fixture()
data['parts'][2]['queues'][0]['joint_sha256'] = data['mono']['queues'][18]['joint_sha256']
result = run(data)
self.assertEqual(result['difference_count'], 1)
self.assertEqual(result['differences'][0]['field'], 'queue.joint_sha256')
def test_fake_request_acceptance_and_ref_hash_fail_independently(self):
for mutate in ('request', 'hash', 'terminal'):
data = fixture()
if mutate == 'request':
data['parts'][0]['requests'][2, 3:] = [1, 0]
elif mutate == 'hash':
data['parts'][1]['state_hashes'][3]['ref_sha256'] = 'a' * 64
else:
data['parts'][0]['state']['ref'][0] = 0
with self.subTest(mutate=mutate):
self.assertFalse(run(data)['pass_all'])
def test_impossible_spikes_during_ref_fail_even_with_self_consistent_records(self):
data = fixture(events=[(0, 0), (1, 0), (23, 0)])
result = run(data)
self.assertFalse(result['pass_all'])
self.assertTrue(any(d['field'] == 'spike_during_refractory' for d in result['differences']))
def test_original_accepted_request_cannot_disappear_from_all_outputs(self):
data = fixture()
rows = sorted(zip(data['requests']['tick'].tolist(), data['requests']['neuron_idx'].tolist()))
rows.append((4, 3)); rows.sort()
data['requests'] = {'tick': np.array([r[0] for r in rows], np.int32),
'neuron_idx': np.array([r[1] for r in rows], np.int32)}
result = run(data)
self.assertFalse(result['pass_all'])
self.assertTrue(any(d['field'] == 'accepted_request_without_spike' for d in result['differences']))
def test_segment_events_cannot_cancel_or_substitute_nodes(self):
data = fixture()
data['parts'][0]['state']['neuron_idx'][1] = 3
data['parts'][0]['state']['counts'] = np.array([1, 0, 0, 1], np.int32)
result = run(data)
self.assertFalse(result['pass_all'])
self.assertTrue(any(d['field'] == 'events.neuron_idx' for d in result['differences']))
def test_missing_queue_or_wrong_cuts_fail_closed(self):
for mutate in ('missing', 'cuts', 'offset', 'dtype', 'nan'):
data = fixture()
if mutate == 'missing':
data['parts'][0]['queues'] = None
elif mutate == 'cuts':
data['cuts'][1] += 1
elif mutate == 'offset':
data['mono']['global_tick_offset'] = 1
elif mutate == 'dtype':
data['effective_weights'] = data['effective_weights'].astype(np.float32)
else:
data['effective_weights'][0] = np.nan
with self.subTest(mutate=mutate), self.assertRaises(ValueError):
run(data)
def test_negative_nonzero_edge_is_valid_witness_without_valence_claim(self):
data = fixture(target_fire=False)
data['effective_weights'][:2] = 0
result = run(data)
self.assertTrue(result['pass_all'])
self.assertEqual(result['first_witness']['global_edge_id'], 20)
self.assertLess(result['first_witness']['effective_weight_SI_volt'], 0)
self.assertFalse(result['unique_causal_or_necessity_claim'])
def test_input_arrays_are_not_modified(self):
data = fixture(); original = copy.deepcopy(data)
run(data)
for key in ('ptr', 'dst', 'edge_id', 'effective_weights'):
self.assertEqual(data[key].tobytes(), original[key].tobytes())
for key in data['requests']:
self.assertEqual(data['requests'][key].tobytes(), original['requests'][key].tobytes())
self.assertEqual(data['mono']['state']['ref'].tobytes(), original['mono']['state']['ref'].tobytes())
def test_frozen_AF_queue_definition_AST_crosscheck_and_chunk_order(self):
source = Path(__file__).resolve().parents[1] / 'author-fullgraph-lab-v1' / 'oracle_full.py'
raw = source.read_bytes()
self.assertEqual(hashlib.sha256(raw).hexdigest(), audit.AF_QUEUE_SOURCE_SHA256)
tree = ast.parse(raw)
definition = [n for n in tree.body if isinstance(n, ast.FunctionDef) and n.name == 'queue_summary']
self.assertEqual(len(definition), 1)
namespace = dict(np=np, hashlib=hashlib, struct=struct, BLOCK_BYTES=8 << 20, QUEUE_HEADER=b'phase4af-queue-v1\0')
exec(compile(ast.Module(body=definition, type_ignores=[]), str(source), 'exec'), namespace)
data = fixture()
saved = audit.CHUNK
try:
audit.CHUNK = 1 # Exercise edge pass then mask pass across boundaries.
for tick, local, sources, ref in [(18, 0, np.array([0, 1]), np.array([4, 4, 22, 0], np.int32)),
(40, 3, np.array([0]), np.zeros(4, np.int32)),
(0, 0, np.empty(0, np.int32), np.zeros(4, np.int32))]:
rows, _, _ = audit._queue(tick, local, sources, ref, data['ptr'], data['dst'], data['edge_id'], data['effective_weights'], [])
indices = np.array([j for s in sources for j in range(data['ptr'][s], data['ptr'][s + 1])], np.int64)
accepted = (ref[data['dst'][indices]] == 0).astype(np.uint8)
for at, row in zip((tick, local), rows):
self.assertEqual(row, namespace['queue_summary'](at, data['edge_id'][indices], accepted, chunk_bytes=8))
finally:
audit.CHUNK = saved
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,214 @@
"""Pure synthetic records, deliberately not a neural simulation or file test."""
import hashlib
import json
import unittest
import numpy as np
import engineering_scopes as s
def sha(array):
return hashlib.sha256(np.ascontiguousarray(array).tobytes()).hexdigest()
def record(start, stop, *, reference=True, blank_requests=False):
n, k, length = 3, 2, stop - start
ticks = np.array([0, 7, 19, 21, 41, 61], np.int32)
nodes = np.array([0, 1, 2, 0, 2, 1], np.int32)
selected = (ticks >= start) & (ticks < stop)
event_ticks, event_nodes = ticks[selected] - np.int32(start), nodes[selected].copy()
v = np.array([-.05, -.04, -.03], np.float64)
g = np.array([-0., float(stop - 1), -.001], np.float64)
ref = np.zeros(n, np.int32)
state = dict(step=event_ticks, neuron_idx=event_nodes, counts=np.bincount(event_nodes, minlength=n).astype(np.int32),
v=v, g=g, ref=ref)
raw_global = np.zeros((19, n), np.uint8)
for tick, node in zip(ticks, nodes):
if max(0, stop - 19) <= tick < stop:
raw_global[tick % 19, node] = 1
normalized = raw_global[(np.arange(19) + stop % 19) % 19]
raw_local = normalized[(np.arange(19) - length % 19) % 19].copy()
generated_all = np.repeat(np.arange(1, stop + 1, dtype=np.float32)[:, None], k, axis=1)
applied_all = np.concatenate((np.zeros((18, k), np.float32), generated_all))[:stop]
pending = np.zeros((18, k), np.float32)
kept = min(stop, 18); pending[-kept:] = generated_all[-kept:]
apl_state = np.full(2, np.float32(stop + 1), np.float32)
terminal = dict(spike_ring=raw_local, apl_state=apl_state, apl_pending=pending, next_tick=np.asarray(length, np.int64))
trace = dict(generated=generated_all[start:stop].copy(), applied=applied_all[start:stop].copy(),
drive=np.repeat(np.arange(start, stop, dtype=np.float32)[:, None], 2, axis=1),
state=np.repeat(np.arange(start + 2, stop + 2, dtype=np.float32)[:, None], 2, axis=1))
hashes, queues = [], []
for local, global_tick in enumerate(range(start, stop)):
hashes.append(dict(tick=local, v_sha256=sha(v), g_sha256=sha(np.array([-0., float(global_tick), -.001], np.float64)),
ref_sha256=sha(ref)))
edges, accepted = np.array([global_tick], np.int64), np.array([global_tick % 2], np.uint8)
queues.append(dict(tick=local, count=1, accepted_count=int(accepted.sum()), ordered_edge_sha256=sha(edges),
accepted_mask_sha256=sha(accepted), joint_sha256=hashlib.sha256(
str(local).encode() + edges.tobytes() + accepted.tobytes()).hexdigest()))
requests = np.array([[int(t), int(i), 1, 1, 0] for t, i in zip(event_ticks, event_nodes)], np.int64).reshape(-1, 5)
if blank_requests:
requests = np.empty((0, 5), np.int64)
summary = dict(complete=True, pass_all=True, all_finite=True, steps=length, completed_ticks=length, nodes=n,
precision='float64', apl_precision='float32', apl_delay_steps=18,
checkpoint_continuation_used=start > 0, global_tick_offset=start, global_next_tick=stop,
effective_weights_raw_sha256='e' * 64, diagnostic_mode='reference' if reference else 'light',
full_APL_history_recorded=reference, full_tick_state_hashes_recorded=reference,
full_queue_hashes_recorded=reference)
return dict(source='synthetic', state=state, terminal=terminal, requests=requests, steps=length,
global_tick_offset=start, summary=summary, traces=trace if reference else {},
state_hashes=hashes if reference else None, queues=queues if reference else None,
terminal_origin='synthetic_recorded')
class SliceTests(unittest.TestCase):
def test_reference_slice_all_observations_exact_and_inputs_unchanged(self):
candidate, whole = record(20, 44), record(0, 64)
before = candidate['terminal']['spike_ring'].tobytes(), whole['requests'].tobytes()
report = s.compare_slice(candidate, whole)
self.assertTrue(report['numerical_exact'], report['failed_checks'])
self.assertTrue(report['observed_scope']['candidate_per_tick_state_hashes'])
self.assertTrue(report['observed_scope']['candidate_per_tick_queue_fields'])
self.assertFalse(report['observed_scope']['queue_joint_hashes_compared'])
self.assertNotEqual(candidate['queues'][0]['joint_sha256'], whole['queues'][20]['joint_sha256'])
self.assertEqual(before, (candidate['terminal']['spike_ring'].tobytes(), whole['requests'].tobytes()))
json.dumps(report, allow_nan=False)
def test_light_uses_final_raw_hash_but_does_not_fabricate_optional_history(self):
report = s.compare_slice(record(20, 44, reference=False), record(0, 64))
self.assertTrue(report['numerical_exact'])
self.assertTrue(report['observed_scope']['final_state_against_reference_tick_hash'])
for name in ('candidate_APL_histories', 'candidate_per_tick_state_hashes', 'candidate_per_tick_queue_fields'):
self.assertFalse(report['observed_scope'][name])
self.assertNotIn('slice.per_tick_state_hashes', report['checks'])
self.assertTrue(any('unobserved' in item for item in report['limitations']))
def test_short_fresh_positive_zero_padding_and_prefix_events(self):
self.assertTrue(s.compare_slice(record(0, 5), record(0, 64))['numerical_exact'])
candidate = record(0, 5, reference=False)
candidate['terminal']['apl_pending'][0, 0] = np.float32(-0.)
report = s.compare_slice(candidate, record(0, 64))
self.assertFalse(report['checks']['slice.APL_pending'])
def test_blank_requests_are_valid_but_wrong_acceptance_is_not_equal(self):
self.assertTrue(s.compare_slice(record(20, 44, blank_requests=True), record(0, 64, blank_requests=True))['numerical_exact'])
candidate = record(20, 44)
candidate['requests'][0, 3:] = [0, 1]
report = s.compare_slice(candidate, record(0, 64))
self.assertFalse(report['checks']['slice.requests_five_fields'])
def test_wrong_count_or_final_hash_binding_is_invalid_not_a_pass(self):
for mutation in ('count', 'terminal', 'last_hash'):
candidate = record(20, 44)
if mutation == 'count': candidate['state']['counts'][0] += 1
elif mutation == 'terminal': candidate['state']['g'][1] += 1.
else: candidate['state_hashes'][-1]['g_sha256'] = '0' * 64
with self.subTest(mutation=mutation), self.assertRaises(ValueError):
s.compare_slice(candidate, record(0, 64))
def test_signed_zero_light_terminal_vs_reference_hash_is_detected(self):
candidate = record(20, 44, reference=False)
candidate['state']['g'][0] = 0.
report = s.compare_slice(candidate, record(0, 64))
self.assertFalse(report['checks']['slice.final_g_raw_hash'])
def test_interior_trace_state_hash_and_queue_mismatch_are_not_hidden(self):
candidate = record(20, 44)
candidate['traces']['drive'][1, 0] += np.float32(1.)
candidate['state_hashes'][1]['v_sha256'] = '0' * 64
candidate['queues'][1]['accepted_mask_sha256'] = '1' * 64
report = s.compare_slice(candidate, record(0, 64))
for key in ('slice.APL_trace.drive', 'slice.per_tick_state_hashes', 'slice.queue_order_and_acceptance'):
self.assertFalse(report['checks'][key])
def test_missing_reference_observations_and_invalid_bounds_fail_closed(self):
for mutation in ('light_mono', 'light_labeled_full_mono', 'mono_hashes', 'candidate_traces', 'partial_trace', 'out_of_bounds', 'nonzero_mono'):
candidate, mono = record(20, 44), record(0, 64)
if mutation == 'light_mono': mono = record(0, 64, reference=False)
elif mutation == 'light_labeled_full_mono': mono['summary']['diagnostic_mode'] = 'light'
elif mutation == 'mono_hashes': mono['state_hashes'] = None
elif mutation == 'candidate_traces': candidate['traces'] = {}
elif mutation == 'partial_trace': candidate['traces'].pop('drive')
elif mutation == 'out_of_bounds': candidate = record(60, 65)
else: mono = record(1, 65)
with self.subTest(mutation=mutation), self.assertRaises(ValueError):
s.compare_slice(candidate, mono)
def test_summary_type_scope_nonfinite_and_paths_rejected(self):
for mutation in ('bool', 'failed', 'offset', 'mode', 'weight', 'nan'):
candidate = record(20, 44)
if mutation == 'bool': candidate['summary']['steps'] = True
elif mutation == 'failed': candidate['summary']['pass_all'] = False
elif mutation == 'offset': candidate['summary']['global_tick_offset'] = 0
elif mutation == 'mode': candidate['summary']['diagnostic_mode'] = 'unknown'
elif mutation == 'weight': candidate['summary'].pop('effective_weights_raw_sha256')
else: candidate['state']['g'][0] = np.nan
with self.subTest(mutation=mutation), self.assertRaises(ValueError):
s.compare_slice(candidate, record(0, 64))
with self.assertRaises(ValueError): s.compare_slice('/never/open/this', record(0, 64))
class CheckpointTests(unittest.TestCase):
def test_local_phase_normalization_without_segment_cumulative_counts_confusion(self):
candidate, prefix = record(20, 64), record(0, 64)
self.assertFalse(s.ec.bit_equal(candidate['terminal']['spike_ring'], prefix['terminal']['spike_ring']))
self.assertFalse(s.ec.bit_equal(candidate['state']['counts'], prefix['state']['counts']))
report = s.compare_checkpoint(candidate, prefix)
self.assertTrue(report['numerical_exact'], report['failed_checks'])
self.assertFalse(report['observed_scope']['counts_compared'])
self.assertFalse(report['observed_scope']['per_tick_state_histories_compared'])
def test_signed_zero_different_stop_or_effective_weights_cannot_pass(self):
for mutation in ('zero', 'stop', 'weight'):
candidate, prefix = record(20, 64, reference=False), record(0, 64, reference=False)
if mutation == 'zero': candidate['state']['g'][0] = 0.
elif mutation == 'stop': prefix = record(0, 63, reference=False)
else: candidate['summary']['effective_weights_raw_sha256'] = 'f' * 64
with self.subTest(mutation=mutation):
self.assertFalse(s.compare_checkpoint(candidate, prefix)['numerical_exact'])
class ParentCarryTests(unittest.TestCase):
def test_short_parent_carry_observed_and_json_safe(self):
report = s.compare_parent_carry(record(20, 25), record(0, 20))
self.assertTrue(report['numerical_exact'], report['failed_checks'])
self.assertEqual(report['observed_scope']['unwritten_spike_slot_count'], 14)
self.assertEqual(report['observed_scope']['unconsumed_APL_row_count'], 13)
self.assertEqual(report['observed_scope']['applied_parent_APL_rows_compared'], 5)
json.dumps(report, allow_nan=False)
def test_unwritten_or_unconsumed_corruption_detected_even_if_terminal_is_self_consistent(self):
for mutation in ('ring', 'pending', 'applied'):
candidate = record(20, 25)
if mutation == 'ring': candidate['terminal']['spike_ring'][10, 0] ^= np.uint8(1)
elif mutation == 'pending': candidate['terminal']['apl_pending'][0, 0] += np.float32(1.)
else: candidate['traces']['applied'][0, 0] += np.float32(1.)
with self.subTest(mutation=mutation):
self.assertFalse(s.compare_parent_carry(candidate, record(0, 20))['numerical_exact'])
def test_light_missing_optional_applied_trace_is_not_fabricated(self):
report = s.compare_parent_carry(record(20, 25, reference=False), record(0, 20))
self.assertTrue(report['numerical_exact'])
self.assertFalse(report['observed_scope']['applied_parent_APL_prefix_checked'])
self.assertIsNone(report['observed_scope']['applied_parent_APL_rows_compared'])
self.assertNotIn('parent.applied_APL_prefix', report['checks'])
def test_long_segment_discloses_no_unwritten_slots_and_wrong_parent_clock_fails(self):
report = s.compare_parent_carry(record(20, 44), record(0, 20))
self.assertTrue(report['numerical_exact'])
self.assertFalse(report['observed_scope']['unwritten_spike_slots_checked'])
self.assertFalse(report['observed_scope']['unconsumed_APL_prefix_checked'])
self.assertEqual(report['observed_scope']['applied_parent_APL_rows_compared'], 18)
self.assertFalse(s.compare_parent_carry(record(20, 44), record(0, 19))['numerical_exact'])
with self.assertRaises(ValueError): s.compare_parent_carry(record(0, 5), record(0, 20))
def test_long_light_carry_scope_explicitly_has_no_observed_carry_array(self):
report = s.compare_parent_carry(record(20, 44, reference=False), record(0, 20))
self.assertTrue(report['numerical_exact'])
self.assertFalse(report['observed_scope']['any_carry_values_directly_compared'])
self.assertTrue(any('clock/shape/weight identity only' in line for line in report['limitations']))
self.assertFalse(report['complete_dynamical_equivalence_proven'])
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,207 @@
import copy
import hashlib
import json
from pathlib import Path
import tempfile
import types
import unittest
from unittest.mock import patch
import numpy as np
import engineering_workflow as w
from test_engineering_inputs import spec
def template(row):
return dict(schema='phase4an-engineering-case-template-v1', row_id=row['row_id'], original_spec=spec(),
start=row['start'], stop=row['stop'], budget_root='/project/new', output_budget_bytes=8 * 1024**3)
class Workflow(unittest.TestCase):
def test_only_exact_plan_row_and_safe_output_identity(self):
reg = {'plan': w.PLAN.build_plan()}
row = reg['plan']['rows'][0]
self.assertEqual(w.row_from(reg, row['row_id']), row)
for bad in ('unknown', '../escape', '/absolute'):
with self.assertRaises(ValueError): w.row_from(reg, bad)
for bad in ('../escape', '/absolute', 'a/b', 'a\\b', '', 'a:b'):
with self.assertRaises(ValueError): w.locations(bad)
reg['plan']['rows'][0]['steps'] = True
with self.assertRaises(ValueError): w.row_from(reg, row['row_id'])
def test_real_parent_is_required_only_for_later_segments(self):
for row in w.PLAN.build_plan()['rows']:
before = template(row)
pin = None if row['parent_row_id'] is None else 'a' * 64
derived, audit = w.derive_case(before, row, pin)
self.assertEqual(derived['steps'], row['steps'])
self.assertEqual(before, template(row))
self.assertEqual(audit['input_interval'], [row['start'], row['stop']])
with self.assertRaises(ValueError):
w.derive_case(before, row, 'a' * 64 if pin is None else None)
def test_forged_template_and_boolean_interval_rejected(self):
row = w.PLAN.build_plan()['rows'][0]
for key, value in (('schema', 'wrong'), ('row_id', 'wrong'), ('start', False), ('stop', 18)):
changed = template(row); changed[key] = value
with self.subTest(key=key), self.assertRaises(ValueError): w.derive_case(changed, row, None)
changed = template(row); changed['extra'] = True
with self.assertRaises(ValueError): w.derive_case(changed, row, None)
def test_json_comparison_does_not_conflate_booleans_and_numbers(self):
self.assertFalse(w.same_json({'n': True}, {'n': 1}))
self.assertFalse(w.same_json({'n': 1.}, {'n': 1}))
with self.assertRaises(ValueError): w.canonical({'n': float('nan')})
def test_actual_precision_flags_are_checked_not_merely_recorded(self):
component = dict(cuda_matmul_allow_tf32=False, cudnn_allow_tf32=True, deterministic_algorithms=False,
cudnn_deterministic=False, cudnn_benchmark=False, float32_matmul_precision='highest',
CUBLAS_WORKSPACE_CONFIG=None, device_name='NVIDIA GeForce RTX 4090', torch='2.test', cuda='13.0',
numpy='2.test', precision_configuration_changed_by_component=False)
recorded = dict(gpu_uuid=w.GPU, settings_changed_by_environment_recorder=False,
matmul_allow_tf32=False, cudnn_allow_tf32=True, deterministic_algorithms=False,
cudnn_deterministic=False, cudnn_benchmark=False, float32_matmul_precision='highest',
CUBLAS_WORKSPACE_CONFIG=None, device_name='NVIDIA GeForce RTX 4090', torch_version='2.test',
torch_cuda_version='13.0', numpy_version='2.test')
w.verify_environment(recorded, component)
for key, value in [('cuda_matmul_allow_tf32', True), ('float32_matmul_precision', 'high'),
('precision_configuration_changed_by_component', True), ('cudnn_allow_tf32', False)]:
altered = dict(component, **{key: value})
with self.subTest(key=key), self.assertRaises(ValueError): w.verify_environment(recorded, altered)
with self.assertRaises(ValueError): w.verify_environment(dict(recorded, matmul_allow_tf32=0), component)
def test_npz_real_compression_measured_not_assumed(self):
with tempfile.TemporaryDirectory() as name:
root = Path(name)
with (root / 'compressed.npz').open('xb') as stream:
np.savez_compressed(stream, v=np.zeros(50, np.float64))
measured = w.npz_storage(root / 'compressed.npz')
self.assertGreater(measured['uncompressed_entry_bytes'], measured['compressed_entry_bytes'])
self.assertEqual(measured['entries'][0]['compression'], 8)
with (root / 'stored.npz').open('xb') as stream:
np.savez(stream, v=np.zeros(50, np.float64))
with self.assertRaises(ValueError): w.npz_storage(root / 'stored.npz')
def test_resource_limit_preserves_existing_payload(self):
with tempfile.TemporaryDirectory() as name, patch.object(w, 'EXP', Path(name)):
path = Path(name) / 'payload.bin'
path.write_bytes(b'not deleted')
reg = {'budget': {'output_budget_bytes': 1, 'minimum_free_reserve_bytes': 1}}
with self.assertRaises(ValueError): w.disk_budget(reg)
self.assertEqual(path.read_bytes(), b'not deleted')
def _temporary_gate(self, root):
for name in ('audit', 'logs', 'receipts', 'runs', 'cases'):
(root / name).mkdir()
reg = {'plan': w.PLAN.build_plan(), 'budget': {'output_budget_bytes': 1 << 20,
'minimum_free_reserve_bytes': 1}}
return reg, types.SimpleNamespace(read_registration=lambda sha: copy.deepcopy(reg))
def test_controller_refuses_partial_without_rerun_or_deletion(self):
with tempfile.TemporaryDirectory() as name, patch.object(w, 'EXP', Path(name)):
root = Path(name); reg, prep = self._temporary_gate(root)
paths = w.locations(reg['plan']['rows'][0]['row_id'])
paths['stdout'].write_bytes(b'original evidence')
with (patch.object(w, 'preparation', return_value=prep), patch.object(w, 'preflight') as pre,
patch.object(w.subprocess, 'run') as runner):
with self.assertRaisesRegex(ValueError, 'Partial row'):
w.run_registered('a' * 64)
runner.assert_not_called(); pre.assert_not_called()
self.assertEqual(paths['stdout'].read_bytes(), b'original evidence')
self.assertFalse(paths['execution'].exists())
self.assertTrue((root / 'audit/stopped.json').exists())
def test_nonzero_worker_exit_cannot_become_completed_receipt(self):
with tempfile.TemporaryDirectory() as name, patch.object(w, 'EXP', Path(name)):
root = Path(name); reg, prep = self._temporary_gate(root)
paths = w.locations(reg['plan']['rows'][0]['row_id'])
failure = types.SimpleNamespace(returncode=1)
with (patch.object(w, 'preparation', return_value=prep), patch.object(w, 'preflight'),
patch.object(w.subprocess, 'run', return_value=failure) as runner):
with self.assertRaisesRegex(ValueError, 'Worker failed'):
w.run_registered('a' * 64)
self.assertEqual(runner.call_count, 1)
self.assertTrue(paths['stdout'].exists() and paths['stderr'].exists())
self.assertFalse(paths['execution'].exists())
self.assertTrue((root / 'audit/stopped.json').exists())
def test_stopped_inventory_cannot_auto_resume(self):
with tempfile.TemporaryDirectory() as name, patch.object(w, 'EXP', Path(name)):
root = Path(name); _, prep = self._temporary_gate(root)
(root / 'audit/stopped.json').write_text('{}')
with patch.object(w, 'preparation', return_value=prep), patch.object(w.subprocess, 'run') as runner:
with self.assertRaisesRegex(ValueError, 'Stopped/completed'):
w.run_registered('a' * 64)
runner.assert_not_called()
def _completion(self, reg, root):
entries = []
for row in reg['plan']['rows']:
path = w.locations(row['row_id'])['execution']
path.write_bytes(row['row_id'].encode())
entries.append(dict(row_id=row['row_id'], execution_sha256=w.R.digest(path), reverified_existing=False))
return dict(complete=True, registration_sha256='a' * 64, expected_calls=32, executed_ticks=11074,
training=False, learning_pass_claimed=False, completed=entries)
def test_inventory_rechecks_every_actual_receipt(self):
with tempfile.TemporaryDirectory() as name, patch.object(w, 'EXP', Path(name)):
root = Path(name); reg, _ = self._temporary_gate(root)
complete = self._completion(reg, root)
w.R._save(root / 'audit/execution-complete.json', complete)
with patch.object(w, 'verify_execution', side_effect=lambda r, s, row: dict(row=row)) as verify:
records = w.verified_inventory(reg, 'a' * 64)
self.assertEqual(verify.call_count, 32)
self.assertEqual(list(records), [row['row_id'] for row in reg['plan']['rows']])
def test_inventory_rejects_reorder_missing_and_changed_receipt(self):
for corruption in ('reorder', 'missing', 'changed', 'claimed_training', 'wrong_ticks'):
with self.subTest(corruption=corruption), tempfile.TemporaryDirectory() as name, patch.object(w, 'EXP', Path(name)):
root = Path(name); reg, _ = self._temporary_gate(root)
complete = self._completion(reg, root)
if corruption == 'reorder': complete['completed'].reverse()
if corruption == 'missing': complete['completed'].pop()
if corruption == 'changed': w.locations(reg['plan']['rows'][0]['row_id'])['execution'].write_bytes(b'changed')
if corruption == 'claimed_training': complete['training'] = True
if corruption == 'wrong_ticks': complete['executed_ticks'] += 1
w.R._save(root / 'audit/execution-complete.json', complete)
with patch.object(w, 'verify_execution'), self.assertRaises(ValueError):
w.verified_inventory(reg, 'a' * 64)
def test_seal_contains_nested_manifests_and_self_checksum(self):
with tempfile.TemporaryDirectory() as name, patch.object(w, 'EXP', Path(name)):
root = Path(name)
(root / 'runs' / 'one').mkdir(parents=True)
(root / 'runs/one/manifest.json').write_bytes(b'nested evidence')
(root / 'payload').write_bytes(b'payload')
result = w.seal_experiment()
manifest = json.loads((root / 'manifest.json').read_text())
self.assertEqual([row['path'] for row in manifest['files']], ['payload', 'runs/one/manifest.json'])
lines = (root / 'checksums.sha256').read_text().splitlines()
self.assertIn(result['manifest_sha256'] + ' manifest.json', lines)
self.assertEqual(len(lines), 3)
self.assertEqual(result['checksums_sha256'], hashlib.sha256((root / 'checksums.sha256').read_bytes()).hexdigest())
before = (root / 'manifest.json').read_bytes()
with self.assertRaises(ValueError): w.seal_experiment()
self.assertEqual((root / 'manifest.json').read_bytes(), before)
def test_seal_rejects_bytecode_instead_of_silently_losing_it(self):
with tempfile.TemporaryDirectory() as name, patch.object(w, 'EXP', Path(name)):
root = Path(name); (root / '__pycache__').mkdir()
(root / '__pycache__/a.pyc').write_bytes(b'cache')
with self.assertRaises(ValueError): w.seal_experiment()
self.assertFalse((root / 'manifest.json').exists())
def test_finalize_rejects_recomputed_difference_before_sealing(self):
with tempfile.TemporaryDirectory() as name, patch.object(w, 'EXP', Path(name)):
root = Path(name); _, prep = self._temporary_gate(root)
w.R._save(root / 'audit/engineering-analysis.json', dict(status='saved'))
with patch.object(w, 'evaluate', return_value=({}, {'status': 'different'})), patch.object(w, 'seal_experiment') as seal:
with self.assertRaisesRegex(ValueError, 'Recomputed analysis'):
w.finalize('a' * 64)
seal.assert_not_called()
self.assertFalse((root / 'audit/final-status.json').exists())
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,399 @@
"""CPU synthetic checkpoint/clock tests; not a brain or GPU equivalence test."""
import ast
import copy
import json
from pathlib import Path
import tempfile
import types
import unittest
from unittest.mock import patch
import numpy as np
import runtime_continuation as c
R = c.R
AJ = Path(__file__).resolve().parent.parent / 'apl-timing-lab-v1/gpu_timing.py'
class RingTests(unittest.TestCase):
def execute(self, start, steps, raw, phase, apl_pending, events, *, admit_global=True):
ring = c.rotate_spike_ring(raw, phase)
apl = apl_pending.copy()
due, applied = [], []
for tick in range(steps):
absolute = start + tick
ring[tick % 19] = events[absolute]
admitted = absolute >= 18 if admit_global else tick >= 18
due.append(ring[(tick - 18) % 19].copy() if admitted else np.zeros(ring.shape[1], np.uint8))
slot = tick % 18
applied.append(apl[slot].copy())
apl[slot] = np.float32(absolute + 1)
head = steps % 18
return ring, steps, np.concatenate((apl[head:], apl[:head])), np.asarray(due), np.asarray(applied)
def test_multi_segments_preserve_all_future_arrivals_and_apl_slots(self):
rng = np.random.default_rng(193)
events = rng.integers(0, 2, (80, 4), dtype=np.uint8)
initial, pending = np.zeros((19, 4), np.uint8), np.zeros((18, 2), np.float32)
full = self.execute(0, 80, initial, 0, pending, events)
for lengths in ((5, 13, 1, 22, 39), (17, 1, 1, 18, 43), (19, 19, 19, 23), (1,) * 80):
ring, phase, tail, offset, due, applied = initial, 0, pending, 0, [], []
for steps in lengths:
ring, phase, tail, d, a = self.execute(offset, steps, ring, phase, tail, events)
due.append(d); applied.append(a); offset += steps
np.testing.assert_array_equal(np.concatenate(due), full[3])
np.testing.assert_array_equal(np.concatenate(applied), full[4])
np.testing.assert_array_equal(c.rotate_spike_ring(ring, phase), c.rotate_spike_ring(full[0], 80))
np.testing.assert_array_equal(tail, full[2])
def test_parent_shorter_than_delay_needs_global_admission(self):
events = np.zeros((24, 2), np.uint8); events[0, 1] = 1
parent = self.execute(0, 5, np.zeros((19, 2), np.uint8), 0, np.zeros((18, 2), np.float32), events)
child = self.execute(5, 19, *parent[:3], events)
wrong = self.execute(5, 19, *parent[:3], events, admit_global=False)
self.assertEqual(child[3][13, 1], 1) # absolute tick 18, local 13
self.assertEqual(wrong[3][13, 1], 0)
def test_local_tick_zero_request_uses_global_clock_and_ring_phase(self):
events = np.zeros((50, 3), np.uint8)
events[19, 2] = 1 # Synthetic US request at child's local tick zero.
first = self.execute(0, 19, np.zeros((19, 3), np.uint8), 0, np.zeros((18, 2), np.float32), events)
child = self.execute(19, 31, *first[:3], events)
self.assertEqual(child[3][18, 2], 1)
self.assertEqual(np.count_nonzero(child[3][:18, 2]), 0)
def test_rotation_is_copy_and_rejects_invalid_ring(self):
raw = np.zeros((19, 2), np.uint8); raw[18, 1] = 1
before = raw.tobytes()
result = c.rotate_spike_ring(raw, 19)
result[:] = 1
self.assertEqual(raw.tobytes(), before)
for value, phase in ((raw.astype(np.int32), 0), (raw[:18], 0), (raw, -1), (raw, True)):
with self.assertRaises(ValueError):
c.rotate_spike_ring(value, phase)
class ASTTests(unittest.TestCase):
def test_only_restore_and_global_admission_edits_in_both_modes(self):
for mode in ('light', 'reference'):
tree, audit = c.derive_continued(AJ.read_bytes(), mode)
self.assertTrue(audit['full_AST_rollback_identical'])
self.assertTrue(audit['kernels_sorting_and_local_ring_math_unchanged'])
self.assertEqual(sum(isinstance(node, ast.If) and ast.unparse(node.test) == c.CLOCK_NEW for node in ast.walk(tree)), 1)
self.assertEqual(sum(isinstance(node, ast.Expr) and ast.unparse(node) == c.RESTORE for node in ast.walk(tree)), 1)
compile(tree, '<pure continuation AST>', 'exec')
def test_reference_loop_diff_is_exactly_admission_condition(self):
base, _ = R.derive_ast(AJ.read_bytes(), 'reference')
actual, _ = c.derive_continued(AJ.read_bytes(), 'reference')
loops = lambda tree: next(node for node in ast.walk(tree) if isinstance(node, ast.For) and
isinstance(node.target, ast.Name) and node.target.id == 'tick')
modified = copy.deepcopy(loops(actual))
changed = [node for node in ast.walk(modified) if isinstance(node, ast.If) and ast.unparse(node.test) == c.CLOCK_NEW]
changed[0].test = ast.parse(c.CLOCK_OLD, mode='eval').body
self.assertEqual(R._dump(modified), R._dump(loops(base)))
def test_compile_never_mutates_sealed_module_namespace(self):
module = types.ModuleType('fake_aj'); module.__file__ = str(AJ)
original = dict(module.__dict__)
runtime = types.SimpleNamespace(write_json=lambda *_: None, audit_sequence=lambda *_: None)
function, _ = c.compile_continued(module, runtime, 'light')
self.assertEqual(module.__dict__, original)
self.assertEqual(function.__globals__['__file__'], str(Path(c.__file__).resolve()))
self.assertIs(function.__globals__['_continuation'], runtime)
class ArrayTarget:
def __init__(self, value):
self.value = np.array(value, copy=True)
def copy_(self, value):
self.value[...] = value
def any(self):
return np.asarray(self.value.any())
class RestoreTests(unittest.TestCase):
def test_reference_fifo_audit_preserves_carried_bits_for_short_segment(self):
initial = np.arange(36, dtype=np.float32).reshape(18, 2)
initial[0, 0] = np.float32(-0.)
generated = np.arange(10, dtype=np.float32).reshape(5, 2)
joined = np.concatenate((initial, generated))
applied, pending = joined[:5], joined[5:]
result = c.audit_carried_sequence(generated, applied, pending, initial)
self.assertTrue(result['sample_bits_preserved'])
bad = applied.copy(); bad[0, 0] = np.float32(0.)
with self.assertRaisesRegex(ValueError, 'bitwise'):
c.audit_carried_sequence(generated, bad, pending, initial)
def test_timing_counts_and_ranges_use_absolute_clock_not_segment_length(self):
obj = object.__new__(c.ContinuedRuntime)
obj.global_offset, obj.spec, obj.mode = 41, {'steps': 5}, 'reference'
obj.parent = {'terminal': {'apl_pending': np.zeros((18, 4064), np.float32)}}
result = obj._timing_metadata({'pending_generated_count': 5, 'pending_initial_zero_prefix': 13,
'initial_padding_positive_zero': True})
self.assertEqual(result['pending_generated_count'], 18)
self.assertEqual(result['pending_initial_zero_prefix'], 0)
self.assertEqual(result['pending_application_tick_start'], 46)
self.assertEqual(result['pending_generated_tick_start'], 28)
self.assertEqual(result['pending_segment_generated_rows'], 5)
self.assertEqual(result['pending_parent_generated_rows'], 13)
self.assertEqual(result['applied_parent_generated_rows'], 5)
self.assertNotIn('initial_padding_positive_zero', result)
def test_restore_carries_exact_state_and_queues_but_not_segment_counts(self):
parent = {'global_next_tick': 41, 'state': {'v': np.array([-.05, -.04], np.float64),
'g': np.array([-0., -.002], np.float64), 'ref': np.array([22, 3], np.int32)},
'terminal': {'spike_ring': np.zeros((19, 2), np.uint8), 'next_tick': np.asarray(5, np.int64),
'apl_pending': np.arange(36, dtype=np.float32).reshape(18, 2), 'apl_state': np.array([2., 3.], np.float32)}}
parent['terminal']['spike_ring'][4, 1] = 1
obj = object.__new__(c.ContinuedRuntime)
obj.parent, obj.installations, obj.restorations = parent, 1, 0
obj.torch = types.SimpleNamespace(float64=np.float64, float32=np.float32, int32=np.int32, uint8=np.uint8)
v, g, ref, counts = (ArrayTarget(np.zeros(2, dtype)) for dtype in (np.float64, np.float64, np.int32, np.int32))
ring, apl_ring = ArrayTarget(np.zeros((19, 2), np.uint8)), ArrayTarget(np.zeros((18, 2), np.float32))
apl = types.SimpleNamespace(state=ArrayTarget(np.zeros(2, np.float32)))
upload = lambda value, dtype: np.array(value, dtype=dtype, copy=True)
obj.restore(v, g, ref, counts, ring, apl_ring, apl, upload)
self.assertEqual(g.value.tobytes(), parent['state']['g'].tobytes())
np.testing.assert_array_equal(ref.value, parent['state']['ref'])
np.testing.assert_array_equal(counts.value, [0, 0])
np.testing.assert_array_equal(ring.value, c.rotate_spike_ring(parent['terminal']['spike_ring'], 5))
np.testing.assert_array_equal(apl_ring.value, parent['terminal']['apl_pending'])
np.testing.assert_array_equal(apl.state.value, parent['terminal']['apl_state'])
with self.assertRaises(ValueError):
obj.restore(v, g, ref, counts, ring, apl_ring, apl, upload)
class CheckpointTests(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.root = Path(self.tmp.name).resolve()
self.n = patch.object(c, 'N', 4); self.e = patch.object(c, 'E', 6)
self.n.start(); self.e.start()
def tearDown(self):
self.e.stop(); self.n.stop(); self.tmp.cleanup()
def save(self, path, value):
R._save(path, value)
def make(self, name, steps=5, parent=None, corrupt_carry=None):
directory = self.root / name; directory.mkdir()
inputs = self.root / (name + '-inputs'); inputs.mkdir()
spec = {'steps': steps}
if parent is not None:
spec['parent_checkpoint_manifest_sha256'] = parent['manifest_sha256']
self.save(inputs / 'spec.json', spec)
overlay_bytes = b'synthetic-bound-overlay-not-a-brain'
(inputs / 'overlay.npz').write_bytes(overlay_bytes)
meta = {'overlay_sha256': R.digest(inputs / 'overlay.npz'), 'mask_sha256': R.MASK_SHA,
'base_candidate_sha256': 'b' * 64}
self.save(inputs / 'overlay-manifest.json', meta)
(directory / 'overlay-input.npz').write_bytes(overlay_bytes)
self.save(directory / 'overlay-manifest.json', meta)
event_tick, event_neuron = np.array([0], np.int32), np.array([1], np.int32)
counts = np.array([0, 1, 0, 0], np.int32)
state = {'step': event_tick, 'neuron_idx': event_neuron, 'counts': counts,
'v': np.full(4, -.052, np.float64), 'g': np.array([-0., -.003, 0., 1.], np.float64),
'ref': np.array([0, 22, 3, 0], np.int32)}
ring = np.zeros((19, 4), np.uint8) if parent is None else c.rotate_spike_ring(parent['terminal']['spike_ring'], parent['steps'])
for tick in range(steps):
ring[tick % 19] = 0
if tick == 0:
ring[0, 1] = 1
pending = np.zeros((18, 4064), np.float32)
if corrupt_carry == 'spike':
ring[18, 0] ^= np.uint8(1)
elif corrupt_carry == 'apl':
pending[0, 0] = np.float32(1.)
terminal = {'spike_ring': ring, 'apl_pending': pending, 'apl_state': np.array([1., 2.], np.float32),
'next_tick': np.asarray(steps, np.int64)}
np.savez_compressed(directory / 'state.npz', **state)
np.savez_compressed(directory / 'terminal-queues.npz', **terminal)
np.save(directory / 'apl-pending-tail.float32.npy', pending)
self.save(directory / 'apl-timing-audit.json', {'apl_delay_steps': 18, 'brain_tail_advanced': False,
'pending_raw_sha256': R._raw(pending), 'final_apl_state_u32_bits': terminal['apl_state'].view(np.uint32).tolist()})
offset = 0 if parent is None else parent['global_next_tick']
summary = {'complete': True, 'pass_all': True, 'all_finite': True, 'steps': steps, 'completed_ticks': steps,
'nodes': 4, 'edges': 6, 'precision': 'float64', 'apl_precision': 'float32',
'within_trial_weight_updates': False, 'checkpoint_continuation_used': parent is not None,
'effective_weights_raw_sha256': 'e'*64, 'global_tick_offset': offset, 'global_next_tick': offset + steps,
'input_sha256': {'spec.json': R.digest(inputs / 'spec.json')}}
if parent is not None:
summary['parent_checkpoint_manifest_sha256'] = parent['manifest_sha256']
self.save(directory / 'summary.json', summary)
self.save(directory / 'finite-audit.json', {'all_finite': True, 'first_failure_tick': -1, 'finite_observed_ticks': steps})
self.save(directory / 'overlay-audit.json', {'complete': True, 'weights_updated_during_run': False,
'effective_weights_raw_sha256': 'e'*64, 'mask_sha256': R.MASK_SHA, 'am_helper_sha256': R.AM_OVERLAY_SHA,
'base_candidate_sha256': 'b'*64})
provenance = {'schema': 'phase4an-reporting-adapter-v1' if parent is None else 'phase4an-continuation-runtime-v1',
'complete': True, 'weights_updated_during_run': False, 'old_globals_patched': False,
'source_file': R.__file__ if parent is None else c.__file__,
'source_sha256': c.LIGHT_SHA if parent is None else R.digest(c.__file__),
'input_sha256': {str(path): R.digest(path) for path in inputs.iterdir()}}
if parent is not None:
provenance.update(parent_checkpoint_manifest_sha256=parent['manifest_sha256'],
global_tick_offset=offset, global_next_tick=offset + steps)
for key, file in (('summary_sha256', 'summary.json'), ('finite_audit_sha256', 'finite-audit.json'),
('terminal_queues_sha256', 'terminal-queues.npz'), ('overlay_audit_sha256', 'overlay-audit.json'),
('state_sha256', 'state.npz')):
provenance[key] = R.digest(directory / file)
self.save(directory / 'runtime-provenance.json', provenance)
if parent is not None:
self.save(directory / c.RECEIPT, {'schema': 'phase4an-continuation-receipt-v1', 'complete': True,
'parent_checkpoint_dir': str(parent['directory']), 'parent_checkpoint_manifest_sha256': parent['manifest_sha256'],
'segment_steps': steps, 'counts_scope': 'segment_only_not_lifetime', 'weights_updated_during_run': False,
'global_tick_offset': offset, 'global_next_tick': offset + steps, 'payload_sha256': c._tree_hashes(directory)})
return directory
def test_fresh_seal_is_exclusive_and_binds_state_bytes(self):
directory = self.make('fresh')
sealed = c.seal_checkpoint(directory)
loaded = c.load_checkpoint(directory, sealed['sha256'])
self.assertEqual(loaded['global_next_tick'], 5)
self.assertEqual(loaded['state']['g'].tobytes(), np.array([-0., -.003, 0., 1.], np.float64).tobytes())
self.assertIn('state.npz', sealed['files_sha256'])
self.assertFalse(sealed['pre_seal_state_integrity_independently_proven'])
with self.assertRaisesRegex(ValueError, 'already sealed'):
c.seal_checkpoint(directory)
with (directory / 'state.npz').open('ab') as stream:
stream.write(b'tampered')
with self.assertRaisesRegex(ValueError, 'inventory/hash'):
c.load_checkpoint(directory, sealed['sha256'])
def test_multisegment_receipts_bind_absolute_offset_and_local_phase(self):
parent = None
total = 0
for i, steps in enumerate((5, 13, 1, 19)):
directory = self.make('segment-' + str(i), steps, parent)
sealed = c.seal_checkpoint(directory)
parent = c.load_checkpoint(directory, sealed['sha256'])
total += steps
self.assertEqual(parent['global_next_tick'], total)
self.assertEqual(parent['terminal']['next_tick'].item(), steps)
self.assertEqual(parent['ancestry_directories'], [self.root / ('segment-' + str(j)) for j in range(i, -1, -1)])
self.assertEqual(total, 38)
def test_run_rejects_output_and_build_in_every_ancestor_before_overlay_or_gpu(self):
parent = None
for i in range(3):
directory = self.make('ancestor-' + str(i), 20, parent)
seal = c.seal_checkpoint(directory)
parent = c.load_checkpoint(directory, seal['sha256'])
case = self.root / 'next-case'; case.mkdir()
self.save(case / 'spec.json', {'max_steps': 5, 'steps': 5, 'dt_ms': .1, 'duration_ms': .5,
'apl_delay_steps': 18, 'record_history': False,
'parent_checkpoint_manifest_sha256': parent['manifest_sha256']})
before = {path: c._tree_hashes(path) for path in parent['ancestry_directories']}
with patch.object(R, 'load_overlay', side_effect=AssertionError('Overlay/GPU preparation must not begin')) as overlay:
for ancestor in parent['ancestry_directories']:
for forbidden in (ancestor, ancestor / 'new-child'):
for output, build in ((forbidden, self.root / 'safe-build'), (self.root / 'safe-output', forbidden)):
with self.subTest(ancestor=ancestor.name, output=output, build=build), self.assertRaises(ValueError):
c.run_continued(case, output, build, self.root / 'unused-overlay.npz',
self.root / 'unused-manifest.json', parent['directory'])
overlay.assert_not_called()
self.assertEqual(before, {path: c._tree_hashes(path) for path in parent['ancestry_directories']})
self.assertFalse((self.root / 'safe-output').exists())
self.assertFalse((self.root / 'safe-build').exists())
def test_reject_bad_external_pin_extra_file_and_error_marker(self):
directory = self.make('fresh')
sealed = c.seal_checkpoint(directory)
with self.assertRaisesRegex(ValueError, 'external pin'):
c.load_checkpoint(directory, '0' * 64)
(directory / 'unexpected.bin').write_bytes(b'extra')
with self.assertRaisesRegex(ValueError, 'inventory/hash'):
c.load_checkpoint(directory, sealed['sha256'])
failure = self.make('failure')
self.save(failure / 'runtime-error.json', {'complete': False})
with self.assertRaisesRegex(ValueError, 'failure/error'):
c.seal_checkpoint(failure)
def test_self_consistent_receipts_do_not_hide_unwritten_carry_slot_corruption(self):
directory = self.make('parent', 20)
sealed = c.seal_checkpoint(directory)
parent = c.load_checkpoint(directory, sealed['sha256'])
for kind, message in (('spike', 'carried spike'), ('apl', 'carried APL')):
child = self.make('bad-' + kind, 5, parent, corrupt_carry=kind)
# The synthetic writer hashes its incorrect output consistently;
# the independent parent-to-child carry invariant must still reject.
with self.subTest(kind=kind), self.assertRaisesRegex(ValueError, message):
c.seal_checkpoint(child)
def test_reject_partial_receipt_and_modified_original_input(self):
directory = self.make('bad-receipt')
(directory / 'summary.json').write_text('{}', encoding='utf-8')
with self.assertRaises(ValueError):
c.seal_checkpoint(directory)
directory = self.make('modified-input')
(self.root / 'modified-input-inputs' / 'spec.json').write_text('{}', encoding='utf-8')
with self.assertRaisesRegex(ValueError, 'input/source'):
c.seal_checkpoint(directory)
def test_descendant_rejects_mutated_ancestor_even_if_own_files_are_unchanged(self):
directory = self.make('ancestor', 20)
sealed = c.seal_checkpoint(directory)
parent = c.load_checkpoint(directory, sealed['sha256'])
child = self.make('descendant', 5, parent)
child_seal = c.seal_checkpoint(child)
with (directory / 'state.npz').open('ab') as stream:
stream.write(b'ancestor changed after child seal')
with self.assertRaisesRegex(ValueError, 'inventory/hash'):
c.load_checkpoint(child, child_seal['sha256'])
def test_self_consistent_hashes_cannot_hide_parent_pin_or_weight_disagreement(self):
directory = self.make('bound-parent', 20)
sealed = c.seal_checkpoint(directory)
parent = c.load_checkpoint(directory, sealed['sha256'])
def rewrite(path, value):
path.write_text(json.dumps(value, sort_keys=True), encoding='utf-8')
for kind in ('case', 'summary', 'provenance', 'weights'):
child = self.make('bound-child-' + kind, 5, parent)
spec_path = self.root / ('bound-child-' + kind + '-inputs') / 'spec.json'
summary = R._read_json(child / 'summary.json')
provenance = R._read_json(child / 'runtime-provenance.json')
if kind == 'case':
spec = R._read_json(spec_path); spec['parent_checkpoint_manifest_sha256'] = '0' * 64
rewrite(spec_path, spec)
summary['input_sha256']['spec.json'] = R.digest(spec_path)
provenance['input_sha256'][str(spec_path)] = R.digest(spec_path)
elif kind in ('summary', 'provenance'):
target = summary if kind == 'summary' else provenance
target['parent_checkpoint_manifest_sha256'] = '0' * 64
else:
summary['effective_weights_raw_sha256'] = 'f' * 64
audit = R._read_json(child / 'overlay-audit.json'); audit['effective_weights_raw_sha256'] = 'f' * 64
rewrite(child / 'overlay-audit.json', audit)
provenance['overlay_audit_sha256'] = R.digest(child / 'overlay-audit.json')
rewrite(child / 'summary.json', summary)
provenance['summary_sha256'] = R.digest(child / 'summary.json')
rewrite(child / 'runtime-provenance.json', provenance)
receipt = R._read_json(child / c.RECEIPT)
receipt['payload_sha256'] = c._tree_hashes(child); receipt['payload_sha256'].pop(c.RECEIPT)
rewrite(child / c.RECEIPT, receipt)
# All individual byte hashes are now consistent. Only semantic
# cross-receipt binding or parent/child weight equality rejects it.
with self.subTest(kind=kind), self.assertRaisesRegex(ValueError, 'weights changed|pin disagrees'):
c.seal_checkpoint(child)
def test_reject_invalid_state_dtype_events_ref_and_ring(self):
directory = self.make('arrays')
state = c._npz(directory / 'state.npz', ('step', 'neuron_idx', 'counts', 'v', 'g', 'ref'))
terminal = c._npz(directory / 'terminal-queues.npz', ('spike_ring', 'apl_state', 'apl_pending', 'next_tick'))
for key, value in (('v', state['v'].astype(np.float32)), ('g', np.full(4, np.nan)),
('ref', np.full(4, 23, np.int32)), ('step', np.array([5], np.int32)),
('counts', np.zeros(4, np.int32))):
with self.subTest(key=key), self.assertRaises(ValueError):
c.validate_state(dict(state, **{key: value}), terminal, 5, fresh=True)
bad = dict(terminal, spike_ring=terminal['spike_ring'].copy()); bad['spike_ring'][0, 1] = 0
with self.assertRaisesRegex(ValueError, 'emitted'):
c.validate_state(state, bad, 5, fresh=True)
if __name__ == '__main__':
unittest.main()
@@ -0,0 +1,298 @@
"""Synthetic AST/array diagnostics only. No real brain/GPU or old-file writes."""
import ast
import copy
import json
from pathlib import Path
import tempfile
import types
import unittest
from unittest.mock import patch
import numpy as np
import runtime_light as r
AJ = Path(__file__).absolute().parent.parent / 'apl-timing-lab-v1/gpu_timing.py'
AM = Path(__file__).absolute().parent.parent / 'wholebrain-plastic-interface-v1/overlay.py'
class ScalarTensor:
"""Tiny NumPy scalar stub for diagnostic algebra, not a neuron simulator."""
def __init__(self, value):
self.value = np.asarray(value)
def logical_and_(self, other):
self.value = self.value & other.value
return self
def __lt__(self, other):
return ScalarTensor(self.value < other)
def __and__(self, other):
return ScalarTensor(self.value & other.value)
def __invert__(self):
return ScalarTensor(~self.value)
def all(self):
return ScalarTensor(self.value.all())
def item(self):
return self.value.item()
class TorchScalars:
bool, int64 = np.bool_, np.int64
ones = staticmethod(lambda shape, device, dtype: ScalarTensor(np.ones(shape, dtype)))
zeros = staticmethod(lambda shape, device, dtype: ScalarTensor(np.zeros(shape, dtype)))
full = staticmethod(lambda shape, value, device, dtype: ScalarTensor(np.full(shape, value, dtype)))
ones_like = staticmethod(lambda value: ScalarTensor(np.ones_like(value.value)))
full_like = staticmethod(lambda value, fill: ScalarTensor(np.full_like(value.value, fill)))
isfinite = staticmethod(lambda value: ScalarTensor(np.isfinite(value)))
where = staticmethod(lambda condition, first, other: ScalarTensor(np.where(condition.value, first.value, other.value)))
class AstTests(unittest.TestCase):
def source(self):
if not AJ.is_file():
self.skipTest('Original pinned AJ source is not installed')
return AJ.read_bytes()
def test_math_projection_equal_light_and_reference_and_guard_explicit(self):
for mode in ('light', 'reference'):
tree, audit = r.derive_ast(self.source(), mode)
self.assertTrue(audit['mathematical_ast_identical'])
self.assertEqual(audit['math_ast_sha256'], audit['derived_math_ast_sha256'])
self.assertIn("spec['max_steps']", ast.unparse(tree.body[0]))
compile(tree, '<synthetic-AST-check>', 'exec')
def test_reference_entire_tick_loop_remains_identical(self):
source = ast.parse(self.source())
original = next(node for node in source.body if isinstance(node, ast.FunctionDef) and node.name == 'run_gpu')
tree, _ = r.derive_ast(self.source(), 'reference')
tick = lambda f: next(n for n in ast.walk(f) if isinstance(n, ast.For) and isinstance(n.target, ast.Name) and n.target.id == 'tick')
self.assertEqual(r._dump(tick(original)), r._dump(tick(tree.body[-1])))
def test_light_retains_events_requests_kernels_and_no_full_state_cpu_copy(self):
tree, _ = r.derive_ast(self.source(), 'light')
function = tree.body[-1]
tick = next(n for n in ast.walk(function) if isinstance(n, ast.For) and isinstance(n.target, ast.Name) and n.target.id == 'tick')
text = ast.unparse(tick)
for expected in ('ext.state_step', 'ext.fill_edges', 'ext.apply_arrivals', 'apl.advance',
'event_steps.append', 'event_nodes.append', 'writer.writerows', '_runtime.after_tick',
'apl_ring[slot].clone()', 'apl_ring[slot].copy_(h)'):
self.assertIn(expected, text)
for forbidden in ('v.cpu()', 'g.cpu()', 'ref.cpu()', 'drive.cpu()', 'h.cpu()', 'queue_summary(',
'queue_output', 'hash_output', 'np.isfinite'):
self.assertNotIn(forbidden, text)
def test_altered_math_is_detected_and_source_pin_rejects_comments(self):
tree, audit = r.derive_ast(self.source(), 'light')
function = copy.deepcopy(tree.body[-1])
node = next(n for n in ast.walk(function) if isinstance(n, ast.Assign) and
ast.unparse(n) == 'g[kc_local_gpu] = g[kc_local_gpu] - delta_si')
node.value.op = ast.Add()
self.assertNotEqual(r.math_projection(function), r.math_projection(tree.body[-1]))
with self.assertRaises(ValueError):
r.derive_ast(self.source() + b'\n# altered\n', 'light')
def test_fresh_namespace_no_fake_file_no_old_globals_changed(self):
self.source()
module = types.ModuleType('synthetic_AJ')
module.__file__ = str(AJ)
module.marker = object()
before = dict(module.__dict__)
runtime = types.SimpleNamespace(write_json=lambda *_: None)
function, audit = r.compile_runtime(module, runtime, 'light')
self.assertEqual(before, module.__dict__)
self.assertEqual(function.__globals__['__file__'], str(Path(r.__file__).absolute()))
self.assertNotEqual(function.__globals__['__file__'], str(AJ))
self.assertFalse(audit['old_globals_patched'])
class ContractTests(unittest.TestCase):
def spec(self, steps=1200):
return dict(max_steps=steps, steps=steps, dt_ms=.1, duration_ms=steps*.1,
apl_delay_steps=18, record_history=False)
def test_directory_parent_traversal_cannot_bypass_input_or_source_boundaries(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()
case, models, graph, bundle = (root / name for name in ('case', 'models', 'graph', 'bundle'))
for path in (case, models, graph, bundle, root / 'new'):
path.mkdir()
safe = root / 'new'
for protected in (case, models, graph, bundle):
alias = safe / '..' / protected.name / 'cache'
for output, build in ((alias, safe / 'build'), (safe / 'out', alias)):
with self.subTest(protected=protected.name, output=output), self.assertRaisesRegex(ValueError, 'cannot write'):
r.execution_paths(case, output, build, (models, graph, bundle))
actual = r.execution_paths(case, safe / '..' / 'safe-output', safe / '..' / 'safe-build', (models, graph, bundle))
self.assertEqual(actual, (case, root / 'safe-output', root / 'safe-build'))
self.assertFalse((root / 'safe-output').exists())
self.assertFalse((root / 'safe-build').exists())
def test_directory_link_alias_cannot_bypass_protected_boundary(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()
case, models = root / 'case', root / 'models'
case.mkdir(); models.mkdir()
alias = root / 'alias-to-models'
try:
alias.symlink_to(models, target_is_directory=True)
except (OSError, NotImplementedError) as error:
self.skipTest('Directory symlink unavailable on this host: ' + str(error))
for output, build in ((alias / 'out', root / 'build'), (root / 'out', alias / 'build')):
with self.subTest(output=output), self.assertRaisesRegex(ValueError, 'cannot write'):
r.execution_paths(case, output, build, (models,))
# A protected root supplied through an alias is also canonicalized.
with self.assertRaisesRegex(ValueError, 'cannot write'):
r.execution_paths(case, root / 'out', models / 'build', (alias,))
def test_explicit_bounds_1000_1200_retention_no_hidden_duration(self):
with tempfile.TemporaryDirectory() as directory:
for steps in (1000, 1200, 100000):
self.assertEqual(r.validate_spec(self.spec(steps), directory)['steps'], steps)
for update in ({'max_steps': 999}, {'max_steps': True}, {'steps': 1200.},
{'duration_ms': 100.}, {'dt_ms': .2}, {'apl_delay_steps': 0},
{'within_trial_weight_updates': True}, {'checkpoint': None}):
with self.subTest(update=update), self.assertRaises(ValueError):
r.validate_spec(dict(self.spec(), **update), directory)
Path(directory, 'arrays.npz').write_bytes(b'unsupported checkpoint')
with self.assertRaises(ValueError):
r.validate_spec(self.spec(), directory)
def test_finite_observer_remembers_nan_that_later_disappears(self):
observer = r.FiniteObserver(TorchScalars, 'CPU-test-stub')
initial = np.array([1., -2.], np.float64)
saved = initial.tobytes()
observer.observe(0, initial)
observer.observe(1, np.array([np.nan], np.float32))
observer.observe(2, np.array([0.], np.float64))
result = observer.result()
self.assertFalse(result['all_finite'])
self.assertEqual(result['first_failure_tick'], 1)
self.assertEqual(result['finite_observed_ticks'], 3)
self.assertEqual(saved, initial.tobytes())
with self.assertRaises(ValueError):
observer.observe(4, initial)
def test_finite_observer_all_valid_reports_minus_one(self):
observer = r.FiniteObserver(TorchScalars, 'CPU-test-stub')
for tick in range(4):
observer.observe(tick, np.array([1, 0], np.int32), np.array([-0., 1.], np.float64))
self.assertTrue(observer.result()['all_finite'])
self.assertEqual(observer.result()['first_failure_tick'], -1)
def test_summary_has_no_fake_outer_learning_claim_and_light_audit_is_not_full(self):
obj = object.__new__(r._Runtime)
obj.mode, obj.spec = 'light', dict(max_steps=1200, steps=1200, outer_protocol='conditioning-owned-by-parent')
obj.result = dict(first_failure_tick=-1, arriving_edges=11, accepted_arriving_edges=7)
inherited = dict(learning=False, arriving_edges=0, accepted_arriving_edges=0, complete=True,
apl_timing={'generated_trace': 'absent.npy', 'applied_trace': 'absent.npy'})
final = obj.summary(inherited)
self.assertNotIn('learning', final)
self.assertFalse(final['within_trial_weight_updates'])
self.assertFalse(final['learning_pass_claimed'])
self.assertFalse(final['full_sequence_bitwise_audit_performed'])
self.assertEqual(final['arriving_edges'], 11)
self.assertIsNone(final['apl_timing']['generated_trace'])
self.assertIsNone(final['apl_timing']['applied_trace'])
self.assertFalse(obj.timing(1200, np.zeros((18, 4064), np.float32))['full_sequence_bitwise_audit_performed'])
self.assertEqual(inherited['arriving_edges'], 0)
def test_terminal_queue_schema_chronological_pending_and_fresh_tick(self):
class CPUArray:
def __init__(self, array):
self.value = array
def cpu(self):
return self
def numpy(self):
return self.value
ring = np.zeros((19, 166700), np.uint8)
ring[2, 3] = 1
apl_ring = np.repeat(np.arange(18, dtype=np.float32)[:, None], 4064, axis=1)
apl = types.SimpleNamespace(state=CPUArray(np.array([1., 2.], np.float32)))
obj = object.__new__(r._Runtime)
obj.mode, obj.installations, obj.finished = 'light', 1, False
obj.observer = types.SimpleNamespace(result=lambda: dict(all_finite=True, first_failure_tick=-1,
finite_observed_ticks=1200, arriving_edges=11, accepted_arriving_edges=7))
with tempfile.TemporaryDirectory() as directory:
obj.finish(CPUArray(ring), CPUArray(apl_ring), apl, 1200, Path(directory))
with np.load(Path(directory)/'terminal-queues.npz', allow_pickle=False) as output:
self.assertEqual(set(output.files), {'spike_ring', 'apl_state', 'apl_pending', 'next_tick'})
self.assertEqual(output['spike_ring'].dtype, np.uint8)
self.assertEqual(output['spike_ring'].tobytes(), ring.tobytes())
self.assertEqual(output['next_tick'].dtype, np.int64)
self.assertEqual(output['next_tick'].item(), 1200)
expected = np.concatenate((apl_ring[1200 % 18:], apl_ring[:1200 % 18]))
np.testing.assert_array_equal(output['apl_pending'], expected)
self.assertTrue(obj.finished)
with self.assertRaises(ValueError):
obj.finish(CPUArray(ring), CPUArray(apl_ring), apl, 1200, Path(directory))
def test_pinned_overlay_helpers_toy_units_and_missing_nonzero_fail_closed(self):
if not AM.is_file():
self.skipTest('Original frozen AM overlay helper not installed')
helper = r._load('phase4an_test_overlay_helper', AM, r.AM_OVERLAY_SHA)
ids, base32 = np.array([3, 9], np.int64), np.array([.275, 1.1], np.float32)
delta = np.float32(-.25) * base32
plan = helper.validate_arrays(ids, delta, ids, base32)
base = np.array([.000275, .0011], np.float64)
result, _ = helper.materialize_numpy(base, ids, np.zeros(2, bool), plan)
self.assertTrue(helper.bit_equal(result, base + delta.astype(np.float64) * .001))
with self.assertRaises(ValueError):
helper.map_installation(base[:1], ids[:1], np.zeros(1, bool), plan)
with self.assertRaises(ValueError):
helper.validate_arrays(ids, np.array([.1, 0], np.float32), ids, base32)
def test_manifest_cannot_self_authorize_replacement_mask(self):
with tempfile.TemporaryDirectory() as directory:
path = Path(directory)
manifest = dict(schema='phase4an-overlay-v1', overlay_name='learned', overlay_sha256='0'*64,
mask_path=str(path/'mask.npy'), mask_sha256='0'*64, base_candidate_path=str(path/'base.npy'),
base_candidate_sha256='0'*64, aj_model_dir=str(AJ.parent), am_overlay_path=str(AM),
am_overlay_sha256=r.AM_OVERLAY_SHA)
(path/'manifest.json').write_text(json.dumps(manifest), encoding='utf-8')
with self.assertRaisesRegex(ValueError, 'authority'):
r.load_overlay(path/'delta.npz', path/'manifest.json')
def test_learned_manifest_uses_external_base_and_requires_parent_v2(self):
if not AM.is_file() or not AJ.is_file():
self.skipTest('Pinned static source dependencies are not installed')
with tempfile.TemporaryDirectory() as directory:
path = Path(directory)
ids = np.arange(14232, dtype=np.int64)
base = np.full(14232, .275, np.float32)
delta = np.zeros(14232, np.float32); delta[0] = np.float32(-.00275)
np.save(path/'mask.npy', ids); np.save(path/'base.npy', base)
np.savez(path/'delta.npz', edge_id=ids, plastic_delta_mv=delta)
mask_sha = r.digest(path/'mask.npy')
manifest = dict(schema='phase4an-overlay-v1', overlay_name='learned',
overlay_sha256=r.digest(path/'delta.npz'), mask_path=str(path/'mask.npy'), mask_sha256=mask_sha,
base_candidate_path=str(path/'base.npy'), base_candidate_sha256=r.digest(path/'base.npy'),
aj_model_dir=str(AJ.parent), am_overlay_path=str(AM), am_overlay_sha256=r.AM_OVERLAY_SHA,
parent_snapshot_sha256='a'*64, eligibility_v2_source_sha256=r.V2_SHA)
(path/'manifest.json').write_text(json.dumps(manifest), encoding='utf-8')
# Only the authority fixture is patched; production pin remains fixed.
with patch.object(r, 'MASK_SHA', mask_sha):
helper, plan, meta, inputs = r.load_overlay(path/'delta.npz', path/'manifest.json')
self.assertEqual(plan['plastic_delta_mv'].tobytes(), delta.tobytes())
self.assertFalse(plan['plastic_delta_mv'].flags.writeable)
self.assertEqual(meta['parent_snapshot_sha256'], 'a'*64)
unexpected = dict(manifest, within_trial_weight_updates=True)
(path/'manifest.json').write_text(json.dumps(unexpected), encoding='utf-8')
with self.assertRaisesRegex(ValueError, 'Unknown'):
r.load_overlay(path/'delta.npz', path/'manifest.json')
manifest.pop('parent_snapshot_sha256')
(path/'manifest.json').write_text(json.dumps(manifest), encoding='utf-8')
with self.assertRaisesRegex(ValueError, 'parent'):
r.load_overlay(path/'delta.npz', path/'manifest.json')
if __name__ == '__main__':
unittest.main()